VR head-mounted system

Abstract

To provide a 6DOF-enabled head-mounted system which allows a user to view an entire-celestial-sphere image even if a head-mounted body is attached with a portable device.SOLUTION: There is provided a head-mounted system 100 in which a portable device SP including a camera CA on a first surface and a display unit DIS on a second surface is attached to a head-mounted body 10, and which allows a user to view an image projected onto the display unit DIS. The head-mounted body 10 includes a storage pocket 12 which stores the portable device such that the first surface faces forward, and which is notched such that the camera can image the front side. The portable device 10 comprises: a rotation amount sensor JS which senses a rotation amount in the three rotation directions of roll, pitch, and yaw; a translation movement amount calculation unit 32 which detects a feature point from a photographed image of the camera and calculates a translation movement amount in the three orthogonal axial directions; and an in-visual field image generation unit 36 which generates an in-visual field image from an entire-celestial-sphere image on the basis of the rotation amount and the translation movement amount.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a VR head-mounted system that uses a portable device such as a smartphone to display spherical images and other images in which the image within the field of view changes according to 6 DOF (Degree of Freedom). [Background technology]

[0002] In fields such as virtual reality (VR), the use of spherical images captured by a camera capable of capturing 360-degree images around is increasing. A user can view spherical images by attaching a portable device such as a smartphone to a head mount and displaying the spherical images on a display unit of the portable device as stereoscopic images with parallax that enable stereoscopic viewing. Patent Document 1 discloses an invention in which a portable device is attached to a head mount.

[0003] When the portable device is attached to the head mount, the portable device's angular velocity sensor can detect three movements (roll, pitch, and yaw) around the X, Y, and Z axes. This allows the user to view 3DOF-compatible spherical images by detecting the rotation and tilt of the user's head.

[0004] However, the head mount to which the portable device of Patent Document 1 is attached cannot detect three types of "translational movement" in the X-axis, Y-axis, and Z-axis directions. This poses a problem in that users cannot view 6DOF-compatible spherical images that include translational movement. Furthermore, in order to not only view spherical images but also rewind and advance them, users need to prepare a handheld controller in addition to a portable device such as a smartphone. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Special Publication No. 2019-510328 Summary of the Invention [Problem to be solved by the invention]

[0006] Therefore, the present embodiment aims to enable viewing of 6DOF compatible spherical images even with a head-mounted portable device, and also to enable processing of the spherical images (stop processing, rewind processing, etc.) without the need for a controller. [Means for solving the problem]

[0007] The head-mounted system of this embodiment is a head-mounted system in which a portable device including a camera on a first surface and a display on a second surface is attached to a head-mounted main body for viewing an image projected on the display. The head-mounted main body has a storage pocket cut out to accommodate the portable device with the first surface facing forward and to allow the camera to capture a forward image. The portable device further has a rotation amount sensor that detects the amount of rotation in three rotational directions (roll, pitch, and yaw), a translational movement amount calculation unit that detects feature points from an image captured by the camera and calculates the amount of translational movement in three orthogonal axis directions, and a field-of-view image generation unit that generates a field-of-view image from a celestial sphere image based on the amount of rotation and the amount of translational movement.

[0008] The portable device also preferably has a hand tracking unit that detects and tracks the user's hand from the image captured by the camera, a hand image generation unit that generates a hand image when the hand is within the captured image for longer than a predetermined time, a GUI image generation unit that generates a GUI image related to the image within the field of view when the hand is within the captured image for longer than the predetermined time, and an image synthesis unit that synthesizes the hand image and GUI image with the image within the field of view.

[0009] The portable device may also have a GUI instruction determination unit that determines whether the hand image has been on the pointing icon of the GUI image for a predetermined period of time, and the GUI instruction determination unit may instruct the execution of an action of the pointing icon when the hand image is on the pointing icon of the GUI image. The portable device also has a shopping store determination unit that determines whether the image within the field of view is a shopping store where products can be purchased, and a product image storage unit that stores multiple product images sold at the shopping store. When the GUI instruction determination unit determines that the hand image is on the shopping GUI image, it is preferable that the product image storage unit supplies the product images to the video composition unit.

[0010] The portable device also has a product identification determination unit that determines whether the hand image has been present in the product image for a predetermined period of time, and it is preferable that the product identification determination unit integrates the hand image and the product image, and separates the hand image and the product image in the cart image. When the product image storage unit supplies product images to the image synthesis unit, it is preferable that the image synthesis unit does not synthesize images within the field of view.

[0011] The method for projecting a field-of-view image of this embodiment involves storing a portable device including a camera on a first surface and a display on a second surface in a storage pocket of a head-mounted main body, and projecting the field-of-view image on the display. The storage pocket is cut out so that the camera can capture images ahead. The portable device then senses the amount of rotation in three rotational directions (roll, pitch, and yaw), detects feature points from the image captured by the camera, calculates the amount of translation in three orthogonal axis directions, and generates a field-of-view image from the omnidirectional image based on the amount of rotation and translation. Projects an image within the field of view. [Effects of the Invention]

[0012] The VR head-mounted system of this embodiment allows users to view 6DOF-compatible spherical images and to issue instructions for processing the spherical images without using a hand controller. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a perspective view of a head-mounted system to which a portable device according to the present embodiment is attached. [Figure 2] (A) is a diagram of the head mount (excluding the portable device) viewed from four directions, (B) is an explanatory diagram of the mechanism for adjusting the interpupillary distance, and (C) is an explanatory diagram of the mechanism for adjusting the focal length of the lens. [Figure 3] FIG. 1 is a block diagram of a portable device used in a VR head-mounted system. [Figure 4] 1 is a flowchart 1 for hand tracking. [Figure 5] (A) is an example of a hand-tracked hand image. (B) is an example of a GUI image, showing five instruction icons. (C) to (F) are examples of the field of view image displayed on the display of a portable device. Note that the image or video on the display is depicted as a single field of view image recognized by the human brain. [Figure 6] 2 is a flowchart for hand tracking. [Figure 7] (A) to (F) are examples of images displayed on the display of a portable device. The images or videos on the display are depicted as a single field of view image recognized by the human brain. DETAILED DESCRIPTION OF THE INVENTION

[0014] [Overall configuration of the head-mounted system] Fig. 1 is a perspective view showing a head-mounted system 100 according to this embodiment in which a portable device SP is attached to a head-mounted main body 10. As shown in Fig. 1, the head-mounted system 100 includes the head-mounted main body 10, a strap 20 for firmly attaching the head-mounted main body 10 to the user's head, and the portable device SP.

[0015] The portable device SP is stored in the storage pocket 12 of the head mount main body 10. The portable device SP disclosed in this embodiment has one or more cameras CA on its first surface (-Y axis side) and a display unit DIS (see FIG. 2) on its second surface (+Y axis side). Examples of the portable device SP include, but are not limited to, a smartphone, a tablet terminal, or a handheld visual media player.

[0016] The exemplary portable device SP includes a central processing unit (CPU) (not shown), a display unit DIS, a camera CA, and a communication unit, and can run applications for use with the system. The portable device SP50 incorporates one or more rotation angle sensors, such as a gyro sensor, an acceleration sensor, a hydrometer, or a magnetometer. In this embodiment, the rotation angle sensor can detect the amount of rotation of the head-mounted system 100 in three rotational directions: roll, pitch, and yaw.

[0017] A strap 20 is attached to the head mount main body 10, and the strap 20 securely fastens the head of the user. For example, the head mount main body 10 can also be incorporated into a helmet-like device that is fixed to the top of the head without a strap.

[0018] Storage pocket 12 for storing portable device SP has a flexible structure like a leaf spring, and holds portable device SP between peripheral wall 11 and storage pocket 12. The width (X-axis direction) of storage pocket 12 is shorter than the length of portable device SP and is formed with a cutout so that camera CA of portable device SP can take pictures of what is in front. If storage pocket 12 is long, the periphery corresponding to camera CA of portable device SP may be cut out so that camera CA can take pictures of what is in front.

[0019] The peripheral wall 11 is provided with a focus adjustment lever 14 for adjusting the focal length of the lens and an interpupillary distance adjustment dial 15 for moving the lens in accordance with the distance between the pupils.

[0020] [Head mount body configuration] Figure 2 shows the head mount main body 10 according to this embodiment. Figure 2(A) shows the head mount main body 10 as viewed from four directions. Figures 2(B) and 2(C) are views showing the periphery of the lens arranged inside the head mount main body 10.

[0021] The peripheral wall 11 and storage pocket 12 of the head mount body 10 are preferably made of a plastic material such as ethylene vinyl acetate (EVA), polyurethane (PU), or ABS resin. These may be used alone or in various combinations. In a preferred embodiment, the peripheral wall 11 and storage pocket 12 are molded by injection molding or the like. The front portion of the peripheral wall 11 has a generally rectangular or box-like shape.

[0022] The distance between the front surface 11a of the peripheral wall 11 and the storage pocket 12 is formed to be slightly shorter than the thickness of the portable device SP, and the storage pocket 12 is S-shaped when viewed from the side (X-axis direction). Due to the elasticity and shape of the plastic itself, the storage pocket 12 acts like a leaf spring, allowing the portable device SP (not shown) to be securely attached to the front surface 11a of the peripheral wall 11. Strap pins 19 are formed on the top and side surfaces of the peripheral wall 11 to which a strap 20 (see Figure 1) can be attached.

[0023] The width L1 of the storage pocket 12 can vary depending on the type and size of the portable device SP, but is typically intended for grasping and holding the portable device SP. The average size of the display of a portable device SP is approximately 5 inches (12.7 cm) to 6.5 inches (16.5 cm), and the camera CA on the first surface of the portable device SP is often located on the upper left side as viewed from the Y-axis direction. When the portable device SP is placed horizontally as shown in Figure 1, the camera CA is located on the upper right side. The width L1 of the storage pocket 12 is cut out so that the camera CA is not hidden.

[0024] The face edge 17, which comes into contact with the user's forehead, is preferably made of a flexible material such as rubber or urethane foam. The peripheral wall 11 and the face edge 17 are joined with an adhesive or a joint. When the face edge 17 comes into contact with the user's face, it prevents light from entering the hollow 13 from the surroundings.

[0025] A pair of lenses LZ are arranged in the hollow 13 for viewing images on the display unit of the portable device SP. The peripheral wall 11 has a focus adjustment lever 14 and an interpupillary distance adjustment dial 15 for moving the pair of lenses LZ in the Y-axis and X-axis directions. The interpupillary distance adjustment dial 15 adjusts the interpupillary distance, which varies depending on the user. As shown in FIG. 2(B), a pair of lens holders 18 that hold the lenses LZ each have a rack 18a. A pinion 15a is formed on the adjustment dial 15, and the rack 18a meshes with the pinion 15a. Rotation of the pinion 15a moves the lenses LZ held in the lens holder 18 toward the center or away from the center. Note that in FIG. 2(B), the pinion 15a and rack 18a of the interpupillary distance adjustment dial 15 are depicted separately for ease of understanding.

[0026] As shown in FIG. 2(C), the user moves the focus adjustment lever 14 back and forth (in the Y-axis direction) to adjust the focal length of the lens LZ. The lens holder 18 has a clamping plate 18b, which is clamped by the clamping portion 14a of the focus adjustment lever 14. The lens LZ is positioned between the user's eyes and the display of the portable device SP. As can be seen, the user's eyes are aligned with the lens LZ, and the user can look through the lens LZ to view the display of the portable device SP. The lens LZ can focus the user's field of view on a discrete area to the left or right of the field-of-view image projected on the display of the portable device SP. Proper alignment of the user's field of view through the lens is particularly important in virtual reality applications.

[0027] [Portable Device Configuration] FIG. 3 is a block diagram showing the functional configuration of the portable device SP. The portable device SP includes a smartphone, a tablet computer, a personal computer, or the like. The display device DIS is a display device such as a liquid crystal display or an organic EL display. The display device DIS may have separate display screens for the right and left eyes of the user, or may have only one display screen. An app is downloaded to the portable device SP from a store such as a website. By activating the app, the portable device SP has the functions described below. Data for multiple omnidirectional images can also be downloaded from the app.

[0028] The rotation amount sensor JS detects the movement of the head-mounted main body 10. The rotation amount sensor JS can detect rotations on three axes, namely yaw, roll, and pitch (3DOF; Degrees Of Freedom). The rotation amount sensor JS may be an IMU (inertial measurement unit) or may be a combination of various sensors such as a gyro sensor. If the rotation amount sensor JS is an IMU, it can detect movement in three directions, namely the Y-axis direction (front-back), the X-axis direction (left-right), and the Z-axis direction (up-down).

[0029] The portable device SP further includes a communication unit WF, an image data storage unit 31, a translational movement calculation unit 32, a hand tracking unit 33, a hand image generation unit 34, a GUI image generation unit 35, an in-field image generation unit 36, an image synthesis unit 37, and a GUI instruction determination unit 38. The portable device SP also includes a shopping store determination unit 41, a product image storage unit 42, and a product identification determination unit 43.

[0030] The communication unit WF performs long-distance communication such as 5G or 4G, or short-distance communication such as WiFi (trademark) or Bluetooth (trademark). The video data storage unit 31 acquires data of a spherical video (hereinafter referred to as video data) via the communication unit WF. A spherical video is a 360° omnidirectional video centered on a certain point, and is a video captured using a spherical camera or a video synthesized from videos captured by multiple cameras. The video data storage unit 31 may acquire the video data by reading out video data stored in the portable device SP. Upon acquiring the video data, the video data storage unit 31 decodes the video data and generates a spherical video. The video data storage unit 31 supplies the generated spherical video to the field-of-view video generation unit 36.

[0031] The translational movement calculation unit 32 periodically acquires images captured by the camera CA and calculates the translational movement amounts in the X, Y, and Z axes using characteristic points in real space as indices. Specifically, this method is disclosed in W. A. ​​Hoff and K. Nguyen, "Computer vision-based registration techniques for augmented reality," Proc. SPIE, vol. 2904, pp. 538-548, November 1996. 3, the translational movement calculation unit 32 is depicted as calculating the translational movement only from the image captured by the camera CA. However, if the rotation amount sensor JS is an IMU (inertial measurement unit), it can detect 6DOF, and therefore the translational movement amount in the XYZ axis direction may be calculated by combining the image captured by the camera CA with the translational direction signals in the XYZ axis direction from the IMU. Specific examples of calculation of the translational movement amount using periodically captured images and a sensor are disclosed in S. You and U. Neumann, "Fusion of vision and gyro tracking for robust augmented reality registration," Proc. IEEE Virtual Reality 2001, pp. 71-78, Mar. 2001. If low accuracy of the translational movement amount in the XYZ axis direction is acceptable, the translational movement amount detected by the IMU (inertial measurement unit) can be used, and the translational movement calculation unit 32 using the image captured by the camera CA may be omitted.

[0032] The hand tracking unit 33 recognizes the user's hand and fingers based on the image captured by the camera CA, and tracks the position of the hand and the movement of the fingers. The hand tracking unit 33 can use a deep learning model to determine whether a hand is present in the image captured by the camera CA.

[0033] The hand image generation unit 34 generates a hand image 51 to be displayed in the video based on the hand tracking unit 33's recognition of the user's hand. The hand image generation unit 34 preferably generates a hand image 51 when the user's hand is present in the captured image for a predetermined period of time. This is to prevent the user's hand from temporarily entering the video field of view when walking and waving. FIG. 5(A) shows an example of a hand image 51 projected on the display unit DIS. The hand image 51 is preferably a semi-transparent image so that the hand image 51 does not interfere with the viewing of the video field of view even when it is superimposed on the video field of view. The hand image generation unit 34 may prepare multiple unique hand images 51 (e.g., a hand image pointing, a hand image grasping an object with a thumb and index finger, etc.) and select a hand image 51 that closely resembles the shape of the hand in the image captured by the camera CA. Alternatively, the hand image 51 may be generated by detecting the outline of the captured hand through image processing. The generated hand image 51 is supplied to the video composition unit 37. When the hand is no longer present in the image captured by camera CA, hand image generating unit 34 stops generating the hand image, and the hand image is deleted.

[0034] The GUI image generation unit 35 generates a GUI image 52 related to the video based on the hand recognition. The GUI image generation unit 35 preferably prepares a plurality of GUI images 52 in advance. The GUI image 52 is, for example, an instruction icon for rewinding, reviewing, stopping, cueing, fast-forwarding, etc. FIG. 5B shows an example of a GUI image 52 projected on the display unit DIS. As will be described later, the GUI image 52 may also be an instruction icon for purchasing a product. Note that in FIG. 3, an arrow is drawn to indicate that the GUI image 52 is generated after the hand image generation unit 34 generates the hand image. However, after the hand tracking unit 33 recognizes the hand, the GUI image 52 may be generated simultaneously with the hand image 51. In other words, the GUI image generation unit 35 preferably generates the GUI image 52 when the user's hand is present in the captured image for a predetermined period of time. It is preferable that the number and type of instruction icons in the GUI image 52 change depending on the video within the field of view.

[0035] It is preferable that the GUI image generation unit 35 positions the generated GUI image 52 in the display space. The generated GUI image 52 is supplied to the image synthesis unit 37. It is preferable that the GUI image 52 is a semi-transparent image so that viewing of the in-field image is not obstructed even when the GUI image 52 is superimposed on the in-field image. As a result, even if the user wearing the head-mounted main body 10 changes direction or moves translationally in real space, the GUI image 52 is substantially fixed at a predetermined position based on the viewpoint in the display space, and the GUI image 52 does not move.

[0036] The field-of-view image generation unit 36 ​​generates a field-of-view image, which is an image to be displayed on the display unit DIS, from the celestial sphere image supplied from the image data storage unit 31. The field-of-view image generation unit 36 ​​acquires the amount of rotation and the amount of translational movement from the rotation amount sensor JS and the translational movement calculation unit 32, and can generate the field-of-view image by extracting a part of the celestial sphere image according to the orientation of the head-mounted main body 10 and the movement of the user. When the field-of-view image generation unit 36 ​​detects the amount of rotation about the three axes of yaw, roll, and pitch and the amount of movement in the X, Y, and Z axis directions, it moves the range of the celestial sphere image that becomes the field-of-view image according to the amount of rotation and movement.

[0037] As a result, when the user moves their head and moves forward, backward, left, right, up, and down, the range of the in-field image in the omnidirectional image moves in accordance with the movement of the head-mounted main body 10, allowing the user to view the omnidirectional image as if looking around. That is, the omnidirectional image realized by the in-field image forms a display space in which the position of the viewpoint is essentially controlled independently with respect to changes in the position of the user's head. Note that, although in-field images are projected onto the display unit DIS for the right eye and the left eye, in Figures 5 and 7, they are depicted as a single in-field image as perceived by the human brain.

[0038] Video synthesis unit 37 synthesizes the in-field video supplied from in-field video generation unit 36, the hand image supplied from hand image generation unit 34, and GUI image 52 supplied from GUI image generation unit 35. The synthesized video is projected onto display unit DIS.

[0039] The GUI instruction determination unit 38 determines whether the user's hand image is present for a predetermined time (1 to 2 seconds) in the GUI image 52 projected onto the display unit DIS by the image synthesis unit 37. If the GUI image 52 includes multiple instruction icons, it determines whether one of the instruction icons has been identified. For example, the user moves their hand to move the hand image to the instruction icon for the fast-forward button on the GUI image 52, and holds their hand there for one second. The GUI instruction determination unit 38 determines that a fast-forward instruction has been issued, and instructs the in-field image generation unit 36 ​​to fast-forward the image.

[0040] The shopping store determination unit 41 determines whether or not a shopping store is present in the image generated by the field-of-view image generation unit 36. The shopping store determination unit 41 determines whether or not a shopping store is present in the field-of-view image using a deep learning model. Instead of using deep learning, a unique signal indicating the presence of a shopping store may be inserted in advance into the spherical image, and the shopping store determination unit 41 may detect the unique signal and determine that a shopping store is present.

[0041] The product image storage unit 42 stores images (still images and video) of products available for sale at the shopping store. For example, if the products are bags, photos of available shoulder bags or handbags are taken in advance, and the product image storage unit 42 stores these images. The product image storage unit 42 also sequentially supplies product images of each product to the video composition unit 37, or supplies multiple products displayed as thumbnails to the video composition unit 37.

[0042] The product identification unit 43 determines whether the user's hand image is present for a predetermined time (1 to 2 seconds) on the product image projected onto the display unit DIS by the image synthesis unit 37, and determines whether the product has been identified. Alternatively, when the hand tracking unit 33 tracks the user's hand motion of pinching the product image with the thumb and index finger, the product identification unit 43 determines that the product has been identified. Then, the process proceeds to purchase the product.

[0043] [Head-mounted system operation] FIG. 4 is a flowchart showing the operation of the head-mounted system 100. First, the user starts an application downloaded to the portable device SP (S401). Then, the user sets the portable device SP in the storage pocket 12 of the head mount main body 10 (S402). The user then wears the head mount system 100.

[0044] The image within the field of view is projected onto the display unit DIS (S403), and the user views an image within the field of view such as that shown in Fig. 5(C). If necessary, the user moves the focus adjustment lever 14 to adjust the focal length of the lens, and also turns the interpupillary distance adjustment dial 15 to move the lens to match the distance between the pupils. The image within the field of view changes based on the signal of the movement (yaw, roll, and pitch) of the head-mounted main body 10 from the rotation amount sensor JS and the signal of the translation amount based on the camera CA of the portable device SP.

[0045] At the same time that the image within the field of view is projected onto the display unit DIS, hand tracking by the camera CA begins (S404). When the user thrusts their hand forward, the user's hand enters the field of view of the camera CA of the portable device SP. The hand tracking unit 33 then recognizes the user's hand and fingers and tracks the position of the hand and the movement of the fingers. If the user's hand remains within the field of view for a predetermined period of time (e.g., one second) (YES in S405), a hand image 51 generated by the hand image generation unit 34 is projected onto the display unit DIS (S406). As shown in FIG. 5(D), a semi-transparent hand image 51 is displayed superimposed on the image within the field of view.

[0046] Simultaneously with or several seconds after the hand image 51 is projected onto the display unit DIS, the GUI image 52 generated by the GUI image generation unit 35 is projected onto the display unit DIS (S407). As shown in FIG. 5(E), the translucent GUI image 52 is displayed superimposed on the image within the field of view in the center. In one example of this embodiment, the GUI image 52 displays, from left to right, a 10-second rewind instruction icon, a review instruction icon, a stop instruction icon, a cue instruction icon, and a 30-second fast-forward instruction icon. Note that when the image within the field of view is stopped, the stop instruction icon in the GUI image 52 is switched to a play instruction icon. In this embodiment, the GUI image 52 is projected at the center of the display unit DIS, but it may also be projected at the top or bottom of the display unit DIS. It is preferable that the GUI image 52 be projected at a constant position even if the image within the field of view changes when the user moves the head-mounted main body 10, for example by moving their head left or right.

[0047] GUI image 52 is projected at a fixed position even when the in-field image changes. Therefore, as shown in FIG. 5(E), when the user moves their hand, hand tracking unit 33 moves, for example, from hand image 51 drawn with a dotted line to hand image 51 drawn with a solid line, and the 10-second rewind instruction icon is maintained for a predetermined time. GUI instruction determination unit 38 determines whether hand image 51 is present on the instruction icon of GUI image 52 (S408). In FIG. 5(E), GUI instruction determination unit 38 determines that the 10-second rewind instruction icon has been selected, and instructs in-field image generation unit 36 ​​to rewind the image by 10 seconds (S409).

[0048] On the other hand, when the user removes his / her hand from the field of view of the camera CA (S405 NO), the image shown in Fig. 5(E) changes to the image shown in Fig. 5(F). That is, the hand image 51 is erased from the display unit DIS (S410), and simultaneously or after a short delay, the GUI image 52 is erased.

[0049] Next, we will explain, using the flowchart in Figure 6 and Figures 7(A) to 7(F), the case where a user selects video data that they want to watch from multiple video data, and the case where a user purchases a product when the video data is shopping video.

[0050] When the portable device SP application is launched, thumbnail displays that allow the user to select from multiple video data categories are projected onto the display unit DIS (S601). FIG. 7A shows an example of such a thumbnail display on the display unit DIS. For example, video data categories include Japan travel videos, Taiwan travel videos, underwater videos, and shopping videos. The user can move a thumbnail display that is not currently displayed by moving their hand and placing the hand image 51 on the horizontal arrow 71 for a predetermined period of time. FIG. 7A shows a state in which the hand image 51 has selected the shopping video category. As shown in FIG. 7B, a thumbnail display that allows the user to select one shopping video from multiple shopping videos is projected onto the display unit DIS. If the user wants to view underwater videos instead of shopping videos, they can move their hand and place the hand image 51 on the superordinate category arrow 73 for a predetermined period of time. In this embodiment, the user places the hand image 51 on shopping video B 74 for a predetermined period of time to select shopping video B.

[0051] The field-of-view image generation unit 36 ​​projects shopping image B (S602). FIG. 7(C) shows an example of a shopping image, which is a virtual shopping mall 74. As the user moves their head and moves back and forth and left and right, the rotation amount and translation amount are obtained from the rotation amount sensor JS and the translational movement calculation unit 32, and the field-of-view image generation unit 36 ​​projects an image of the shopping mall and an image of a store in the shopping mall. When the user moves and enters a virtual store (e.g., a bag store) in the shopping mall, the shopping store determination unit 41 determines whether the shopping store is a shopping store where the user can purchase a product. If the user can purchase a product at all virtual stores in the shopping mall, the shopping store determination unit 41 is not necessary. However, if the user can purchase a product at, for example, some virtual stores in the shopping mall, the shopping store determination unit 41 detects a unique signal included in the spherical image or recognizes it using deep learning to indicate to the user whether the virtual store the user has entered is a shopping store (S603).

[0052] If the virtual store through which the user is moving is a shopping store where products can be purchased (YES in S603), the shopping store determination unit 41 projects the shopping GUI image 52a onto the display unit DIS (S604). FIG. 7(D) shows an example in which the shopping GUI image 52a is projected onto the bottom edge of the display unit DIS in the field-of-view image of the shopping store 75. When the user moves their head in the field-of-view image of the shopping store 75, the field-of-view image changes, but the shopping GUI image 52a remains projected in a fixed position. When the user moves back and forth or left and right and leaves the shopping store 75 and enters an aisle or the like of the shopping mall 74, the shopping GUI image 52a is erased. If the virtual store through which the user is moving is not a shopping store where products can be purchased (NO in S604), the field-of-view image of the shopping mall continues to be projected onto the display unit DIS (S403).

[0053] Next, GUI instruction determination unit 38 determines whether hand image 51 is present in shopping GUI image 52a (S605). FIG. 7(D) shows a state in which hand image 51 is present in shopping GUI image 52a. If hand image 51 is present in shopping GUI image 52a (YES in S605), product image storage unit 42 supplies multiple product images (still images or video) of that shopping store 75 to video composition unit 37. If hand image 51 is not present in shopping GUI image 52a, hand tracking unit 33 continues to track the position of the hand and the movement of the fingers based on the image captured by camera CA (S404).

[0054] When the product image storage unit 42 supplies multiple product images to the video composition unit 37, a product image 76 is projected as shown in FIG. 7(E), and the user can view another product image 76 by moving the thumbnail image (product image 76) using the horizontal arrow 71. Note that in FIG. 7(E), the in-field image of the shopping store 75 is not composited, and only the product image 76 is projected so that the user can easily view the product image 76. In other words, the video composition unit 37 does not composite the in-field image supplied from the in-field image generation unit 36. However, the in-field image of the shopping store 75 may be projected semi-transparently.

[0055] The user moves their hand, and the hand tracking unit 33 tracks the position of the hand and the movement of the fingers. When the hand image 51 is maintained on a certain product image 76 for a predetermined time, the product identification determination unit 43 determines that the product image 76 has been identified (S608). Alternatively, when the hand tracking unit 33 tracks a movement of the user's hand such as pinching the product image 76 with the thumb and index finger, the product identification unit 43 determines that the product image 76 has been identified (S608). When the product image 76 is identified, the hand image 51 and the product image 76 become integrated and can be moved.

[0056] Next, the product identification determination unit 43 determines whether the hand image 51 has been integrated with the product image 76 and moved to the cart 77, i.e., whether the hand image 51 is present in the cart image 77 (S609). If the hand image 51 is present in the cart image 77 (S609 YES), the product image 76 enters the cart 77, and the hand image 51 and the product image 77 are separated (S610). Then, as shown in FIG. 7(F), a product purchase GUI image 52b is projected (S611). Icons projected on the product purchase GUI image 52b include a "Proceed to Purchase" instruction icon, a "Return Product" instruction icon, a "Continue Shopping" instruction icon, and an "Exit Shopping" instruction icon. The user moves their hand to move the hand image 51 to one of these instruction icons. If the hand image 51 remains on the "Proceed to Purchase" instruction icon for a predetermined period of time, the user can proceed to a screen for purchasing the product. [Industrial Applicability]

[0057] In this embodiment, an example has been described in which products can be purchased in the case of shopping footage, but for example, if an image of a snorkel or fins appears in the middle of an underwater video, a screen for purchasing the snorkel or fins may be projected. [Explanation of symbols]

[0058] 100...head-mounted system 10... Head-mounted body 11 ... Peripheral wall, 12 ... Storage pocket 14...Focus adjustment lever, 15...Pupillary distance adjustment dial 17 ... Face edge, 20 ... Strap 31 ... video data storage unit, 32 ... translational movement calculation unit 33 ... Hand tracking unit, 34 ... Hand image generation unit 35 ... GUI image generation unit, 36 ... Field of view image generation unit 37...Video synthesis section, 38...GUI instruction judgment section 41 ... Shopping store determination unit, 42 ... Product image storage unit 43 … Product Specification Judgment Department 51 ... Hand image, 52 ... GUI image CA: Camera, DIS: Display, JS: Rotation sensor SP: Portable device, WF: Communication unit

Claims

1. A head-mounted system for viewing an image projected on the display unit, in which a portable device including a camera on a first surface and a display unit on a second surface is attached to a head-mounted main body, The head mount body includes: the portable device is stored with the first surface facing forward, and the storage pocket is cut out so that the camera can photograph a scene in front of it; The portable device comprises: a rotation amount sensor that detects the amount of rotation in three rotation directions of roll, pitch, and yaw; a translational movement amount calculation unit that detects feature points from the image captured by the camera and calculates translational movement amounts in three orthogonal axis directions; a visual field image generation unit that generates a visual field image from the omnidirectional image based on the rotation amount and the translational movement amount; a hand tracking unit that detects and tracks the user's hand from the image captured by the camera; a hand image generating unit that generates a hand image when the hand is within the captured image for a predetermined period of time; a GUI image generating unit that generates a GUI image related to the image within the field of view when the hand is within the captured image for a predetermined period of time; an image synthesis unit that synthesizes the hand image and the GUI image with the image within the field of view; a GUI instruction determination unit that determines whether the hand image has been present on the instruction icon of the GUI image for a predetermined period of time; a shopping store determination unit that determines whether the image within the field of view is of a shopping store where products can be purchased; a product image storage unit that stores a plurality of product images sold at the shopping store, The head-mounted system, wherein the GUI instruction determination unit determines that the hand image exists on a GUI image of a shopping store for a predetermined period of time, and the product image storage unit supplies the product image to the video synthesis unit.

2. The portable device comprises: a product identification determination unit that determines whether the hand image has been present on the product image for a predetermined period of time; The head-mounted system according to claim 1 , wherein the commodity identification determination unit integrates the hand image and the commodity image, and separates the hand image and the commodity image in the cart image.

3. When the product image storage unit supplies the product image to the video composition unit, The head-mounted system according to claim 1 , wherein the image synthesizer does not synthesize the in-field image.

4. A method for projecting an image within a field of view onto a display unit by storing a portable device including a camera on a first surface and a display unit on a second surface in a storage pocket of a head mount body, the method comprising: The storage pocket is cut out so that the camera can photograph a front view, the portable device It detects the amount of rotation in three directions: roll, pitch, and yaw. Detecting feature points from the image captured by the camera and calculating translational movement amounts in three orthogonal axis directions; generating a field-of-view image from the omnidirectional image based on the amount of rotation and the amount of translation; Detecting and tracking the user's hand from the image captured by the camera; generating a hand image when the hand is within the captured image for a predetermined period of time; generating a GUI image related to the image within the field of view when the hand is within the captured image for a predetermined period of time; synthesizing the hand image and the GUI image with the image within the field of view; determining whether the hand image has been present on the pointing icon of the GUI image for a predetermined period of time; determining whether the image within the field of view is a shopping store where a product can be purchased; storing a plurality of product images sold at the shopping store; The method further comprises providing the product image to the in-field image when the hand image has been present on a GUI image of a shopping store for a predetermined period of time.

5. The portable device, determining whether the hand image has been present on the product image for a predetermined period of time; The method of claim 4 , further comprising integrating the hand image and the product image and separating the hand image and the product image in a cart image.

6. A method as described in claim 4, wherein when the product image is supplied to the field of view image, the field of view image is not synthesized.

Images