Information processing apparatus, information processing method, and program

Abstract

To provide a technique for appropriately notifying a user of a risk of collision in an MR technique.SOLUTION: An information processing apparatus comprises: acquisition means that acquires a captured image obtained by capturing real space including a first object and a second object; generation means that generates a virtual object; composition means that combines the captured image and the virtual object to generate a composite image; display control means that controls display means so as to display the composite image; and notification means that performs notification to a user in accordance with the distance between the first object and the second object in a first case where the virtual object is superimposed on a front side of the second object in space expressed by the composite image.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] In MR systems, there is a hand masking technology that uses an image of the user's actual hand to express the foreground / background relationship between the CG object and the hand through occlusion. By using hand masking technology, the user can understand the foreground / background relationship between the virtual CG object and the user's hand. This allows the user to feel as if the CG object is a real object.

[0003] Patent Document 1 describes a technique related to a hand mask. Patent Document 2 describes a technique for changing the transparency of a CG when an HMD (head mounted display) approaches behind the CG. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2002-157606 [Patent Document 2] Japanese Patent Publication No. 2020-536305 Summary of the Invention [Problem to be solved by the invention]

[0005] With hand masking technology, when a user's actual hand is positioned between a virtual CG object and the HMD, the CG object is masked in the shape of the hand, making the actual hand visible to the user. This makes the user's hand appear to be in front of the CG object. Also, when the actual hand is behind the virtual CG object, the actual hand is hidden by the virtual CG object. This also allows the user to experience the sensation that the virtual CG is actually present.

[0006] However, with hand masking technology, if a real object is hidden behind a virtual CG object, the user cannot see the real object, which means the user cannot grasp the distance between their real hand and the real object, and is therefore unable to sense the risk of contact.

[0007] Therefore, an object of the present invention is to provide a technique that can appropriately notify a user of the risk of a collision using MR technology. [Means for solving the problem]

[0008] One aspect of the present invention is an acquisition means for acquiring a captured image of a real space including the first object and the second object; a generating means for generating a virtual object; a synthesis means for synthesizing the captured image and the virtual object to generate a synthesized image; a display control means for controlling a display means to display the composite image; a notification means for providing a notification to a user in accordance with a distance between the first object and the second object in a first case where the virtual object is superimposed in front of the second object in the space represented by the composite image; The information processing device is characterized by having:

[0009] One aspect of the present invention is a method for producing a medicament for the treatment of a pulmonary arthritis. an acquisition step of acquiring a captured image of a real space including the first object and the second object; a generation step of generating a virtual object; a synthesis step of synthesizing the captured image and the virtual object to generate a synthetic image; a display control step of controlling a display means to display the composite image; a notification step of notifying a user of a distance between the first object and the second object in a first case where the virtual object is superimposed in front of the second object in the space represented by the composite image; The information processing method is characterized by comprising: [Effects of the Invention]

[0010] According to the present invention, it is possible to provide a technique that can appropriately notify a user of the risk of a collision using MR technology. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a configuration diagram of an HMD system according to an embodiment. [Figure 2] 1 is a configuration diagram of an HMD system according to a first embodiment. [Figure 3] FIG. 1 is a diagram illustrating an MR space according to the first embodiment. [Figure 4] 10A to 10C are diagrams illustrating a process for emphasizing the display of a hand according to the first embodiment. [Figure 5] 10A to 10C are diagrams illustrating a process for emphasizing the display of a hand according to the first embodiment. [Figure 6] 10 is a flowchart of a process for emphasizing a display of a hand according to the first embodiment. [Figure 7] FIG. 10 is a configuration diagram of an HMD system according to a second embodiment. [Figure 8] 10A and 10B are diagrams illustrating a process for highlighting the display of an operation member according to the second embodiment. [Figure 9] 10A and 10B are diagrams illustrating a process for emphasizing the display of feet according to the second embodiment. [Figure 10]10 is a flowchart of an emphasis process according to the second embodiment. [Figure 11] FIG. 10 is a configuration diagram of an HMD system according to a third embodiment. [Figure 12] 13 is a flowchart of a first example of a notification process according to the third embodiment. [Figure 13] 13 is a flowchart of a second example of a notification process according to the third embodiment. [Figure 14] 13 is a flowchart of a third example of a notification process according to the third embodiment. [Figure 15] FIG. 10 is a diagram illustrating an example 1 of a notification process according to the third embodiment. [Figure 16] FIG. 10 is a diagram illustrating a second example of a notification process according to the third embodiment. [Figure 17] FIG. 10 is a diagram illustrating a third example of a notification process according to the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, each embodiment of the present invention will be described in detail with reference to the accompanying drawings. First, matters common to each embodiment will be described.

[0013] First, one method to prevent contact between a real object hidden behind a virtual CG object and the user's real hand is to increase the transparency of the CG when the HMD approaches the real object, allowing the user to see the real object. However, this method does not provide the same immersive feeling as if the CG were real, and it also poses UX (User eXperience) issues, such as interruptions to work.

[0014] 1 is a block diagram showing the configuration of an HMD system according to an embodiment. The HMD system is a head-mounted system that displays an image of real space and virtual computer graphics (CG) in an overlapping manner. As such, the embodiment can be applied to various HMD systems, including HMDs that can be worn on a user's head.

[0015] The HMD system includes a lens unit 101, an imaging unit 102, an imaging processing unit 103, an image processing unit 104, an image synthesis unit 105, a display unit 106, an eyepiece optical system 107, an imaging control unit 108, and an HMD control unit 109. The HMD system also includes a mask generation unit 110, a distance sensor 111, a sensor control unit 112, a power supply unit 113, an operation unit 114, a shake detection unit 115, a self-position estimation unit 116, a speaker 117, a speaker control unit 118, and an HMD communication control unit 119. The HMD system also includes a haptic device 120, a haptic control unit 121, an HMD communication unit 122, a haptic communication unit 123, an object distance acquisition unit 124, a comparison unit 125, a background distance acquisition unit 126, and a space map generation unit 127. The HMD system also includes a memory unit 128, a CG generation unit 129, an object emphasis unit 130, a transparency change unit 131, and a CG communication control unit 132.

[0016] The imaging unit 102 captures an image of a real space including a plurality of objects such as a user's hands via the lens unit 101.

[0017] The image (captured image) output from the imaging unit 102 is subjected to image processing by the imaging processing unit 103 and the image processing unit 104, and then sent to the image synthesis unit 105. The image synthesis unit 105 synthesizes the captured image with computer graphics (CG; virtual object) generated by the CG generation unit 129 to generate a synthesized image.

[0018] The display unit 106 displays the composite image, which is then focused onto the retina of the eye through the eyepiece optical system 107.

[0019] The imaging control unit 108 controls exposure in the imaging unit 102 .

[0020] The distance sensor 111 is built into the HMD. The distance sensor 111 captures a distance image indicating distance information under the control of the sensor control unit 112. In the distance image, each pixel indicates the distance to an object that appears in that pixel.

[0021] The speaker 117 is a device that reproduces sound by being controlled by the speaker control unit 118. The haptic device 120 is a separate unit from the HMD. The haptic device 120 is a device that vibrates in accordance with a haptic vibration pattern (a vibration pattern received by the haptic communication unit 123 from the HMD communication unit 122). The haptic device 120 is controlled by the haptic control unit 121.

[0022] The power supply unit 113 supplies power (electricity) according to the purpose of the entire system. The operation unit 114 is an operation unit used by the user to operate the HMD system. The operation unit 114 outputs an operation signal to the HMD control unit 109. The shake detection unit 115 detects the amount of shake applied to the HMD and generates a detection signal indicating the amount of shake. The shake detection unit 115 outputs the detection signal to the self-position estimation unit 116 together with the captured image processed by the image processing unit 104. The self-position estimation unit 116 estimates the self-position of the HMD based on the detection signal indicating the amount of shake and the captured image.

[0023] The HMD control unit 109 controls the entire HMD system. The HMD control unit 109 includes a CPU. The HMD control unit 109 communicates with a CG communication control unit 132 via an HMD communication control unit 119.

[0024] The object distance acquisition unit 124 acquires the distance from the HMD to the subject based on the captured image (stereo camera image) acquired from the imaging unit 102 and / or the distance image acquired from the distance sensor 111. Specifically, the object distance acquisition unit 124 acquires a distance image (hereinafter referred to as an "object map") indicating the distance to a specific object based on the image acquired from the imaging unit 102 or the distance sensor 111.

[0025] The background distance acquisition unit 126 acquires a distance image of the background based on an image obtained from the imaging unit 102 or the distance sensor 111. The space map generation unit 127 generates a distance image of the real space excluding the specific object (hereinafter referred to as a "space map") based on the distance image of the background acquired by the background distance acquisition unit 126. The comparison unit 125 calculates the distance between the specific object and the background based on the object map and the space map.

[0026] The mask generation unit 110 uses an object map to generate a mask for cutting out the CG. The CG generation unit 129 generates the CG. Furthermore, the CG generation unit 129 uses the mask generated by the mask generation unit 110 to cut out (exclude) the mask area from the generated CG. The cut-out CG is combined with the captured image by the image composition unit 105. The image composition unit 105 performs display control so that the display unit 106 displays the MR image including the CG.

[0027] The object emphasizing unit 130 performs image processing to emphasize the area of ​​a specific object according to the distance between the specific object and the background.

[0028] The transparency change unit 131 changes the transparency of the CG according to the distance between a specific object and the background.

[0029] The speaker 117 notifies the user of the risk of collision by controlling the volume of the warning sound according to the distance between the specific object and the background calculated by the comparison unit 125.

[0030] Next, an example of the general operation of the HMD system having the above configuration will be described.

[0031] (1) A captured image of real space including the hand is acquired by the lens unit 101 and the imaging unit 102. The object distance acquisition unit 124 acquires a distance image of the hand area based on the captured image or from a distance image obtained by the distance sensor 111.

[0032] (2) The background distance acquisition unit 126 acquires the distance from the HMD to the background based on the captured image or the distance image obtained from the distance sensor 111. The space map generation unit 127 generates a space map based on the distance image indicating the distance from the HMD to the background. The comparison unit 125 compares the distance of the hand with the distance of the background. At the same time, the mask generation unit 110 generates a mask image of the hand based on the hand map.

[0033] (3) The object emphasis unit 130 and the CG generation unit 129 acquire a mask image of the hand and information on the distance between the hand and the background. The object emphasis unit 130 performs emphasis processing on the image of the hand according to the distance between the hand and the background. The CG generation unit 129 generates the CG. If there is an area in the CG that overlaps with the area of ​​the hand, the CG generation unit 129 excludes that area from the CG.

[0034] (4) The image synthesis unit 105 synthesizes the enhanced image of the hand with CG (CG with the hand area removed) to generate a synthetic image. The display unit 106 delivers light to the user via the eyepiece optical system 107, allowing the user to perceive the synthetic image.

[0035] The above process highlights the hand area according to the distance between the hand and the background, thereby realizing a user experience (UX) that can notify the user of the risk of collision between the hand and an object.

[0036] <Embodiment 1> In the first embodiment, an HMD system that highlights a hand area according to the distance between the hand and an object will be described. The HMD system may be any electronic device (device) such as an HMD, a digital camera, or a smartphone.

[0037] 2 shows the configuration of the HMD system according to embodiment 1. Of the configurations of the HMD system shown in FIG. 1, FIG. 2 shows the configuration particularly related to embodiment 1.

[0038] The HMD system includes an optical system 201, an image sensor 202, an image processing unit 203, a synthesis unit 204, a display unit 206, a camera control unit 207, a position estimation unit 208, a CG generation unit 209, and a distance sensor 210. The HMD system also includes a distance sensor control unit 211, a hand map acquisition unit 212, a distance comparison unit 213, a mask generation unit 214, a distance image acquisition unit 215, a space map generation unit 216, and an emphasis unit 217.

[0039] The optical system 201 includes a lens unit 101 .

[0040] The image sensor 202 captures an image of a real space including a plurality of objects such as the user's hands, thereby acquiring a captured image. The image sensor 202 corresponds to the imaging unit 102.

[0041] The image processing unit 203 performs image processing on the captured image acquired by the image sensor 202. The image processing unit 203 corresponds to the image capturing processing unit 103 and the image processing unit 104.

[0042] The camera control unit 207 controls the imaging sensor 202. The camera control unit 207 corresponds to the imaging control unit .

[0043] The position estimation unit 208 estimates the position of the image sensor 202 (HMD) based on the captured image. The position estimation unit 208 corresponds to the shake detection unit 115 and the self-position estimation unit .

[0044] The distance sensor 210 captures an image of a subject to obtain a distance image. The distance sensor 210 corresponds to the distance sensor 111.

[0045] The distance sensor control unit 211 controls the distance sensor 210. The distance sensor control unit 211 corresponds to the sensor control unit 112.

[0046] Distance image acquisition section 215 converts the captured image into a distance image. Alternatively, distance image acquisition section 215 acquires a distance image from distance sensor 210. Distance image acquisition section 215 may also correct the distance image acquired from distance sensor 210 based on the captured image. Hereinafter, the distance image acquired by distance image acquisition section 215 will be referred to as the "whole distance image."

[0047] Based on the whole distance image, hand map acquisition unit 212 acquires an image showing distance information of the hand area (hereinafter referred to as "hand map"). Hand map acquisition unit 212 corresponds to object distance acquisition unit 124.

[0048] The space map generating unit 216 obtains a space map as an image showing distance information of the real space excluding the hand based on the whole distance image. The space map generating unit 216 corresponds to the space map generating unit 127.

[0049] Distance comparison unit 213 calculates the distance between the hand and an object (an object existing in real space) based on the hand map and the space map. Note that distance comparison unit 213 may calculate the distance between the hand and the object by a method other than a method based on the hand map and the space map. Distance comparison unit 213 may, for example, calculate the distance between the hand and the object by a method other than a method based on the hand map and the space map. The distance between the hand and the object may be calculated based on the captured image.

[0050] The mask generation unit 214 generates a mask for the hand region based on the hand map. The mask generation unit 214 corresponds to the mask generation unit 110.

[0051] The CG generation unit 209 generates a CG (virtual object) based on the position of the HMD estimated by the position estimation unit 208. Then, when the CG generation unit 209 determines that the CG overlaps with the hand region based on the mask of the hand region, it excludes the range corresponding to the hand region from the CG. The CG generation unit 209 corresponds to the CG generation unit 129.

[0052] The highlighting unit 217 controls the degree of highlighting of the hand region according to the distance between the hand and the object (the distance calculated by the distance comparing unit 213). The highlighting unit 217 corresponds to the object highlighting unit .

[0053] The synthesis unit 204 generates a synthesized image by synthesizing the “CG” and the “image of the hand that has been subjected to the enhancement processing.” The synthesis unit 204 corresponds to the image synthesis unit 105.

[0054] The display unit 206 displays the composite image. This allows the display unit 206 to display an image of the hand that is highlighted according to the distance between the hand and the background. This allows the display unit 206 to notify the user of the risk of collision between the "real object behind the CG" and the "hand." The display unit 206 corresponds to the display unit 106.

[0055] Fig. 3C is a diagram showing an image 303 in which the CG printer 302 shown in Fig. 3B is superimposed on the actual desk 301 shown in Fig. 3A. An AR marker has been placed in advance on the actual desk 301. In MR, there is a use case in which a CG printer is placed on the AR marker and the user can touch the CG printer with their hand.

[0056] 4A and 4B are diagrams illustrating a function in which a real user's hand touches the "CG printer superimposed on a real desk" shown in FIG. 3C. In addition, in FIG. 4B, the hand area is highlighted based on the distance between the real hand and the real desk hidden behind the CG, thereby notifying the user of the risk of collision. In FIG. 4A, hand 401 is positioned slightly far from the real desk located behind the CG printer, so hand 401 is not highlighted. On the other hand, in FIG. 4B, hand 402 is positioned close to the real desk located behind the CG printer, so hand 402 is highlighted. In this way, the degree of emphasis on the real hand area allows the user to roughly grasp the distance between the real desk located behind the CG and the hand. This allows the user to enjoy a mixed reality experience with excellent UX.

[0057] 5A and 5B are diagrams that also visualize the actual desk located behind the CG. In Fig. 5A, hand 501 is positioned a little far from the actual desk located behind the CG printer, so hand 501 is not emphasized. In Fig. 5B, hand 502 is positioned close to the actual desk located behind the CG printer, so hand 502 is emphasized.

[0058] The hand region enhancement process according to the first embodiment will be described with reference to the flowchart in Fig. 6. Note that the following description will be given on the assumption that the hand and CG (virtual object) are superimposed on the composite image (space represented by the composite image). However, the process of the flowchart in Fig. 6 may be executed in other cases (for example, when the hand and CG are closer than a predetermined distance) as well. Furthermore, the hand region enhancement process may not be executed in cases other than when the hand and CG (virtual object) are superimposed on the composite image (space represented by the composite image). This flowchart starts in step S601.

[0059] In step S602, distance image acquisition unit 215 acquires an overall distance image based on the captured image acquired by image sensor 202. Alternatively, distance image acquisition unit 215 acquires an overall distance image based on the distance image acquired by distance sensor 210.

[0060] In step S603, the space map generation unit 216 generates a space map indicating distance information of the space in real space that does not include the hand, based on the entire distance image.

[0061] In step S604, hand map acquisition unit 212 generates a hand map based on the entire distance image.

[0062] In step S605, the mask generation unit 214 generates a mask for the hand region.

[0063] In step S606, the CG generation unit 209 generates CG. Then, the CG generation unit 209 uses a mask for the hand area to exclude the masked area from the CG. Alternatively, the CG generation unit 209 makes the masked area of ​​the CG transparent.

[0064] In step S607, the distance comparison unit 213 calculates the distance between the hand and a real object (for example, among multiple objects, the object located at the back (behind) of the hand and closest to the hand) based on the space map and the hand map.

[0065] In step S608, highlighting unit 217 controls the highlighting of the display of the hand region depending on the distance from the hand to the real object. Note that "controlling the highlighting of the hand region" may be interpreted as "controlling the display form of the hand region depending on the distance from the hand to the real object." For example, the shorter the distance from the hand to the real object, the brighter the hand region, the closer the color of the hand region to a specific color, or the larger the hand region.

[0066] The processing of step S608 may be performed only when the CG is superimposed in front of the real object in the space represented by the synthetic image (MR space). Note that if at least a portion of the real object is displayed in the synthetic image, the highlighting unit 217 may not execute "controlling the highlighting of the display of the hand area according to the distance from the hand to the real object." In this case, the user can infer the positional relationship between the hand and the real object by looking at the synthetic image, so the need for highlighting is reduced. Furthermore, not performing highlighting can also improve the user's sense of immersion in the MR.

[0067] In step S609, the highlighting unit 217 determines whether or not the termination condition is satisfied. If it is determined that the termination condition is not satisfied, the process returns to step S602. If it is determined that the termination condition is satisfied, the process of this flowchart ends in step S610. When the process of this flowchart ends, the synthesis unit 204 generates a composite image in which the CG and the captured image (the captured image in which the degree of enhancement of the hand area has been controlled) are synthesized, and the composite image is displayed on the display unit 206.

[0068] According to the first embodiment, the HMD system highlights the hand according to the distance between the "real hand" and the "real object located behind the CG." This allows the HMD system to warn the user to prevent the user's hand from colliding with a real object hidden by the CG. Note that the first embodiment does not require methods that would break the immersion or interrupt the work, such as changing the transparency of the CG or issuing a warning sound to the user. This allows the user to obtain an excellent UX that allows them to experience the hand mask function of MR.

[0069] <Embodiment 2> In the first embodiment, the degree of emphasis of the hand display is controlled according to the distance between the hand and the object. However, any object that the user can freely move may be used instead of the hand. Therefore, in the second embodiment, the HMD system performs emphasis processing on the area of ​​a specific object according to the distance between the "specific object (body part or operation member)" and an "object other than the specific object." Hereinafter, the "object other than the specific object" will be referred to as a "general object."

[0070] Fig. 7 shows the configuration of an HMD system according to embodiment 2. The HMD system includes an optical system 201, an image sensor 202, an image processing unit 203, a synthesis unit 204, a display unit 206, a camera control unit 207, a position estimation unit 208, a CG generation unit 709, and a distance sensor 210. The HMD system also includes a distance sensor control unit 211, an object map acquisition unit 712, a distance comparison unit 713, a mask generation unit 714, a distance image acquisition unit 215, a space map generation unit 216, and an enhancement unit 717. Of these components, the components shown in Fig. 2 perform the processing described in embodiment 1, and therefore will not be described here.

[0071] Object map acquisition unit 712 acquires an image (hereinafter referred to as an "object map") that shows distance information for the area of ​​a specific object based on the entire distance image acquired by distance image acquisition unit 215. The specific object may be an object designated in advance by the user, or may be the object closest to the user's hand.

[0072] The distance comparison unit 713 compares the distance between a specific object and a general object (such as the background) based on the object map and the space map.

[0073] The mask generator 714 generates a mask for a region of a particular object based on the object map.

[0074] The CG generation unit 709 generates the CG based on the position of the HMD estimated by the position estimation unit 208. Then, when the CG generation unit 709 determines that the CG overlaps the area of ​​the specific object based on the mask of the area of ​​the specific object, it excludes the range corresponding to the area of ​​the specific object from the CG.

[0075] The highlighting unit 717 controls the degree of highlighting of the region of the specific object according to the distance between the specific object and the general object (the distance calculated by the distance comparing unit 713).

[0076] 8A and 8B are diagrams relating to a function that allows a user to touch a CG printer superimposed on a real desk with an operating member (a specific object) such as a screwdriver. Also, Fig. 8A and 8B relate to a UI that highlights the area of ​​the operating member according to the distance between the "actual operating member such as a screwdriver" and the "actual desk hidden behind the CG" to warn the user of the risk of collision.

[0077] In Fig. 8A, operation member 801 is located a little far from the actual desk located behind the CG printer, and therefore no emphasis processing is performed on operation member 801. On the other hand, in Fig. 8B, operation member 802 is located close to the actual desk located behind the CG printer, and therefore emphasis processing is performed on operation member 802. In this way, the degree of emphasis on the area of ​​the actual operation member allows the user to grasp the approximate distance between the actual desk located behind the CG and the operation member.

[0078] 9A and 9B are diagrams related to the function of experiencing riding in a CG car seat superimposed on a real chair. Also, Fig. 9A and 9B explain how the real feet (part of the body) are treated as specific objects and the foot area is highlighted according to the distance between the "real chair hidden behind the CG" and the "real feet."

[0079] In Fig. 9A, foot 901 is located a little far from the real chair located at the back of the CG car, so no emphasis processing is performed on foot 901. In Fig. 9B, foot 902 is located close to the real desk located at the back of the CG printer, so emphasis processing is performed on foot 902. In this way, the degree of emphasis of the real foot area allows the user to grasp the approximate distance between the real chair located at the back of the CG and the foot.

[0080] The processing according to the second embodiment will be described with reference to the flowchart of Fig. 10. Note that in Fig. 10, steps with the same names as those in the flowchart of Fig. 6 perform the same processing, and therefore descriptions thereof will be omitted.

[0081] In step S1004, the object map acquisition unit 712 generates an object map based on the entire distance image.

[0082] In step S1005, the mask generation unit 714 generates a mask for the area of ​​the specific object.

[0083] In step S1006, the CG generating unit 709 generates CG, and then, using a mask of the area of ​​the specific object, the CG generating unit 709 excludes the range corresponding to the area of ​​the specific object from the CG.

[0084] In step S1007, the distance comparison unit 713 calculates the distance between the specific object and the real object (general object) that is closest to the specific object, based on the space map and the object map.

[0085] In step S1008, the highlighting unit 717 controls the highlighting of the region of the specific object according to the distance calculated by the distance comparing unit 713. For example, the shorter the distance from the specific object to the general object, the more the highlighting unit 717 highlights the specific object so that it is easier for the user to understand. Specifically, the shorter the distance from the specific object to the general object, the brighter the region of the specific object or the larger the specific object becomes. The processing of step S1008 may be executed only when the CG is superimposed in front of the general object in the space represented by the composite image (MR space).

[0086] According to the second embodiment, the HMD system highlights a specific object (a part of an operating member or a part of the body) depending on the distance between the specific object and another object located behind the CG. This allows the HMD system to warn the user to prevent a collision between a real object hidden by the CG and the specific object. The HMD system does not change the transparency of the CG or issue a warning to the user, thereby eliminating the sense of immersion or interrupting the work. This makes it possible to provide an excellent UX that allows the user to experience the hand mask function of MR.

[0087] <Embodiment 3> In the third embodiment, the HMD system highlights the hand area when the hand is located in front of the CG. When the hand is located behind the CG and cannot be seen, the HMD system does not highlight the hand. In this case, the HMD system Change the vibration strength of a haptic device, change the volume of a warning sound depending on the distance, or change the transparency of CG depending on the distance.

[0088] FIG. 11 shows a configuration diagram of an HMD system according to a third embodiment. In addition to the configuration of the HMD system shown in FIG. 7, the HMD system according to the third embodiment includes a haptic control unit 1118, an audio control unit 1119, a haptic device 1120, and a speaker 1121. The newly added configuration executes a process of notifying a user of the risk of collision when the specific object is not displayed in the composite image because the specific object is located further back than the CG. The haptic device 1120, the speaker 1121, and the display unit 206 notify the user of the risk of collision, and therefore these configurations can also be collectively referred to as a "notification unit." Below, three examples are shown to explain the process of notifying a user of the risk of collision executed by the haptic device 1120, the speaker 1121, and the display unit 206.

[0089] In a first example (Example 1), the haptics control unit 1118 switches the control pattern of the haptics intensity (vibration intensity) depending on the distance between the specific object and the general object. The haptics device 1120 vibrates according to the switched control pattern to notify the user of the risk of collision between the specific object and the general object behind the CG.

[0090] In a second example (Example 2), the audio control unit 1119 switches the volume of the output from the speaker 1121 depending on the distance between the specific object and the general object. The speaker 1121 emits a sound that notifies the user of the risk of collision between the specific object and a general object behind the CG, depending on the switched output volume.

[0091] In a third example (Example 3), the CG generation unit 709 changes the transparency of the CG depending on the distance between the specific object and the general object. The compositing unit 204 performs compositing using the CG with the changed transparency. The display unit 206 displays a composite image using the CG with the changed transparency, thereby notifying the user of the risk of collision between the specific object and the general object behind the CG.

[0092] In this way, when a specific object such as a hand is positioned in front of the CG, the specific object is emphasized. On the other hand, when a specific object such as a hand is positioned behind the CG, the haptic intensity, volume, or CG transparency is changed. This allows the user to be notified of a collision with a real object behind the CG, even when the hand or other object is hidden by the CG and cannot be seen.

[0093] The processing according to the third embodiment for the above-mentioned example 1 will be described with reference to the flowchart in Fig. 12. In Fig. 12, the steps with the same names as those in the flowchart in Fig. 10 execute the same processing as in the second embodiment, and therefore the description thereof will be omitted.

[0094] In step S1201, distance comparison unit 713 determines whether the specific object is buried in the CG (whether the specific object is located further back than the CG). If it is determined that the specific object is buried in the CG, the process proceeds to step S1202. If it is determined that the specific object is not buried in the CG, the process proceeds to step S1005.

[0095] In step S1202, the haptics control unit 1118 generates a haptic signal indicating vibration intensity according to the distance calculated by the distance comparison unit 713. Specifically, for example, the haptics control unit 1118 increases the vibration intensity indicated by the haptic signal as the distance calculated by the distance comparison unit 713 becomes shorter.

[0096] In step S1203, the haptics control unit 1118 transmits the generated haptic signal to the haptic device 1120.

[0097] In step S1204, the haptic device 1120 vibrates with a vibration intensity according to the received haptic signal.

[0098] The processing according to the third embodiment for the above-mentioned example 2 will be described with reference to the flowchart in Fig. 13. In Fig. 13, the steps with the same names as those in the flowchart in Fig. 10 execute the same processing as in the second embodiment, and therefore the description thereof will be omitted.

[0099] In step S1301, distance comparison unit 713 determines whether the specific object is buried in the CG (whether the specific object is located further back than the CG). If it is determined that the specific object is buried in the CG, the process proceeds to step S1302. If it is determined that the specific object is not buried in the CG, the process proceeds to step S1005.

[0100] In step S1302, the audio control unit 1119 generates a sound signal indicating a volume corresponding to the distance calculated by the distance comparison unit 713. Specifically, for example, the audio control unit 1119 increases the volume indicated by the sound signal as the distance calculated by the distance comparison unit 713 becomes shorter.

[0101] In step S 1303 , the audio control unit 1119 transmits the generated sound signal to the speaker 1121 .

[0102] In step S1304, the speaker 1121 outputs a sound (voice) indicating a warning at a volume corresponding to the received sound signal.

[0103] The processing according to the third embodiment for the above-mentioned example 3 will be described with reference to the flowchart in Fig. 14. In Fig. 14, the steps with the same names as those in the flowchart in Fig. 10 execute the same processing as in the second embodiment, and therefore the description thereof will be omitted.

[0104] In step S1401, distance comparison unit 713 determines whether the specific object is buried in the CG (whether the specific object is located further back than the CG). If it is determined that the specific object is buried in the CG, the process proceeds to step S1402. If it is determined that the specific object is not buried in the CG, the process proceeds to step S1005.

[0105] In step S1402, the CG generation unit 709 controls the transparency of the generated CG in accordance with the distance calculated by the distance comparison unit 713. Specifically, for example, the CG generation unit 709 increases the transparency of the CG as the distance calculated by the distance comparison unit 713 becomes shorter.

[0106] In step S1403, the CG generation unit 709 transmits the CG with controlled transparency to the synthesis unit 204.

[0107] In step S1404, the composition unit 204 combines the CG with controlled transparency with the captured image to generate a composite image.

[0108] In Fig. 15A, the hand is positioned in front of the CG, so the area of ​​the hand is highlighted according to the distance between the hand and the "real desk behind the CG." In Fig. 15B, the hand is positioned behind the CG, so the vibration intensity of the haptic device 1120 attached to the finger is changed according to the distance between the hand and the "real desk behind the CG." This allows the user to intuitively grasp the distance between the hand hidden by the CG and the real desk, even if the hand is hidden by the CG, as shown in Fig. 15B.

[0109] In Fig. 16A, the hand is positioned in front of the CG, so the area of ​​the hand is highlighted according to the distance between the hand and the "real desk behind the CG." In Fig. 16B, the hand is positioned behind the CG, so the volume of the warning sound from speaker 1121 is changed according to the distance between the hand and the "real desk behind the CG." This allows the user to intuitively grasp the distance between the hand hidden by the CG and the real desk, even if the hand is hidden by the CG, as shown in Fig. 16B.

[0110] In Fig. 17A, the hand is positioned in front of the CG, so the area of ​​the hand is highlighted according to the distance between the hand and the "real desk behind the CG." In Fig. 17B, the hand is positioned behind the CG, so the transparency of the CG is changed according to the distance between the hand and the "real desk behind the CG." This allows the user to intuitively grasp the distance between the hand hidden by the CG and the real desk, even if the hand is hidden by the CG, as shown in Fig. 17B.

[0111] According to the third embodiment, the HMD system does not highlight a specific object if the specific object is located behind the CG and is not visible. In this case, the HMD system uses a method of changing the vibration intensity of the haptic device according to the distance between the specific object and a general object, a method of changing the volume of the warning sound according to the distance, or a method of changing the transparency of the CG according to the distance. This allows the user to recognize the distance between the specific object and other objects even if the specific object is hidden by the CG. The user can avoid the risk of the specific object colliding with other objects.

[0112] Furthermore, in the above, "If A is greater than or equal to B, proceed to step S1; if A is less than (lower than) B, proceed to step S2" may be read as "If A is greater than (higher than) B, proceed to step S1; if A is less than or equal to B, proceed to step S2." Conversely, "If A is greater than (higher than) B, proceed to step S1; if A is less than (lower than) B, proceed to step S2" may be read as "If A is greater than (higher than) B, proceed to step S1; if A is less than (lower than) B, proceed to step S2." Therefore, unless a contradiction arises, "greater than or equal to A" may be read as "greater than (higher; longer; more) than A," and "less than or equal to A" may be read as "less than (lower; shorter; fewer) than A." Furthermore, "greater than (higher; longer; more) than A" may be read as "greater than or equal to A," and "less than (lower; shorter; fewer) than A" may be read as "less than or equal to A."

[0113] The various controls described above may or may not be performed by a single piece of hardware (e.g., a processor or circuit). The entire device may be controlled by multiple pieces of hardware (e.g., multiple processors, multiple circuits, or a combination of one or more processors and one or more circuits) sharing the processing.

[0114] The above processor is a processor in the broad sense, and includes general-purpose processors and dedicated processors. General-purpose processors include, for example, CPUs (Central Processing Units), MPUs (Micro Processing Units), and DSPs (Digital Signal Processors). Dedicated processors include, for example, GPUs (Graphics Processing Units), ASICs (Application Specific Integrated Circuits), and PLDs (Programmable Logic Devices). Programmable logic devices include, for example, FPGAs (Field Programmable Gate Arrays) and CPLDs (Complex Programmable Logic Devices).

[0115] Although the embodiments of the present invention have been described in detail, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Furthermore, each of the above-described embodiments merely represents one embodiment of the present invention, and each embodiment can be combined as appropriate.

[0116] In the above-described embodiment, the present invention has been described as being applied to an HMD system (HMD), but the present invention is not limited to this example and can be applied to any electronic device or information processing device (information processing system) that can perform image processing on captured images. The electronic device or information processing device (information processing system) may be a computer, a smartphone, a tablet terminal, a digital camera, or a home appliance.

[0117] <Other embodiments> The present invention can also be realized by a process in which a program that realizes one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device read and execute the program, or by a circuit that realizes one or more functions.

[0118] The disclosure of the above embodiments includes the following configurations, methods, and programs. (Configuration 1) an acquisition means for acquiring a captured image of a real space including the first object and the second object; a generating means for generating a virtual object; a synthesis means for synthesizing the captured image and the virtual object to generate a synthesized image; a display control means for controlling a display means to display the composite image; a notification means for providing a notification to a user in accordance with a distance between the first object and the second object in a first case where the virtual object is superimposed in front of the second object in the space represented by the composite image; An information processing device comprising: (Configuration 2) the second object is an object that is located behind the first object and is closest to the first object among a plurality of objects in the real space; 2. The information processing device according to configuration 1, (Configuration 3) In the first case, the notification means performs the notification by controlling emphasis of the display of the first object in accordance with the distance between the first object and the second object. 3. The information processing device according to configuration 1 or 2. (Configuration 4) In the first case, the shorter the distance between the first object and the second object, the more emphasized the display of the first object is. 4. The information processing device according to configuration 3. (Configuration 5) In the first case, the notification means performs the notification by controlling a vibration intensity of the first device in accordance with a distance between the first object and the second object. 5. The information processing device according to any one of configurations 1 to 4. (Configuration 6) In the first case, the notification means controls the volume of the warning sound emitted from the second device in accordance with the distance between the first object and the second object. and then give the notification. 6. The information processing device according to any one of configurations 1 to 5. (Configuration 7) In the first case, the notification means performs the notification by controlling transparency of the virtual object in accordance with a distance between the first object and the second object. 7. The information processing device according to any one of configurations 1 to 6. (Configuration 8) the first object is a body part; 8. The information processing device according to any one of configurations 1 to 7. (Configuration 9) the first object is a hand; 9. The information processing device according to configuration 8. (Configuration 10) the first object is a part of an operating member; 8. The information processing device according to any one of configurations 1 to 7. (Configuration 11) In the first case, the notification means In a second case where the first object is superimposed in front of the virtual object in the space represented by the synthetic image, the notification is performed by controlling emphasis of the display of the first object according to a distance between the first object and the second object; In a third case where the virtual object is superimposed in front of the first object in the space represented by the composite image, a notification according to the distance between the first object and the second object is given to the user by a method other than controlling emphasis of the display of the first object. 11. The information processing device according to any one of configurations 1 to 10. (Configuration 12) In the third case, the notification means controls a vibration intensity of the first device, a volume of a warning sound emitted from the second device, or a transparency of the virtual object according to a distance between the first object and the second object. 12. The information processing device according to configuration 11. (method) an acquisition step of acquiring a captured image of a real space including the first object and the second object; a generation step of generating a virtual object; a synthesis step of synthesizing the captured image and the virtual object to generate a synthetic image; a display control step of controlling a display means to display the composite image; a notification step of notifying a user of a distance between the first object and the second object in a first case where the virtual object is superimposed in front of the second object in the space represented by the composite image; An information processing method comprising: (program) 13. A program for causing a computer to function as each means of the information processing device according to any one of configurations 1 to 12. [Explanation of symbols]

[0119] 202: Image sensor, 209: CG generation unit, 204: Composition unit, 206:Display section

Claims

1. an acquisition means for acquiring a captured image of a real space including the first object and the second object; a generating means for generating a virtual object; a synthesis means for synthesizing the captured image and the virtual object to generate a synthesized image; a display control means for controlling a display means to display the composite image; a notification means for providing a notification to a user in accordance with a distance between the first object and the second object in a first case where the virtual object is superimposed in front of the second object in the space represented by the composite image; An information processing device comprising:

2. the second object is an object that is located behind the first object and is closest to the first object among a plurality of objects in the real space; 2. The information processing apparatus according to claim 1, wherein:

3. In the first case, the notification means performs the notification by controlling emphasis of the display of the first object in accordance with the distance between the first object and the second object.

3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

4. In the first case, the shorter the distance between the first object and the second object, the more emphasized the display of the first object is.

4. The information processing apparatus according to claim 3,

5. In the first case, the notification means performs the notification by controlling a vibration intensity of the first device in accordance with a distance between the first object and the second object.

3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

6. In the first case, the notification means performs the notification by controlling a volume of an alarm sound emitted from the second device in accordance with a distance between the first object and the second object.

3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

7. In the first case, the notification means performs the notification by controlling transparency of the virtual object in accordance with a distance between the first object and the second object.

3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

8. the first object is a body part; 3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

9. the first object is a hand; 9. The information processing apparatus according to claim 8,

10. the first object is a part of an operation member; 3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

11. In the first case, the notification means in a second case where the first object is superimposed in front of the virtual object in the space represented by the composite image, the notification is performed by controlling emphasis of the display of the first object according to a distance between the first object and the second object; In a third case where the virtual object is superimposed in front of the first object in the space represented by the composite image, a notification according to the distance between the first object and the second object is given to the user by a method other than controlling emphasis of the display of the first object.

3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

12. In the third case, the notification means controls a vibration intensity of the first device, a volume of a warning sound emitted from the second device, or a transparency of the virtual object according to a distance between the first object and the second object.

12. The information processing apparatus according to claim 11,

13. an acquisition step of acquiring a captured image of a real space including the first object and the second object; a generation step of generating a virtual object; a synthesis step of synthesizing the captured image and the virtual object to generate a synthetic image; a display control step of controlling a display means to display the composite image; a notification step of notifying a user of a distance between the first object and the second object in a first case where the virtual object is superimposed in front of the second object in the space represented by the composite image; An information processing method comprising:

14. A program for causing a computer to function as each of the means of the information processing device according to claim 1 or 2.

Images