Information processing system, information processing method, and program

Abstract

To achieve interactive control (operation) with respect to CG objects with excellent operability.SOLUTION: An information processing system controls a display device to allow a user to visually recognize a specific space in which a virtual object is placed in a real space including a real object, the information processing system including: acquisition means for acquiring an image of the virtual object whose form is controlled in accordance with a degree of proximity between the real object and the virtual object in a first case in which it is determined that a distance between the real object and the virtual object in the specific space is shorter than a prescribed distance; display control means for controlling the display device to display an image of the virtual object whose form is controlled in the first case; and notification means for issuing a specific notification indicating that the real object and the virtual object have come into proximity before the image of the virtual object whose form is controlled is displayed on the display device in the first case.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to an information processing system, an information processing method, and a program.

Background Art

[0002] Conventionally, there has been a technology of superimposing a CG image of a virtual world (hereinafter referred to as "virtual space") generated by a computer on an image of the real world (hereinafter referred to as "real space") and displaying it on a head-mounted display (HMD) or a smartphone. For example, such technologies include Mixed Reality (MR) technology and Augmented Reality (AR) technology.

[0003] In particular, in MR technology, an image in which an image of the real space and a CG image of the virtual space are superimposed is displayed on an HMD worn by a user. Therefore, the user can feel as if a virtual space exists in front of the user's eyes, so the sense of presence is very high. Also, various methods have been considered to enable interactive operations on the CG image of the virtual space visible in front of the user's eyes.

[0004] Patent Document 1 discloses a technique of stopping the movement of a CG object that is operated and moved in a virtual space when detecting contact between the CG object and another CG. Patent Document 2 discloses a technique of uploading a real image captured by an HMD of the real space and input data of a controller to a game cloud server and streaming and displaying an augmented reality scene in which the real image and a CG image are superimposed on the HMD.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, in the technology described in Patent Document 1, the rendering method and the calculation location of CG are not disclosed in detail. In the technology described in Patent Document 2, the input data of the controller is transmitted to the cloud server together with the image, and the device in the cloud performs the processing. Therefore, if there is a delay during communication with the cloud, the response to the input operation deteriorates. Thus, in the prior art, it has not been possible to realize interactive control (operation) on a CG object with high operability.

[0007] Therefore, an object of the present invention is to realize interactive control (operation) on a CG object with high operability.

Means for Solving the Problems

[0008] One aspect of the present invention is an information processing system that controls a display device so as to allow a user to visually recognize a specific space in which virtual objects are arranged in a real space including real objects, acquisition means for acquiring an image of the virtual object whose form is controlled according to the proximity between the real object and the virtual object when it is determined in a first case that the distance between the real object and the virtual object in the specific space is shorter than a predetermined distance; in the first case, the image of the virtual object whose form is controlled according to the proximity display control means for controlling the display device to display; in the first case, notification means for performing a specific notification indicating that the real object and the virtual object are close before the image of the virtual object whose form is controlled according to the proximity is displayed on the display device; An information processing system characterized by comprising.

[0009] One aspect of the present invention is An information processing method for controlling a display device to allow a user to visually recognize a specific space in which virtual objects are arranged in a real space including real objects, In a first case where it is determined that the distance between the real object and the virtual object in the specific space is shorter than a predetermined distance, an acquisition step of acquiring an image of the virtual object whose form is controlled according to the degree of proximity between the real object and the virtual object; In the first case, a display control step of controlling the display device to display an image of the virtual object whose form is controlled according to the degree of proximity; In the first case, a notification step of performing a specific notification indicating that the real object and the virtual object have come close before the image of the virtual object whose form is controlled according to the degree of proximity is displayed on the display device; An information processing method characterized by comprising:

Advantages of the Invention

[0010] According to the present invention, interactive control (operation) of a CG object can be realized with high operability.

Brief Description of the Drawings

[0011]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Modes for Carrying Out the Invention

[0012] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings, taking an HMD as an example.

[0013] <Embodiment 1> Hereinafter, an information processing system according to Embodiment 1 will be described. First, the overall configuration of the information processing system will be described with reference to FIGS. 1 and 2. The information processing system is a system that allows a user to visually recognize a composite reality space in which CG objects are arranged in a reality space including real objects such as hands.

[0014] The information processing system includes a first information processing device 1000 and a second information processing device 1100. The first information processing device 1000 includes an HMD 1001, an image processing device 1010, and a controller 1020.

[0015] The HMD 1001 is a display device worn on the user's head.

[0016] As shown in FIG. 1, the image processing device 1010 is connected to the HMD 1001 by wire or wirelessly. Alternatively, the image processing device 1010 may be integrated with the HMD 1001. The image processing device 1010 communicates with the HMD 1001 and transmits a CG image (virtual image), a composite image (an image obtained by compositing CG and a background), etc. to the HMD 1001. Further, the image processing device 1010 is connected to the second information processing device 1100 via a communication network.

[0017] The controller 1020 is an operation member held by the user's hand. The controller 1020 can give instructions according to the user's operations to the image processing device 1010 and the HMD 1001.

[0018] The second information processing device 1100 generates a CG image (virtual image) to be displayed on the HMD 1001 by rendering a CG object (virtual object) based on virtual CG data in a three-dimensional space. For this purpose, a device with higher processing performance than the first information processing device 1000 (for example, a high-performance server machine or cloud computing) is used for the second information processing device 1100.

[0019] The CG image generated by the second information processing device 1100 is sent to the image processing device 1010 together with the distance information (depth information) of the CG image. The CG image is displayed on the HMD 1001 after synthesis processing based on the distance information. Hereinafter, the distance information is an image indicating the depth (distance) to the subject for each pixel. The distance information of the CG image is information indicating the depth (distance) to the CG object (virtual object) imaged by the pixel for each pixel of the CG image.

[0020] FIG. 2 is a block diagram of the information processing system according to Embodiment 1.

[0021] The first information processing device 1000 and the second information processing device 1100 communicate data with each other. The communication between the first information processing device 1000 and the second information processing device 1100 may be realized via another information processing device. For example, the communication between the two devices may be realized via "an information processing device with lower computing performance than the first information processing device 1000 and the second information processing device 1100", a router for communication, or a communication access point.

[0022] The first information processing device 1000 includes an HMD 1001, an image processing device 1010, and a controller 1020. The image processing device 1010 and the controller 1020 may be integrated inside the housing of the HMD 1001.

[0023] The HMD 1001 includes an imaging unit 1002, an image display unit 1003, a distance information acquisition unit 1004, an attitude sensor 1005, and an HMD control unit (not shown) that controls all of these.

[0024] The imaging unit 1002 includes two cameras. One of the two cameras is arranged to be located near the position of the user's left eye when the HMD 1001 is worn. The other of the two cameras is arranged to be located near the position of the user's right eye when the HMD 1001 is worn. According to the image (hereinafter referred to as "captured image") captured by the imaging unit 1002 of the surroundings, the position of the HMD 1001 can be calculated by the SLAM (Simultaneous Localization and Mapping) technology.

[0025] The image display unit 1003 includes, for example, an organic EL panel or the like. The image display unit 1003 is configured such that an optical system is arranged in front of each of the user's eyes. The image display unit 1003 displays the composite image acquired from the image processing device 1010. Therefore, by wearing the HMD 1001, the user can visually recognize the composite image (the composite reality space indicated by the composite image) displayed on the image display unit 1003. And the user can experience various augmented realities in which the CG data read from the second information processing device 1100 is fused with the real space. be possible.

[0026] Note that it is also possible to use a device using a semi-transmissive half mirror for the image display unit 1003. In this case, a technique (a technique generally called AR) of superimposing and observing a CG object directly on the real space that can be seen through is used. Also, a technique (a technique generally called VR) of displaying a complete virtual space on a display device and allowing the user to visually recognize it without using the imaging unit 1002 may be used.

[0027] The distance information acquisition unit 1004 has a distance measurement sensor such as a TOF (Time of Flight) sensor. The distance information acquisition unit 1004 acquires the distance information of the captured image by measuring the distance from the HMD 1001 to the subject. The distance information acquisition unit 1004 may use two cameras included in the imaging unit 1002 as a stereo camera and acquire the distance information of the captured image by stereo camera distance measurement.

[0028] The attitude sensor 1005 is an inertial measurement device capable of acquiring the attitude information of the user wearing the HMD 1001. The attitude sensor 1005 has an IMU (Inertial Measurement Unit).

[0029] The image processing device 1010 includes a display image generation unit 1011, a distance determination unit 1012, a communication unit 1013, and a CG data acquisition unit 1014.

[0030] The display image generation unit 1011 acquires the captured image (background image), the attitude information of the user, and the distance information of the captured image from the HMD 1001. The display image generation unit 1011 performs image processing on the captured image to cancel the aberration of the optical systems of the imaging unit 1002 and the image display unit 1003.

[0031] In addition, the display image generation unit 1011 synthesizes the captured image and the CG image of an arbitrary CG object based on the attitude information of the user, the distance information of the captured image, and the distance image of the CG image. Further, the display image generation unit 1011 performs display control of the image display unit 1003 (HMD 1001) so as to display the synthesized image obtained by synthesizing the captured image and the CG image. The display image generation unit 1011 can also change (control) the form (position, orientation, size, etc.) of the CG object based on the attitude information of the user and the distance information of the CG image.

[0032] The display image generation unit 1011 compares the distance information of the captured image and the distance information of the CG image, and synthesizes the captured image and the CG image so that an object with a smaller depth (distance) is displayed in the foreground. As a result, a composite image can be generated in which a CG object that does not exist in the real space appears as if it exists in the real space.

[0033] The distance determination unit 1012 compares the position of a real object such as a hand or the controller 1020 with the position indicated by the position information of the CG object acquired in advance from the second information processing device 1100. Then, based on the difference between the two positions, the distance determination unit 1012 calculates the shortest distance between the real object and the CG object. When the shortest distance between the real object and the CG object is equal to or less than a specific threshold value, the distance determination unit 1012 determines that the real object and the CG object are close to each other. The distance determination unit 1012 notifies at least one of the display image generation unit 1011 and the notification device 1021 of information indicating that the real object and the CG object are close to each other (hereinafter referred to as "proximity information"). The display image generation unit 1011 generates a display item or the like that conveys that the real object and the CG object are close to each other. Then, the display image generation unit 1011 controls the image display unit 1003 to display the generated display item.

[0034] Even when the user tries to touch the CG object, the CG object only exists in the virtual three-dimensional space (composite reality space) represented by the computer. Therefore, it is necessary to reproduce a real object (hand or controller 1020) existing in the real space in the virtual three-dimensional space and measure the distance in that three-dimensional space. For this reason, the imaging unit 1002 and the distance information acquisition unit 1004 measure the position of the user's hand or the operation controller 1020. Here, in the second information processing device 1100, based on the measured position information, a hand is reproduced in the three-dimensional space reconstructed by the computer, and it is conceivable that proximity determination (determination of whether the real object and the CG object are close to each other) is performed. On the other hand, in Embodiment 1, the distance determination unit 1012 performs proximity determination. Since the distance determination unit 1012 exists inside the first information processing device 1000 that is closer to the user, the result of the proximity determination can be quickly notified to the user.

[0035] The communication unit 1013 realizes communication with the second information processing device 1100. The communication unit 1013 acquires a CG image, distance information of the CG image, and position information of the CG object from the second information processing device 1100.

[0036] The CG data acquisition unit 1014 collectively acquires the CG image and the distance information of the CG image from the communication unit 1013.

[0037] The controller 1020 has a notification device 1021.

[0038] The notification device 1021 notifies the user so that the user can perceive by touch, hearing, or vision that the hand and the CG object are close. The notification device 1021 includes a speaker, a haptics device, or a light-emitting LED, etc. The notification device 1021 notifies the user that the hand and the CG object are close by a speaker, a haptics device, or a light-emitting LED, etc.

[0039] For example, when the notification device 1021 acquires proximity information, it emits a notification sound such as an alarm through a speaker. Also, when the notification device 1021 acquires proximity information, it may vibrate a haptics device so as to reproduce tactile information such as the firmness of the CG object, or simply vibrate the haptics device with a notification vibration function. When the notification device 1021 acquires proximity information, it may control light such as the blinking of a light-emitting LED. Note that the notification device 1021 may be built into the HMD 1001 instead of a controller held by a hand or body of a user performing an MR experience.

[0040] The second information processing apparatus 1100 includes a data storage unit 1101, a rendering unit 1102, and a communication unit 1103.

[0041] The data storage unit 1101 stores contents such as a model of a CG object (CG model) and other information necessary for CG rendering.

[0042] The rendering unit 1102 generates a CG image by rendering a CG object based on the information stored in the data storage unit 1101 (and the user's posture information). Also, the rendering unit 1102 can render a CG object in a form corresponding to the proximity degree between the hand and the CG object. Thereby, the rendering unit 1102 generates a CG image considering the proximity between the hand and the CG object.

[0043] The communication unit 1103 realizes communication with the first information processing apparatus 1000.

[0044] Referring to the flowchart of FIG. 3, the processing when a real object is close to a CG object will be described. This will be described below. In the following, the processing when the hand is close to the CG object will be described, but it may be the processing when any arbitrary real object other than the hand is close to the CG object. The processing of the flowchart in FIG. 3 starts from a state where the arrangement of the CG object is completed in the composite reality space (virtual space) and the CG image is displayed on the HMD 1001.

[0045] In step S2010, the distance information acquisition unit 1004 detects the position of the hand in the real space using the distance measurement sensor. Further, the distance information acquisition unit 1004 detects the depth of the hand in the real space using the distance measurement sensor and acquires the distance information of the captured image. The position and depth of the hand may be detected by the image processing device 1010 using a stereo camera without using the distance measurement sensor.

[0046] In step S2020, the CG data acquisition unit 1014 acquires the position information of the CG object in the virtual space and the distance information of the CG image. The CG object exists as data in a virtual three-dimensional space in the second information processing device 1100. Therefore, the CG data acquisition unit 1014 acquires the position information of the CG object and the distance information of the CG image together with the data of the CG image and outputs them to the display image generation unit 1011.

[0047] In step S2030, the distance determination unit 1012 calculates the distance (shortest distance) between the hand and the CG object in the composite reality space based on the position information of the hand and the position information of the CG object. Hereinafter, the distance between the hand and the CG object in the composite reality space will be referred to as the "distance between objects".

[0048] In step S2040, the distance determination unit 1012 determines whether the distance between objects is equal to or less than a specific threshold. If it is determined that the distance between objects is greater than the specific threshold, the process returns to step S2010. If it is determined that the distance between objects is equal to or less than the specific threshold, the process proceeds to step S2050.

[0049] The specific threshold value may be any distance, but in Embodiment 1, it is "0". In this case, if the distance between the objects is greater than 0, it can be said that the hand and the CG object are separated. On the other hand, if the distance between the objects is 0 or a negative value less than 0, it can be said that the hand and the CG object are in contact.

[0050] In step S2050, the distance determination unit 1012 determines that the hand and the CG object are close in the composite reality space.

[0051] In step S2060, the notification device 1021 notifies the user that the hand and the CG object are close so that the user can perceive it by touch, hearing, or vision. Note that the image display unit 1003 may display an image (display item) indicating that the hand and the CG object are close. The notification that the hand and the CG object are close is called a "proximity notification".

[0052] In step S2070, the distance determination unit 1012 notifies the second information processing device 1100 (cloud computer) of information on the degree of proximity between the hand and the CG object almost simultaneously with the processing in step S2060. The information on the degree of proximity is, for example, information indicating the degree to which the hand has entered the CG object or the size of the range of the hand that has entered the CG object.

[0053] In step S2080, the rendering unit 1102 controls the form of the CG object to be rendered according to the degree of proximity between the hand and the CG object. The rendering unit 1102 renders the CG object whose form has been controlled according to the degree of proximity to generate a CG image. For example, when it is determined that a finger of the hand has entered the CG object, the rendering unit 1102 controls the shape of the CG object to be a concave shape according to the shape of the hand.

[0054] In step S2090, the rendering unit 1102 transmits the rendered CG image together with the distance information of the CG image to the first information processing device 1000.

[0055] In step S2100, the display image generation unit 1011 compares the distance information of the captured image and the distance information of the CG image, and synthesizes the captured image and the CG image so as not to conflict in the depth direction (front-back relationship) to generate a synthesized image. The synthesized image is displayed on the image display unit 1003 of the HMD 1001. At this time, when the HMD 1001 is performing proximity notification to the user, the proximity notification may be terminated at this timing to make the CG object with proximity reflected easier to see. On the other hand, the notification device 1021 may continue the proximity notification.

[0056] After step S2100, the notification device 1021 may terminate the proximity notification after a certain period of time has elapsed since the start of the proximity notification. Also, the notification device 1021 may terminate the proximity notification when the distance between the hand and the CG object is farther than the threshold value. Note that the position of the hand in the synthesized image is updated according to the passage of time. Also, at each time, the display of the CG image is updated according to the distance between the hand and the CG object.

[0057] (First example of proximity notification) FIGS. 4A to 4C are diagrams for explaining a first example of the proximity notification process in step S2060. FIGS. 4A to 4C are part of the synthesized image displayed on the HMD 1001 (image display unit 1003). FIGS. 4A to 4C show the changes in the display of the CG image and the hand in the synthesized image in chronological order.

[0058] The CG image 4000 is a CG image obtained by rendering a spherical CG object in the virtual space. The CG image 4000 is displayed in front of the captured image in the synthesized image synthesized with the captured image.

[0059] The hand 4010 is the hand of the user in the real space who is trying to touch the CG object. In the composite image, the real hand may be displayed as it is, or a CG object of the hand may be displayed at the same position as the hand.

[0060] Figure 4A shows the state before the hand 4010 approaches the CG object. That is, the distance between the CG object and the hand 4010 is separated by a certain distance or more, and the hand 4010 is not close to the CG object.

[0061] Figure 4B shows the state where the hand 4010 is brought closer to the CG object from the state of Figure 4A. In Figure 4B, the hand 4010 and the CG object are close to each other. A part of the hand 4010 is displayed overlapping the CG image 4000. In this example, since the hand 4010 is in front of the CG object, the hand 4010 is displayed overlapping the CG image 4000, but the CG image 4000 itself is not controlled according to the proximity. However, since it is determined that the hand 4010 and the CG object are close to each other, as a proximity notification, vibration is transmitted to the controller 1020 worn on the user's wrist by the notification device 1021. The wavy line in Figure 4B represents that the hand 4010 is vibrating.

[0062] Figure 4C shows the state where more time has passed from the state of Figure 4B. In the CG image 4020, the CG object is partially recessed according to the proximity of the hand and is rendered in the recessed shape. The CG image 4020 shown in Figure 4C is delayed by the time required for rendering and the time required for data communication between the first information processing device 1000 and the second information processing device 1100, and then displayed.

[0063] (Second example of proximity notification) Figs. 5A to 5C are diagrams for explaining a second example of the proximity notification process in step S2060. Figs. 5A to 5C are part of the composite image displayed on the HMD 1001. Figs. 5A to 5C show the changes in the CG image and the hand display in chronological order. In Figs. 5A to 5C, instead of Fig. 4B in Figs. 4A to 4C, a transition is made to the state of Fig. 5B.

[0064] Fig. 5B shows a state where the hand 4010 and the CG object are approaching. Fig. 5B shows an example of the case where a display change is made as a proximity notification from the state shown in Fig. 5A. The CG image 4030 is a CG image pre-stored in the first information processing device 1000 to notify the proximity of the hand 4010 and the CG object. The image display unit 1003 notifies the proximity to the user by displaying the CG image 4030 at or near the position where the hand and the CG object are approaching. Ideally, when it is determined that the hand 4010 and the CG object are approaching, it is desirable that the CG image 4020 shown in Fig. 5C be immediately displayed. However, at the time of the proximity notification shown in Fig. 5B, since it is immediately after the proximity is determined, the CG image 4020 has not been generated yet. For this reason, the pre-prepared CG image 4030 is superimposed on the CG image 4000 until the CG image 4020 becomes displayable. Note that in the proximity notification, a notification such as vibration or sound may be further performed by the notification device 1021 to make it easier for the user to recognize the proximity.

[0065] (Third example of proximity notification) Figs. 6A to 6C are diagrams for explaining a third example of the proximity notification process in step S2060. Figs. 6A to 6C show part of the composite image displayed on the HMD 1001. In Figs. 6A to 6C, instead of Fig. 4B in Figs. 4A to 4C, a transition is made to the state of Fig. 6B.

[0066] FIG. 6B shows a state where the hand 4010 and the CG object are in proximity from the state shown in FIG. 6A. FIG. 6B is an example of a case where a display change is made as a proximity notification. The CG image 4040 is an image for proximity notification. Specifically, the CG image 4040 is an image obtained by processing the CG image 4000 by the display image generation unit 1011 of the first information processing apparatus 1000. Since it takes time to generate the CG image 4020 by the second information processing apparatus 1100, the CG image 4040 is generated based on the already generated CG image 4000 at the timing when the proximity determination is made.

[0067] The CG image 4040 is generated by simple processing (processing with a short image processing time). Specifically, the shape of the CG object in the CG image 4040 is the same as the shape of the CG object in the CG image 4000. On the other hand, in the CG image 4040, the color of the CG object is changed to a single red color from the CG image 4000, or the brightness (luminance) or saturation of the CG object is changed, or the like. Alternatively, the transmittance of the CG object may be changed in the CG image 4040 from the CG image 4000. For the generation of the CG image 4040, high-definition image processing such as re-rendering of the CG object is not performed. By this, it is possible to immediately change the display and give a proximity notification when determining the proximity of the hand 4010 and the CG object.

[0068] In FIG. 6B, the CG image 4040 is highlighted. Similarly, the hand 4010, which is a real object, may be highlighted to give a proximity notification. For example, the image display unit 1003 may display a CG image of the hand 4010 so as to overlap the position of the hand 4010. Then, the image display unit 1003 may change the color, brightness, or transmittance of the CG image of the hand 4010 at the time of proximity. Alternatively, the image display unit 1003 may continue to display the CG image of the hand 4010 while maintaining the display position (the display position of the hand in the composite image) of the CG image of the hand 4010 at the time of proximity. shown position (the display position of the hand in the composite image) of the hand 4010 to continue to display the CG image of the hand 4010.

[0069] In many cases, the computing performance of the rendering unit 1102 of the second information processing apparatus 1100 is higher than that of the display image generation unit 1011 of the first information processing apparatus 1000. For this reason, the second information processing apparatus 1100 performs computationally intensive operations such as rendering of CG objects. However, when using the Internet line for communication between the first information processing apparatus 1000 and the second information processing apparatus 1100, data communication may be delayed due to congestion of the communication line or the like. Therefore, in order to improve the response to the user, the first information processing apparatus 1000 may determine whether or not the real object and the CG object have approached each other to a predetermined distance.

[0070] On the other hand, when the computing performance of the first information processing apparatus 1000 is low, there is also a demand to reduce the operations performed by the first information processing apparatus 1000 as much as possible. Therefore, in a situation where high speed is not required for the response of the proximity determination and when the communication delay between the first information processing apparatus 1000 and the second information processing apparatus 1100 is small, the second information processing apparatus 1100 may perform the proximity determination. Thereby, the computational load on the first information processing apparatus 1000 can be reduced. In this case, before the generation of the CG image according to the result of the proximity determination is completed, the second information processing apparatus 1100 notifies the first information processing apparatus 1000 of the result of the proximity determination. Then, the notification device 1021 or the like can notify the user of the proximity between the hand and the CG object without waiting for the completion of the generation of the CG image according to the result of the proximity determination.

[0071] As described above, the information processing system may switch the information processing apparatus that performs the proximity determination between the first information processing apparatus 1000 and the second information processing apparatus 1100 according to the user's situation or the status of the communication line.

[0072] In the first embodiment, the real object whose proximity to the CG object is determined is described as the hand. However, the real object may be any object such as the controller 1020 as long as its position in the real space can be detected.

[0073] In Embodiment 1, when the hand is close to the CG object, the second information processing device 1100 generates a CG image by rendering according to the degree of proximity. Also, the first information processing device 1000 controls the HMD 1001 or the controller 1020 to notify the user of the proximity without waiting for the generation of the CG image according to the degree of proximity to be completed. As a result, the user can grasp the proximity between the CG object and the real object in real time. Therefore, the user can perform interactive control (operation) on the CG object with high operability.

[0074] <Embodiment 2> With reference to the flowchart of FIG. 7, the information processing system according to Embodiment 2 will be described.

[0075] In Embodiment 2, the information processing system performs proximity determination in the first information processing device 1000 by a simpler method. Note that the configuration of the information processing system according to Embodiment 2 is the same as the configuration of the information processing system according to Embodiment 1, as shown in FIG. 2.

[0076] With reference to the flowchart of FIG. 7, the processing when the real object is close to the CG object will be described.

[0077] In step S3010, the distance information acquisition unit 1004 detects the position and depth of the hand in the real space by measurement using a camera or a distance measurement sensor. Also, the distance information acquisition unit 1004 acquires distance information (distance information of the captured image) indicating the depth to the subject for each pixel of the captured image (display area of the image display unit 1003). At this time, the distance information acquisition unit 1004 sets a finite value indicating the distance to the hand as the depth for the pixels in which the hand is captured, and sets a specific value V1 (for example, 0 or the maximum value) as the depth for the other pixels. As a result, the first information processing device 1000 can determine the pixels in which the hand is captured in the captured image only by referring to the distance information of the captured image.

[0078] In step S3020, the CG data acquisition unit 1014 acquires the position of the CG object and the distance information of the CG image in the virtual space. The CG data acquisition unit 1014 sets the distance information of the CG image by the same method as in step S3010. Hereinafter, a set of pixels indicating the distance to the hand or the CG object without having a specific value V1 is referred to as a "distance pixel group".

[0079] In step S3030, the distance determination unit 1012 determines whether the distance pixel group of the captured image overlaps with the distance pixel group of the CG image. If it is determined that the two distance pixel groups do not overlap, the process returns to step S3010. If it is determined that the two distance pixel groups overlap, the process proceeds to step S3040.

[0080] In step S3040, the distance determination unit 1012 determines the distance between the hand and the CG object (object - to - object distance). If it is determined that the object - to - object distance is longer than the threshold value, the process returns to step S3010. If it is determined that the object - to - object distance is equal to or less than the threshold value, the process proceeds to step S3050.

[0081] Specifically, the distance determination unit 1012 focuses on two pixels that overlap each other among the distance pixel group of the captured image and the distance pixel group of the CG image, and compares the distances set for the two pixels. Then, the distance determination unit 1012 regards the difference between the distances set for the two pixels as the object - to - object distance, and determines whether the object - to - object distance is equal to or less than the threshold value. Note that the distance determination unit 1012 may regard the difference between the average value of the distances set for the distance pixel group of the captured image and the average value of the distances set for the distance pixel group of the CG image as the object - to - object distance, and determine whether the object - to - object distance is equal to or less than the threshold value. The threshold value may be set to 0, for example, or may be set to any value greater than 0.

[0082] In step S3050, the distance determination unit 1012 determines that the hand and the CG object are in proximity.

[0083] In step S3060, the notification device 1021 notifies the user that the hand and the CG object are close to each other. Different from Embodiment 1, the image display unit 1003 may notify the user that the hand and the CG object are close to each other by emphasizing and displaying only the range that overlaps with the distance pixel group of the hand among the distance pixel groups of the CG object.

[0084] In steps S3070 to S3100, the same processing as steps S2070 to S2100 shown in the flowchart of FIG. 3 is executed. Here, in step S3070, the distance determination unit 1012 may notify only the information of the pixels that overlap with the distance pixel group of the captured image among the distance pixel groups of the CG image to the second information processing device 1100. Then, the rendering unit 1102 uses the information of the pixels that overlap with the distance pixel group of the captured image among the distance pixel groups of the CG image as the proximity information to render the CG object. In this case, the position and depth information of a real object such as a hand can be omitted from the data exchange between the two devices. That is, the data exchange between the first information processing device 1000 and the second information processing device 1100 can be reduced.

[0085] According to Embodiment 2, the information processing system can perform proximity notification to the user by a simpler method than Embodiment 1.

[0086] In each of the above embodiments, an example in which a composite image obtained by combining a captured image and a CG image is displayed on a video see-through type HMD has been described. However, the HMD may be an optical see-through type HMD through which the user can visually recognize the outside through the display surface of the HMD. In this case, the first information processing device transmits the CG image to the HMD as it is without generating a composite image. Then, the HMD displays the CG image. As a result, the user can visually recognize the real space through the display surface of the HMD and also visually recognize the CG object in the CG image. That is, the user can visually recognize a space in which the CG object is arranged in the real space.

[0087] As described above, the present invention has been described in detail based on its preferred embodiments. However, the present invention is not limited to these specific embodiments, and various forms within the scope not departing from the gist of the present invention are also included in the present invention. Some of the above-described embodiments may be appropriately combined.

[0088] Also, in the above, "When A is greater than or equal to B, proceed to step S1, and when A is less than B, proceed to step S2" may be read as "When A is greater than B, proceed to step S1, and when A is less than or equal to B, proceed to step S2". Conversely, "When A is greater than B, proceed to step S1, and when A is less than or equal to B, proceed to step S2" may be read as "When A is greater than or equal to B, proceed to step S1, and when A is less than B, proceed to step S2". Therefore, as long as there is no contradiction, "A or more" may be read as "greater than (higher; longer; more) than A", and "A or less" may be read as "less than (lower; shorter; less) than A". And "greater than (higher; longer; more) than A" may be read as "A or more", and "less than (lower; shorter; less) than A" may be read as "A or less".

[0089] Note that each functional unit of each of the above embodiments (each modification) may be individual hardware, or not. The functions of two or more functional units may be realized by common hardware. Each of the multiple functions of one functional unit may be realized by individual hardware. Two or more functions of one functional unit may be realized by common hardware. Also, each functional unit may be realized by hardware such as ASIC, FPGA, DSP, or not. For example, the device may have a processor and a memory (storage medium) in which a control program is stored. And the functions of at least some of the functional units of the device may be realized by the processor reading out and executing the control program from the memory.

[0090] (Other Embodiments) The present invention can also be realized by supplying a program that implements one or more functions of the above-described embodiment to a system or apparatus via a network or a storage medium, and causing one or more processors in a computer of the system or apparatus to read and execute the program. It can also be realized by a circuit (for example, ASIC) that implements one or more functions.

[0091] The disclosure of the above embodiment includes the following configurations, methods, and programs. (Configuration 1) An information processing system that controls a display device so as to visually present to a user a specific space in which virtual objects are arranged in a real space including real objects, wherein when a first case is determined in which a distance between the real object and the virtual object in the specific space is shorter than a predetermined distance, acquisition means for acquiring an image of the virtual object whose form is controlled according to the degree of proximity between the real object and the virtual object; display control means for controlling the display device to display the image of the virtual object whose form is controlled according to the degree of proximity in the first case; notification means for performing a specific notification indicating that the real object and the virtual object are close to each other before the image of the virtual object whose form is controlled according to the degree of proximity is displayed on the display device in the first case; An information processing system characterized by comprising the above. (Configuration 2) The notification means performs the specific notification so that the user can perceive by touch, hearing, or vision that the real object and the virtual object are close to each other. The information processing system according to Configuration 1, characterized by the above. (Configuration 3) The notification means performs the specific notification by vibration of a specific member. The information processing system according to Configuration 2, characterized by the above. (Configuration 4) The notification means performs the specific notification by emitting a specific sound. The information processing system according to Configuration 2 or 3, characterized in that... (Configuration 5) The notification means performs the specific notification by controlling the display device to display a specific image. The information processing system according to any one of Configurations 2 to 4, characterized in that... (Configuration 6) The notification means performs the specific notification by controlling the display device to display a first image. The first image is an image obtained by changing the color, brightness, or transmittance of the image of the virtual object before the form is controlled according to the proximity degree. The information processing system according to any one of Configurations 2 to 5, characterized in that... (Configuration 7) The notification means performs the specific notification by controlling the display device to perform a display that emphasizes the position of the real object. The information processing system according to any one of Configurations 2 to 6, characterized in that... (Configuration 8) The display device displays a virtual image of the real object. The notification means performs the specific notification by controlling the display device to maintain the display position of the virtual image of the real object. The information processing system according to any one of Configurations 2 to 6, characterized in that... (Configuration 9) It further includes a first information processing device and a second information processing device. The first information processing device includes the acquisition means, the display control means, and the notification means. The second information processing device includes a generation means for generating an image of the virtual object whose form is controlled according to the proximity degree in the first case. The information processing system according to any one of Configurations 1 to 8, characterized in that... (Configuration 10) In the second case, the first information processing apparatus determines whether the distance between the real object and the virtual object in the specific space is shorter than the predetermined distance. In the third case, the second information processing apparatus determines whether the distance between the real object and the virtual object in the specific space is shorter than the predetermined distance. The information processing system according to Configuration 9, characterized in that. (Method) An information processing method for controlling a display device so as to allow a user to visually recognize a specific space in which a virtual object is arranged in a real space including a real object, In a first case where it is determined that the distance between the real object and the virtual object in the specific space is shorter than a predetermined distance, an acquisition step of acquiring an image of the virtual object whose form is controlled according to the degree of proximity between the real object and the virtual object; In the first case, a display control step of controlling the display device to display an image of the virtual object whose form is controlled according to the degree of proximity; In the first case, a notification step of performing a specific notification indicating that the real object and the virtual object are close to each other before the image of the virtual object whose form is controlled according to the degree of proximity is displayed on the display device; An information processing method, characterized by comprising. (Program) A program for causing a computer to function as each means of the information processing system according to any one of Configurations 1 to 10.

Explanation of Reference Numerals

[0092] 1000: First information processing apparatus, 1100: Second information processing apparatus, 1001: HMD, 1011: Display image generation unit, 1012: Distance determination unit, 1014: CG data acquisition unit, 1020: Notification device

Claims

1. An information processing system for controlling a display device so as to allow a user to visually recognize a specific space in which a virtual object is arranged in a real space including a real object, comprising: an acquisition means for acquiring an image of the virtual object, the form of which is controlled in accordance with a degree of proximity between the real object and the virtual object, in a first case in which it is determined that the distance between the real object and the virtual object in the specific space is shorter than a predetermined distance; a display control means for controlling the display device to display an image of the virtual object whose form is controlled in accordance with the degree of proximity in the first case; In the first case, a notification means for performing a specific notification indicating that the real object and the virtual object have come close to each other before an image of the virtual object whose form has been controlled in accordance with the degree of proximity is displayed on the display device; An information processing system comprising:

2. the notification means issues the specific notification so that the user can perceive the proximity of the real object and the virtual object by touch, hearing, or vision.

2. The information processing system according to claim 1 .

3. The notification means performs the specific notification by vibration of a specific member.

3. The information processing system according to claim 2.

4. The notification means issues the specific notification by emitting a specific sound.

3. The information processing system according to claim 2.

5. the notification means performs the specific notification by controlling the display device to display a specific image.

3. The information processing system according to claim 2.

6. the notification means performs the specific notification by controlling the display device to display a first image; The first image is an image obtained by changing a color, a brightness, or a transmittance of an image of the virtual object before the form is controlled according to the proximity degree.

3. The information processing system according to claim 2.

7. the notification means performs the specific notification by controlling the display device to perform a display that emphasizes the position of the real object.

3. The information processing system according to claim 2.

8. the display device displays a virtual image of the real object; the notification means performs the specific notification by controlling the display device to maintain a display position of the virtual image of the real object.

3. The information processing system according to claim 2.

9. Further comprising a first information processing device and a second information processing device, the first information processing device includes the acquisition means, the display control means, and the notification means; the second information processing device has a generating means for generating an image of the virtual object whose form is controlled in accordance with the degree of proximity in the first case; 9. The information processing system according to claim 1,

10. In a second case, the first information processing apparatus determines whether or not a distance between the real object and the virtual object in the specific space is shorter than the predetermined distance; In a third case, the second information processing device determines whether or not a distance between the real object and the virtual object in the specific space is shorter than the predetermined distance.

10. The information processing system according to claim 9.

11. 1. An information processing method for controlling a display device so as to allow a user to visually recognize a specific space in which a virtual object is arranged in a real space including a real object, comprising: an acquisition step of acquiring an image of the virtual object, the form of which is controlled in accordance with a degree of proximity between the real object and the virtual object, in a first case in which it is determined that the distance between the real object and the virtual object in the specific space is shorter than a predetermined distance; a display control step of controlling the display device to display an image of the virtual object whose form is controlled in accordance with the degree of proximity in the first case; In the first case, a notification step of performing a specific notification indicating that the real object and the virtual object have come close to each other before an image of the virtual object whose form has been controlled in accordance with the degree of proximity is displayed on the display device; 13. An information processing method comprising:

12. A program for causing a computer to function as each of the means of the information processing system according to any one of claims 1 to 8.

Images