Information processing device

Abstract

To reduce inadequate display of a person image (an image obtained by imaging a person) in a three-dimensional space.SOLUTION: An information processing device of an invention comprises: acquisition means which acquires information on the positional relation between a first user and the display position of a second user, in a first three-dimensional space viewed by the first user; and determination means which determines at least one of image processing to execute on an image obtained by imaging the first user and an image obtained by imaging the second user, based on the information.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to an information processing device, and more particularly to mixed reality image display. [Background technology]

[0002] Mixed reality (MR) technology is known as a technology that seamlessly fuses the real world and the virtual world in real time. MR technology is sometimes used in video see-through head-mounted displays (HMDs).

[0003] In a video see-through HMD, for example, a range of real space that corresponds to the field of view of a user wearing the HMD is captured by a video camera. Then, a computer graphic (CG) is synthesized with an image (video) of the real space acquired by the video camera, and the synthesized image (an image obtained by synthesizing the image of the real space with the CG) is displayed on a display panel inside the HMD. By viewing the synthesized image displayed on the display panel, the user can get the sensation that a virtual object represented by the CG exists in the real space.

[0004] In addition, a video see-through HMD can provide a user wearing the HMD with the experience (sensation) of being in the same space as a person in a remote location by synthesizing an image of the real space with an image of a person in a remote location.

[0005] Patent Document 1 discloses a technology in which a user's character is displayed at a position in a virtual space corresponding to the user's position in real space, and the character is changed when two users high-five each other in real space. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2019-122496 A Summary of the Invention [Problem to be solved by the invention]

[0007] Consider a case where a user wearing an HMD and a person in a remote location communicate with each other using images (video) captured of each of them. In this case, the display of a person image (an image captured of the user wearing the HMD or an image captured of a person in a remote location) in a three-dimensional space (for example, a displayed real space) may be inappropriate. For example, even if a person in a remote location does not want to be able to see his / her own image in detail from a close distance, the user wearing the HMD may be able to see the image of the person in the remote location in detail from a close distance. Even if the technology disclosed in Patent Document 1 is used, such a problem cannot be solved.

[0008] An object of the present invention is to provide a technique capable of preventing inappropriate display of a person image (a captured image of a person) in a three-dimensional space. [Means for solving the problem]

[0009] A first aspect of the present invention is characterized in that the present invention has an acquisition means for acquiring information regarding a positional relationship between a first user and a display position of a second user in a first three-dimensional space viewed by the first user, and a decision means for deciding, based on the information, at least one of image processing to be applied to an image captured of the first user and image processing to be applied to an image captured of the second user. The information processing device has the following characteristics.

[0010] A second aspect of the present invention is an information processing device having a processing means for applying image processing to an image of a second user to be displayed to a first user, or an image of the first user to be displayed to a second user, and characterized in that the image processing differs between a case where a distance from the first user to a display position of the second user in a three-dimensional space seen by the first user is a first distance and a case where a second distance different from the first distance.

[0011] A third aspect of the present invention is a control method for an information processing device, characterized by having a step of acquiring information regarding a positional relationship between a first user and a display position of a second user in a first three-dimensional space seen by the first user, and a step of determining, based on the information, at least one of image processing to be applied to an image captured of the first user and image processing to be applied to an image captured of the second user.

[0012] A fourth aspect of the present invention is a program for causing a computer to function as each of the means of the information processing device described above.A fifth aspect of the present invention is a computer-readable storage medium storing a program for causing a computer to function as each of the means of the information processing device described above. Effect of the Invention

[0013] According to the present invention, it is possible to prevent inappropriate display of a person image (an image obtained by capturing a person) in a three-dimensional space. [Brief description of the drawings]

[0014] [Figure 1] 1 is a schematic diagram showing a configuration of an information processing system according to a first embodiment. [Diagram 2] 1 is a block diagram showing a configuration of an information processing system according to a first embodiment. [Diagram 3] 1 is a flowchart showing an overall process according to the first embodiment. [Figure 4] 11 is a flowchart showing a host side setting process according to the first embodiment. [Diagram 5] 5 is a flowchart showing a guest-side setting process according to the first embodiment. [Figure 6] 11 is a flowchart showing a host side transmission process according to the first embodiment. [Figure 7] 11 is a flowchart showing a Guest-side transmission process according to the first embodiment. [Figure 8] 5 is a flowchart showing a host-side display process according to the first embodiment. [Figure 9] 5 is a flowchart showing a guest-side display process according to the first embodiment. [Figure 10] 10 is a flowchart showing a host-side display process according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] (First embodiment) Hereinafter, a first embodiment of the present invention will be described. FIG. 1 is a schematic diagram showing a configuration of an information processing system 100 according to the first embodiment, and FIG. 2 is a block diagram showing the configuration of the information processing system 100. The information processing system 100 includes head mounted displays (HMDs) 110 and 120, cameras 130 and 140, a marker 210, and controllers 220 and 230. The HMDs 110, 120, the cameras 130, and 140 are connected to a network 190 and can communicate with each other via the network 190. The controller 220 can communicate with the HMDs 110 and the cameras 130 wirelessly or wired. The controller 230 can communicate with the HMDs 120 and the cameras 140 wirelessly or wired. A space 170 (e.g., the person's room) in which a person 150 who is a user of the HMD 110 is present is different from a space 180 (e.g., the person's room) in which a person 160 who is a user of the HMD 120 is present. A person 150 wears an HMD 110 on his / her head, and a person 160 wears an HMD 120 on his / her head. A camera 130 is installed in a space 170 where the person 150 is, and a camera 140 is installed in a space 170 where the person 160 is. 80. Person 150 wears controller 220 on his arm, and person 160 wears controller 230 on his arm. Marker 210 is placed in space 170 in which person 150 is present.

[0016] The HMD 110 includes a CPU 111, a ROM 112, a RAM 113, a posture sensor 114, a communication unit 115, an imaging unit 116, and a display unit 117. The CPU 111 is a computing device that controls the entire HMD 110, and executes various programs stored in the ROM 112 to perform various processes. The ROM 112 is a read-only non-volatile memory that stores various information (for example, various programs and various parameters). The RAM 113 is a memory that temporarily stores various information, and is also used as a work memory for the CPU 111. The posture sensor 114 is a sensor that detects the posture of the HMD 110, and includes at least one of an acceleration sensor and a geomagnetic sensor. A gyro sensor is a type of acceleration sensor. The communication unit 115 communicates with an external device by wire or wirelessly. The imaging unit 116 captures a range (forward of the HMD 110) corresponding to the field of view of the person 150 wearing the HMD 110 in the real space, thereby acquiring a field of view image representing the range. The imaging unit 116 includes an imaging sensor such as a CCD sensor or a CMOS sensor. The display unit 117 can display various images and various information. The display unit 117 includes a display panel such as a liquid crystal panel or an organic EL panel. The person 150 can see the image (video) displayed on the display unit 117 by wearing the HMD 110. The HMD 110 is a video see-through type HMD, and the CPU 111 can display the field of view image obtained by the imaging unit 116 on the display unit 117. The CPU 111 can also detect the marker 210 from the field of view image and obtain information regarding the positional relationship between the HMD 110 and the marker 210 (for example, the distance from the HMD 110 to the marker 210, or the angle of the direction from the HMD 110 toward the marker 210).

[0017] The HMD 120 includes a CPU 121, a ROM 122, a RAM 123, a posture sensor 124, a communication unit 125, an imaging unit 126, and a display unit 127. The configuration of the HMD 120 is similar to the configuration of the HMD 110.

[0018] The camera 130 is an imaging device that captures an image of the person 150 (the space 170 in which the person 150 is present) to obtain a spatial image that represents at least a part of the space 170. The camera 130 can perform image processing on the obtained spatial image. The camera 130 transmits the spatial image of the space 170 to the HMD 120 via the network 190. In the HMD 120, at least a part of the spatial image of the space 170 is displayed on the display unit 127 under the control of the CPU 121. Of the spatial image of the space 170, a portion (display range) displayed on the display unit 127 can be changed according to the attitude of the HMD 120 (detection result of the attitude sensor 124). By looking at the image displayed on the display unit 127, the person 160 can get the sensation of being in the same space 170 as the person 150.

[0019] The camera 140 is an imaging device that captures an image of the person 160 (the space 180 in which the person 160 is present) to obtain a spatial image that represents at least a part of the space 180. The camera 140 can perform image processing on the obtained spatial image. The camera 140 transmits the spatial image of the space 180 to the HMD 110 via the network 190. In the HMD 110, the area of ​​the person 160 is extracted from the spatial image of the space 180 under the control of the CPU 111. This results in a person image that is an image of the person 160 (an image that represents only the person 160). Then, the person image is synthesized with the field of view image (field of view image of the person 150) obtained by the imaging unit 116, and the synthesized image (an image in which the field of view image and the person image are synthesized) is displayed on the display unit 117. The person image is synthesized at the position of the marker 210 based on information (positional relationship information) regarding the positional relationship between the HMD 110 and the marker 210. By looking at the image displayed on the display unit 117, the person 150 feels as if the person 160 is in the same space as the person 150. You can get the feeling that you are at 170.

[0020] The marker 210 is an object on which a predetermined pattern is printed, and indicates a position in the space 170. For example, based on the size, orientation, and inclination of the marker 210 (pattern) in the field of view image obtained by the imaging unit 116 of the HMD 110, positional relationship information between the marker 210 and the HMD 110 can be acquired. The positional relationship information indicates, for example, the distance from the HMD 110 to the marker 210, or the angle of the direction from the HMD 110 to the marker 210. Note that, although the marker 210 is an object, and the positional relationship information between the marker 210 and the HMD 110 is acquired by detecting the marker 210 from the field of view image obtained by the imaging unit 116, this is not limited thereto. For example, the person 150 may specify a position in the space 170 (a position in the field of view image), and the positional relationship information between the specified position and the HMD 110 may be acquired.

[0021] The controller 220 receives an operation (for example, an image processing setting operation) from the person 150. The controller 220 generates control information according to the received operation, and transmits the control information to at least one of the HMD 110 and the camera 130 via wireless communication or wired communication.

[0022] The controller 230 receives an operation (for example, an image processing setting operation) from the person 160. The controller 230 generates control information according to the received operation, and transmits the control information to at least one of the HMD 120 and the camera 140 via wireless communication or wired communication.

[0023] In this way, in the information processing system 100 , the person 150 in the space 170 and the person 160 in the space 180 can feel as if they are in the same space 170 .

[0024] According to the above-mentioned operation, person 160 is invited to space 170 in which person 150 is present. Therefore, space 170 in which person 150 is present can be regarded as the Host side, and space 180 in which person 160 is present can be regarded as the Guest side.

[0025] Each of the cameras 130 and 140 may be a monocular camera, but is preferably a stereo camera. If the camera 130 is a stereo camera, the person 150 can wear the HMD 110 and stereoscopically view two images with parallax, thereby obtaining a higher sense of immersion. Similarly, if the camera 140 is a stereo camera, the person 160 can wear the HMD 120 and stereoscopically view two images with parallax, thereby obtaining a higher sense of immersion.

[0026] In addition, it is preferable that the imaging range of the Host-side camera 130 is wide, and for example, the viewing angle (angle of view) may be in the range of 180 degrees, or the viewing angle may be in the range of 360 degrees. The image obtained by the Host-side camera 130 may be a semi-spherical image (VR 180 image), a celestial sphere image (omnidirectional image), or a panoramic image. Since the imaging range of the Host-side camera 130 is wide, the Guest-side person 160 can see the image of the Host-side space 170 even if he or she changes the direction of his or her face significantly, and can get the feeling that he or she is in the Host-side space 170.

[0027] In the first embodiment, the Host-side person 150 can get the sensation that the Guest-side person 160 is present at the position of the marker 210. When the positional relationship between the Host-side HMD 110 and the marker 210 changes, the image (video) of the Guest-side person 160 displayed on the display unit 117 also changes. For example, when the Host-side person 150 approaches the marker 210, the distance from the Host-side HMD 110 to the marker 210 becomes shorter, and the size of the image of the Guest-side person 160 displayed on the display unit 117 becomes larger. This allows the Host-side person 150 to get the sensation that he is getting closer to the Guest-side person 160.

[0028] In the first embodiment, the positional relationship between the Guest-side person 160 and the display position of the Host-side person 150 in the Host-side space 170 seen by the Guest-side person 160 is determined according to the positional relationship between the Host-side HMD 110 and the marker 210. The positional relationship between the Host-side HMD 110 and the marker 210 may be interpreted as the positional relationship between the Host-side person 150 and the display position of the Guest-side person 160 in the Host-side space 170 seen by the Host-side person 150. The positional relationship between the Host-side HMD 110 and the marker 210 may be interpreted as the positional relationship between the Guest-side person 160 and the display position of the Host-side person 150 in the Host-side space 170 seen by the Guest-side person 160. Therefore, when the Host-side person 150 approaches the marker 210, the Host-side person 150 can get the sensation of getting closer to the Guest-side person 160, and the Guest-side person 160 can get the sensation of being approached by the Host-side person 150.

[0029] In the first embodiment, one Host-side person 150 and one Guest-side person 160 communicate with each other, but the number of people who communicate is not particularly limited. For example, one Host-side person may communicate with two Guest-side people.

[0030] Fig. 3 is a flowchart showing the overall process performed by the information processing system 100. For example, when the information processing system 100 (the host side HMD 110, the guest side HMD 120, the host side camera 130, and the guest side camera 140) is started, the overall process of Fig. 3 starts.

[0031] In step S301, the CPU 111 of the Host-side HMD 110 determines whether the Host-side HMD 110, the Guest-side HMD 120, the Host-side camera 130, and the Guest-side camera 140 are connected to one another (via the network 190). The CPU 111 waits until it is determined that these four devices are connected to one another, and when it is determined that the four devices are connected to one another, it performs the process of step S302.

[0032] In step S301, the CPU 121 of the Guest-side HMD 120 also determines whether the four devices are connected to each other. The CPU 121 waits until it is determined that the four devices are connected to each other, and when it is determined that the four devices are connected to each other, it performs the process of step S303.

[0033] In step S302, the CPU 111 of the host-side HMD 110 performs host-side setting processing. Details of the host-side setting processing will be described later with reference to FIG.

[0034] In step S303, the CPU 121 of the Guest side HMD 120 performs a Guest side setting process. The Guest side setting process will be described in detail later with reference to FIG.

[0035] In FIG. 3, step S303 is shown as the next step after step S302. However, the processing of step S303 may be performed before the processing of step S302, or the processing of step S302 and the processing of step S303 may be performed in parallel.

[0036] In step S304, the host-side camera 130 performs a host-side transmission process. The details of the host-side transmission process will be described later with reference to FIG.

[0037] In step S305, the Guest-side camera 140 performs a Guest-side transmission process. The details of the Guest-side transmission process will be described later with reference to FIG.

[0038] In FIG. 3, step S305 is shown as the next step after step S304. However, step S305 may be performed before step S304, or step S304 and step S305 may be performed in parallel.

[0039] In step S306, the CPU 111 of the host-side HMD 110 performs host-side display processing. The host-side display processing will be described in detail later with reference to FIG.

[0040] In step S307, the CPU 121 of the Guest side HMD 120 performs a Guest side display process. The details of the Guest side display process will be described later with reference to FIG.

[0041] In FIG. 3, step S307 is shown as the next step after step S306. However, the processing of step S307 may be performed before the processing of step S306, or the processing of step S306 and the processing of step S307 may be performed in parallel.

[0042] In step S308, the CPU 111 of the host-side HMD 110 determines whether or not to stop the information processing system 100. The CPU 111 repeatedly performs the host-side setting process (step S302) and the host-side display process (step S306) for each frame until it determines that the information processing system 100 is to be stopped. Then, when the CPU 111 determines that the information processing system 100 is to be stopped, it ends the overall process in Fig. 3. The host-side camera 130 repeatedly performs the host-side transmission process (step S304) for each frame until the CPU 111 determines that the information processing system 100 is to be stopped.

[0043] In step S308, the CPU 121 of the Guest side HMD 120 also determines whether or not to stop the information processing system 100. The CPU 121 repeatedly performs the Guest side setting process (step S303) and the Guest side display process (step S307) for each frame until it determines that the information processing system 100 is to be stopped. Then, when the CPU 121 determines that the information processing system 100 is to be stopped, it ends the overall process in Fig. 3. The Guest side camera 140 repeatedly performs the Guest side transmission process (step S305) for each frame until the CPU 121 determines that the information processing system 100 is to be stopped.

[0044] 4 is a flowchart showing the host side setting process (step S302 in FIG. 3). The host side setting process is performed by the CPU 111 of the HMD 110 on the host side.

[0045] In step S3021, the CPU 111 determines the image processing to be performed by the Host-side HMD 110 and the image processing to be performed by the Host-side camera 130 based on the information expanded in the RAM 113. The image processing to be performed by the Host-side HMD 110 may be interpreted as image processing to be viewed by the Host-side person 150, and the image processing to be performed by the Host-side camera 130 may be interpreted as image processing to be shown to the Guest-side person 160. For example, the Host-side person 150 can set the image processing to be performed by the Host-side HMD 110 and the image processing to be performed by the Host-side camera 130 using the controller 220. The image processing includes, for example, at least one of blurring processing, sharpness processing, white balance processing, skin beautification processing (for example, processing to change the skin color or blur the skin area), and mosaic processing.

[0046] The CPU 111 can also determine image processing (for example, at least one of blurring processing, sharpness processing, white balance processing, skin beautification processing, and mosaic processing) based on positional relationship information between the marker 210 and the host side HMD 110. The CPU 111 determines image processing that reduces the visibility of details to a greater degree of reduction as the distance from the host side HMD 110 to the marker 210 becomes shorter. The CPU 111 determines image processing that improves the appearance to a greater degree of improvement as the distance from the host side HMD 110 to the marker 210 becomes shorter. The distance from the Host side HMD 110 to the marker 210 may be interpreted as the distance from the Host side person 150 to the display position of the Guest side person 160.

[0047] For example, the CPU 111 may determine strong blurring when the distance from the host side HMD 110 to the marker 210 is less than a threshold Th1, and may determine medium blurring when the distance is equal to or greater than the threshold Th1 and less than a threshold Th2. The CPU 111 may determine weak blurring or may determine not to perform blurring when the distance from the host side HMD 110 to the marker 210 is equal to or greater than the threshold Th2. The CPU 111 may determine strong skin beautifying when the distance from the host side HMD 110 to the marker 210 is less than the threshold Th1, and may determine medium skin beautifying when the distance is equal to or greater than the threshold Th1 and less than the threshold Th2. The CPU 111 may determine weak skin beautifying or may determine not to perform skin beautifying when the distance from the host side HMD 110 to the marker 210 is equal to or greater than the threshold Th2.

[0048] Note that the image processing may be determined taking into consideration the angle of the direction from the Host-side HMD 110 to the marker 210. The image processing may be determined based on the three-dimensional positional relationship between the Host-side HMD 110 and the marker 210. Based on the positional relationship information between the marker 210 and the Host-side HMD 110, it is also possible to grasp the location where the Guest-side person 160 is displayed on the Host-side HMD 110 and the location where the Host-side person 150 is displayed on the Guest-side HMD 120. The image processing may be determined taking into consideration these locations. As the image processing performed by the Host-side HMD 110 and the image processing performed by the Host-side camera 130, a common image processing may be determined, or individual image processing may be determined. Here, consider a case where the Host-side person 150 wants to see the Guest-side person 160 in detail, but does not want the Guest-side person 160 to see him (the Host-side person 150) in detail. In this case, weak blurring may be determined as the image processing to be performed by the host-side HMD 110, and strong blurring may be determined as the image processing to be performed by the host-side camera 130. Only one of the image processing to be performed by the host-side HMD 110 and the image processing to be performed by the host-side camera 130 may be determined.

[0049] In step S3022, the CPU 111 transmits positional relationship information between the marker 210 and the Host side HMD 110 to the Guest side HMD 120 via the network 190.

[0050] 5 is a flowchart showing the Guest side setting process (step S303 in FIG. 3). The Guest side setting process is performed by the CPU 121 of the Guest side HMD 120.

[0051] In step S3031, the CPU 121 determines the image processing to be performed by the Guest side HMD 120 and the image processing to be performed by the Guest side camera 140 based on the information expanded in the RAM 123. The image processing to be performed by the Guest side HMD 120 may be interpreted as image processing to be viewed by the Guest side person 160, and the image processing to be performed by the Guest side camera 140 may be interpreted as image processing to be shown to the Host side person 150. For example, the Guest side person 160 can set the image processing to be performed by the Guest side HMD 120 and the image processing to be performed by the Guest side camera 140 using the controller 230. The image processing includes, for example, at least one of blurring processing, sharpness processing, white balance processing, skin beautification processing (for example, processing to change the skin color or blur the skin area), and mosaic processing.

[0052] The CPU 121 receives the positional relationship information (positional relationship information between the marker 210 and the host-side HMD 110) transmitted in step S3022 of FIG. 4. Then, the CPU 121 performs image processing (for example, blurring, sharpness processing, and white balance processing) based on the received positional relationship information. The CPU 121 may also determine image processing to be performed by the host side HMD 110 so as to reduce the visibility of details to a greater degree as the distance from the host side HMD 110 to the marker 210 becomes shorter. The CPU 111 may also determine image processing to improve the appearance to a greater degree as the distance from the HMD 110 to the marker 210 becomes shorter. The distance from the host side HMD 110 to the marker 210 may be interpreted as the distance from the guest side person 160 to the display position of the host side person 150.

[0053] For example, the CPU 121 may determine strong blurring when the distance from the host side HMD 110 to the marker 210 is less than a threshold Th1, and may determine medium blurring when the distance is equal to or greater than the threshold Th1 and less than a threshold Th2. The CPU 121 may determine weak blurring or may determine not to perform blurring when the distance from the host side HMD 110 to the marker 210 is equal to or greater than the threshold Th2. The CPU 121 may determine strong skin beautification when the distance from the host side HMD 110 to the marker 210 is less than the threshold Th1, and may determine medium skin beautification when the distance is equal to or greater than the threshold Th1 and less than the threshold Th2. The CPU 121 may determine weak skin beautification or may determine not to perform skin beautification when the distance from the host side HMD 110 to the marker 210 is equal to or greater than the threshold Th2.

[0054] The image processing may be determined in consideration of the angle of the direction from the Host-side HMD 110 to the marker 210. The image processing may be determined based on the three-dimensional positional relationship between the Host-side HMD 110 and the marker 210. Based on the positional relationship information between the marker 210 and the Host-side HMD 110, the location where the Guest-side person 160 is displayed on the Host-side HMD 110 and the location where the Host-side person 150 is displayed on the Guest-side HMD 120 can also be grasped. The image processing may be determined in consideration of these locations. A common image processing may be determined as the image processing performed by the Guest-side HMD 120 and the image processing performed by the Guest-side camera 140, or individual image processing may be determined. Here, consider a case where the Guest-side person 160 wants to see the Host-side person 150 in detail, but does not want the Host-side person 150 to see him (the Guest-side person 160) in detail. In this case, weak blurring may be determined as the image processing to be performed by the Guest-side HMD 120, and strong blurring may be determined as the image processing to be performed by the Guest-side camera 140. Only one of the image processing to be performed by the Guest-side HMD 120 and the image processing to be performed by the Guest-side camera 140 may be determined.

[0055] 6 is a flowchart showing the host side transmission process (step S304 in FIG. 3). The host side transmission process is performed by the host side camera 130.

[0056] In step S3041, the host-side camera 130 applies the image processing (image processing to be performed by the host-side camera 130) determined in step S3021 of Fig. 4 to the captured spatial image (spatial image of the host-side space 170). For example, the CPU 111 notifies the host-side camera 130 of the image processing determined in step S3021 (image processing to be performed by the host-side camera 130), and the host-side camera 130 starts executing the image processing in response to the notification of the image processing. The notification of the image processing may be interpreted as control for executing the image processing. The image processing may be applied to the entire spatial image, or may be applied only to the area of ​​the host-side person 150.

[0057] In step S3042, the Host-side camera 130 transmits the spatial image after the image processing in step S3041 to the Guest-side HMD 120 via the network 190. The transmitted spatial image (image of the Host-side person 150) is displayed to the Guest-side person 160 on the Guest-side HMD 120. The transmission of the image is performed by using the The notification of image processing may be interpreted as control for displaying an image. Since the transmission and display of an image are performed in response to the notification of image processing, the notification of image processing may be interpreted as control for displaying an image. By performing the image processing in step S3041, it is possible to prevent an inappropriate image (for example, an image that is not desired to be shown to the Guest-side person 160) from being transmitted and displayed.

[0058] 7 is a flowchart showing the Guest side transmission process (step S305 in FIG. 3). The Guest side transmission process is performed by the Guest side camera 140.

[0059] In step S3051, the Guest-side camera 140 applies the image processing (image processing to be performed by the Guest-side camera 140) determined in step S3031 of Fig. 5 to the captured spatial image (spatial image of the Guest-side space 180). For example, the CPU 121 notifies the Guest-side camera 140 of the image processing determined in step S3031 (image processing to be performed by the Guest-side camera 140), and the Guest-side camera 140 starts executing the image processing in response to the notification of the image processing. The notification of the image processing may be interpreted as control for executing the image processing. The image processing may be applied to the entire spatial image, or may be applied only to the area of ​​the Guest-side person 160.

[0060] In step S3052, the Guest-side camera 140 transmits the spatial image after the image processing in step S3051 to the Host-side HMD 110 via the network 190. The transmitted spatial image (image of the Guest-side person 160) is displayed to the Host-side person 150 in the Host-side HMD 110. The transmission of the image may be interpreted as control for displaying the image (display control). Since the transmission and display of the image are performed in response to the notification of the image processing, the notification of the image processing may be interpreted as control for displaying the image. By performing the image processing in step S3051, it is possible to prevent an inappropriate image (for example, an image that is not desired to be shown to the Host-side person 150) from being transmitted and displayed.

[0061] 8 is a flowchart showing the host side display process (step S306 in FIG. 3). The host side display process is performed by the CPU 111 of the host side HMD 110.

[0062] In step S3061, the CPU 111 receives (acquires) a spatial image of the Guest side space 180 from the Guest side camera 140 using the communication unit 115. If a part of the Guest side person 160 is not shown in the spatial image of the Guest side space 180, an unnatural composite image (an unnatural composite image in which a part of the Guest side person 160 is not drawn) may be obtained. Therefore, it is preferable that the whole body of the Guest side person 160 is shown in the spatial image of the Guest side space 180.

[0063] In step S3062, the CPU 111 performs a geometric transformation process on the spatial image acquired in step S3061 as necessary. For example, when the Guest-side camera 140 captures an image using a super-wide-angle lens such as a fisheye lens, the CPU 111 needs to perform a geometric transformation process such as equirectangular transformation to obtain a spatial image suitable for synthesis.

[0064] In step S3063, the CPU 111 extracts the area of ​​the Guest-side person 160, which is the main subject, from the spatial image acquired in step S3061 (the spatial image after the geometric transformation process in step S3062). This results in a person image, which is an image of the Guest-side person 160 (an image showing only the Guest-side person 160). The method of extracting the area of ​​the Guest-side person 160 is not particularly limited. For example, the Guest-side person 160 may be imaged against a green background, and the area of ​​the Guest-side person 160 may be extracted by chromakey processing. The area of ​​the Guest-side person 160 may be extracted using a calculator (trained model) trained by machine learning such as deep learning. The extraction method is selected, for example, taking into consideration system resources and extraction accuracy.

[0065] In step S3064, the CPU 111 determines a synthesis position (display position) of the person image acquired in step S3063. In the first embodiment, the position of the marker 210 is determined as the synthesis position. Note that the synthesis position is not limited to this, and for example, information on the movement of the Guest-side person 160 may be acquired, and a position shifted from the position of the marker 210 in accordance with the movement of the Guest-side person 160 may be determined as the synthesis position.

[0066] In step S3065, the CPU 111 performs the image processing (image processing to be performed by the host-side HMD 110) determined in step S3021 of Fig. 4 on the person image acquired in step S3063. By performing this image processing, it is possible to prevent an inappropriate image (for example, an image with excessively high visibility of details) from being displayed.

[0067] In step S3066, the CPU 111 composites the person image after the image processing in step S3065 with the field of view image (field of view image of the Host-side person 150) obtained by the imaging unit 116 of the Host-side HMD 110. The person image is composited at the composite position determined in step S3064. This provides a composite image in which the Guest-side person 160 is in the Host-side space 170. At this time, the composite size (display size) of the person image is also determined. For example, the composite size is determined based on the performance of the Guest-side camera 140 and the Host-side HMD 110 so that the Guest-side person 160 feels as if he or she is actually there.

[0068] In step S3067, CPU 111 displays on display unit 117 the composite image acquired in step S3066.

[0069] 9 is a flowchart showing the Guest side display process (step S307 in FIG. 3). The Guest side display process is performed by the CPU 121 of the Guest side HMD 120.

[0070] In step S3071, the CPU 121 receives (acquires) a spatial image of the host-side space 170 from the host-side camera 130 using the communication unit 125. If a part of the host-side person 150 is not captured in the spatial image of the host-side space 170, an unnatural composite image (an unnatural composite image in which a part of the host-side person 150 is not drawn) may be obtained. For this reason, it is preferable that the whole body of the host-side person 150 is captured in the spatial image of the host-side space 170.

[0071] In step S3072, the CPU 121 performs a geometric transformation process on the spatial image acquired in step S3071 as necessary. For example, if the host-side camera 130 is a wide-angle stereo camera, the CPU 121 needs to perform a geometric transformation process such as equirectangular transformation or perspective projection transformation to obtain a spatial image suitable for stereoscopic viewing.

[0072] In step S3073, the CPU 121 extracts the area of ​​the host-side person 150, which is the main subject, from the spatial image acquired in step S3071 (the spatial image after the geometric transformation process in step S3072). This results in a person image, which is an image of the host-side person 150 (an image showing only the host-side person 150). The method of extracting the area of ​​the host-side person 150 is not particularly limited. For example, the host-side person 150 may be imaged against a green background, and the area of ​​the host-side person 150 may be extracted by chromakey processing. The area of ​​the host-side person 150 may be extracted using a calculator (trained model) trained by machine learning such as deep learning. The extraction method is selected, for example, taking into consideration system resources and extraction accuracy.

[0073] In step S3074, the CPU 121 performs A synthesis position (display position) is determined. In the first embodiment, the synthesis position is determined based on the positional relationship information (positional relationship information between the marker 210 and the Host-side HMD 110) transmitted in step S3022 in Fig. 4. For example, the synthesis position (display position) of the Host-side person 150 is determined so that the positional relationship between the Host-side person 150 and the Guest-side person 160 matches between the Host-side and Guest-side.

[0074] In step S3075, the CPU 121 performs the image processing (image processing to be performed by the Guest-side HMD 120) determined in step S3031 of Fig. 5 on the person image acquired in step S3073. By performing this image processing, it is possible to prevent an inappropriate image (for example, an image with excessively high visibility of details) from being displayed.

[0075] In step S3076, the CPU 121 composites the person image after the image processing in step S3075 with the space image acquired in step S3071. The person image is composited at the composite position determined in step S3074. This results in a composite image in which the host-side person 150 is in the host-side space 170. At this time, the composite size (display size) of the person image is also determined. For example, the composite size is determined based on the performance of the host-side camera 130 and the guest-side HMD 120 so that the host-side person 150 feels as if he or she is actually there. In addition, image processing is also performed to naturally express the area (part of the space image) from which the person image is extracted.

[0076] In step S3077, CPU 121 displays on display unit 127 the composite image acquired in step S3076.

[0077] Although the composite position (display position) of the host-side person 150 is determined using the positional relationship information between the marker 210 and the host-side HMD 110, this is not limiting. For example, the composite position of the host-side person 150 does not need to be changed from the position in the spatial image acquired in step S3071. In that case, image processing may be performed on the spatial image acquired in step S3071 without extracting the person image of the host-side person 150.

[0078] According to the above processing, the Host-side HMD 110 applies image processing to an image of the Guest-side person 160, and the Guest-side HMD 120 applies image processing to an image of the Host-side person 150. The image processing differs between when the distance from the Host-side person 150 to the display position of the Guest-side person 160 in the Host-side space 170 seen by the Host-side person 150 is a first distance and when the distance is a second distance different from the first distance.

[0079] As described above, according to the first embodiment, information (positional relationship information) regarding the positional relationship between the first user and the display position of the second user in the three-dimensional space seen by the first user is acquired. Then, based on the acquired positional relationship information, at least one of image processing to be applied to the image capturing the first user and image processing to be applied to the image capturing the second user is determined. In this way, it is possible to prevent inappropriate display of a person image (an image capturing a person) in the three-dimensional space.

[0080] Second embodiment The second embodiment of the present invention will be described below. Note that, in the following, the description of the same points as the first embodiment (for example, the same configuration and processing as the first embodiment) will be omitted, and only the points different from the first embodiment will be described.

[0081] 10 is a flowchart showing a host-side display process (step S306 in FIG. 3) according to the second embodiment. The host-side display process is performed by the CPU 111 of the host-side HMD 110. The host side display process according to the second embodiment includes step S3068 between step S3066 and step S3067.

[0082] In step S3068, the CPU 111 receives (acquires) the spatial image (spatial image of the host-side space 170 after image processing) transmitted in step S3042 in Fig. 6. Then, the CPU 111 generates a simulated image that reproduces the composite image displayed on the display unit 127 of the guest-side HMD 120 based on the positional relationship information between the marker 210 and the host-side HMD 110 and the received spatial image. The CPU 111 updates the composite image by combining the simulated image with the composite image generated in step S3066.

[0083] As described above, according to the second embodiment, an image of the first user arranged in a three-dimensional space as seen by the second user is generated based on the positional relationship information, and the generated image is displayed to the first user. In this way, the first user (Host side person 150) can easily check how he or she appears to the second user (Guest side person 160).

[0084] The above-described embodiment (including the modified examples) is merely an example, and the present invention also includes configurations obtained by appropriately modifying or changing the configurations of the above-described embodiment within the scope of the gist of the present invention. The present invention also includes configurations obtained by appropriately combining the configurations of the above-described embodiment.

[0085] For example, although an example of using a video see-through HMD has been described, an optical see-through HMD may be used. The optical see-through HMD may or may not have an imaging unit for acquiring a visual field image. The Guest-side HMD 120 may perform the same processing as the Host-side display processing. By doing so, the Guest-side person 160 can get the feeling that the Host-side person 150 is in the same Guest-side space 180 as the Guest-side person 160. A plurality of users may experience different spaces. The space experienced by the user may be a virtual space that is not an imaged real space.

[0086] Furthermore, at least a part of the above-mentioned processing performed by the camera may be performed by an information processing device other than the camera. At least a part of the above-mentioned processing performed by the HMD may be performed by an information processing device other than the HMD. The device to which the present invention is applied is not particularly limited. For example, the present invention may be applied to a personal computer connected to the HMD. The present invention may be applied to a cloud server provided on the network 190.

[0087] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0088] The disclosure of the present embodiment includes the following configuration, method, program, and medium. (Configuration 1) An acquisition means for acquiring information regarding a positional relationship between a first user and a display position of a second user in a first three-dimensional space viewed by the first user; a determination means for determining at least one of image processing to be applied to an image captured of the first user and image processing to be applied to an image captured of the second user based on the information; 13. An information processing device comprising: (Configuration 2) A second user and the first user in a second three-dimensional space seen by the second user. The positional relationship with the display position of the user is determined according to the positional relationship between the display positions of the first user and the second user in the first three-dimensional space. 2. The information processing device according to configuration 1. (Configuration 3) The determining means determines image processing to be applied to an image of the first user based on the information; The information processing device includes: a first control means for controlling the image processing determined by the determination means to be performed on the image captured of the first user; Further having 3. The information processing device according to configuration 1 or 2. (Configuration 4) a first display control means for controlling the image of the first user after the image processing to be displayed to the second user; Further having 4. The information processing device according to configuration 3. (Configuration 5) The determining means determines image processing to be applied to an image of the second user based on the information; and The information processing device includes: a second control means for controlling the image processing determined by the determination means to be performed on the image captured of the second user; Further having 5. The information processing device according to any one of configurations 1 to 4. (Configuration 6) a second display control means for controlling the image of the second user after the image processing to be displayed to the first user; Further having 6. The information processing device according to configuration 5. (Configuration 7) The image processing determined by the determining means includes at least one of blurring processing, sharpness processing, white balance processing, skin beautification processing, and mosaic processing. 7. The information processing device according to any one of configurations 1 to 6. (Configuration 8) The determining means determines image processing for reducing visibility of details to a greater degree as a distance from the first user to a display position of the second user in the first three-dimensional space becomes shorter. 8. The information processing device according to any one of configurations 1 to 7. (Configuration 9) The determining means determines image processing that improves appearance to a greater degree as the distance from the first user to the display position of the second user in the first three-dimensional space becomes shorter. 8. The information processing device according to any one of configurations 1 to 7. (Configuration 10) a positional relationship between the second user and a display position of the first user in a second three-dimensional space viewed by the second user is determined according to a positional relationship between the first user and the display position of the second user in the first three-dimensional space; The information processing device includes: generating means for generating an image of the first user positioned in the second three-dimensional space, as viewed by the second user, based on the information; a third display control means for controlling the image generated by the generation means to be displayed to the first user; Further having 10. The information processing device according to any one of configurations 1 to 9. (Configuration 11) a processing means for performing image processing on an image of a second user to be displayed to a first user or an image of a first user to be displayed to a second user; The image processing is different depending on whether a distance from the first user to a display position of a second user in a three-dimensional space viewed by the first user is a first distance or a second distance different from the first distance. 23. An information processing apparatus comprising: (method) acquiring information regarding a positional relationship between a first user and a display position of a second user in a first three-dimensional space viewed by the first user; determining at least one of image processing to be applied to an image captured of the first user and image processing to be applied to an image captured of the second user based on the information; 13. A method for controlling an information processing apparatus comprising the steps of: (program) 12. A program for causing a computer to function as each of the means of the information processing device according to any one of configurations 1 to 11. (medium) 12. A computer-readable storage medium storing a program for causing a computer to function as each of the means of the information processing device according to any one of configurations 1 to 11. [Explanation of symbols]

[0089] HMD110,120 CPU111,121

Claims

1. Acquisition means for acquiring information regarding the positional relationship between the first user and the display position of the second user in a first three-dimensional space viewed by the first user, Based on the aforementioned information, a determination means determines at least one of the image processing to be performed on the image of the first user and the image processing to be performed on the image of the second user. An information processing device characterized by having the following features.

2. The positional relationship between the second user and the display position of the first user in the second three-dimensional space viewed by the second user is determined according to the positional relationship between the first user and the display position of the second user in the first three-dimensional space. The information processing apparatus according to feature 1.

3. Based on the information, the determination means determines the image processing to be applied to the image of the first user. The aforementioned information processing device is First control means control the application of the image of the first user to the image determined by the determination means. Furthermore, it has The information processing apparatus according to feature 1.

4. A first display control means controls the display of the processed image of the first user to the second user. Furthermore, it has The information processing apparatus according to claim 3.

5. Based on the information, the determination means determines the image processing to be applied to the image of the second user. The aforementioned information processing device is A second control means controls the image of the second user to be subjected to the image processing determined by the determination means. Furthermore, it has The information processing apparatus according to feature 1.

6. A second display control means controls the display of the processed image of the second user to the first user. Furthermore, it has The information processing apparatus according to feature 5.

7. The image processing determined by the determination means includes at least one of blurring, sharpening, white balance, skin smoothing, and mosaic processing. The information processing apparatus according to feature 1.

8. The determination means determines an image processing method that reduces the visibility of details to a greater degree the shorter the distance from the first user to the second user's display position in the first three-dimensional space. The information processing apparatus according to feature 1.

9. The determination means determines an image processing method that improves the appearance to a greater degree the shorter the distance from the first user to the second user's display position in the first three-dimensional space. The information processing apparatus according to feature 1.

10. The positional relationship between the second user and the display position of the first user in the second three-dimensional space viewed by the second user is determined according to the positional relationship between the first user and the display position of the second user in the first three-dimensional space. The aforementioned information processing device is Based on the aforementioned information, a generation means generates an image that the first user views and that is placed in the second three-dimensional space, A third display control means controls the display of the image generated by the generation means to the first user. Furthermore, it has The information processing apparatus according to feature 1.

11. The system includes processing means for performing image processing on an image of a second user to be displayed to a first user, or an image of a first user to be displayed to a second user. The image processing differs depending on whether the distance from the first user to the second user's display position in the three-dimensional space viewed by the first user is a first distance or a second distance different from the first distance. An information processing device characterized by the following:

12. A step of obtaining information regarding the positional relationship between the first user and the display position of the second user in a first three-dimensional space viewed by the first user, A step of determining, based on the aforementioned information, at least one of the image processing to be performed on the image of the first user and the image processing to be performed on the image of the second user. A control method for an information processing device, characterized by having the following features.

13. A program for causing a computer to function as one of the means of an information processing apparatus described in any one of claims 1 to 11.

14. The computer is used as one of the means in the information processing apparatus described in any one of claims 1 to 11. A computer-readable storage medium containing the program required to run it.

Images