Image processing device, image processing method, and program

Abstract

To determine a play area in remote communication between a first user and a second user.SOLUTION: An image processing device of the present disclosure is an image processing device for implementing remote communication between a first user and a second user present in an environment different from the first user, and includes: acquisition means which acquires a first environmental information being information for detecting a three-dimensional structure around the first user, and second environmental information for detecting a three-dimensional structure around the second user, and determination means which determines a play area for remote communication of at least one of the first user and the second user on the basis of the first environmental information and the second environmental information acquired by the acquisition means.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present disclosure relates to a technology for controlling a user's movable range in a mixed reality (MR) space. [Background technology]

[0002] In recent years, development has been progressing on next-generation communication systems that utilize MR to display a 3D model of the other person directly in front of the user, making it feel as if the other person is actually there. For example, a camera or 3D sensor can capture a picture of the remote person in real time, and a 3D model of that person can be created based on the captured data. By displaying this in the MR space of a user wearing a head-mounted display (hereinafter referred to as HMD), the user can communicate with the remote person as if they were in the same space.

[0003] When using an HMD, a user may set a play area in advance, a range in which the user can move, based on surrounding walls and obstacles. Patent Document 1 discloses a technology in which, when using an HMD, objects around the user in real space are detected and the user's play area is set based on the objects. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2018-190432 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in remote communication using MR, a problem may occur in which the user's 3D model is displayed as being embedded in the wall of the other user's room. For example, consider a case in which a first user and a second user are remotely communicating with each other, and the first user is in a room larger than the second user. In the technology described in Patent Document 1, the first user's play area is set based on the position of the wall of the first user's room, so the first user can move up to the wall of the first user's room. However, if the second user's room is smaller than the first user's room, the position corresponding to the wall of the first user's room will be outside the second user's room. As a result, the first user's 3D model will be displayed as being embedded in the wall when viewed from the second user's perspective. [Means for solving the problem]

[0006] The image processing device of the present disclosure is an image processing device for remote communication between a first user and a second user who exists in an environment different from the first user, and includes an acquisition means for acquiring first environmental information, which is information for detecting a three-dimensional structure around the first user, and second environmental information, which is information for detecting a three-dimensional structure around the second user, and a determination means for determining a play area for the remote communication of at least one of the first user and the second user based on the first environmental information and the second environmental information acquired by the acquisition means. [Effects of the Invention]

[0007] The present disclosure allows for determining a play area for remote communication between a first user and a second user. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a configuration diagram of an HMD system as an image processing system. [Figure 2] FIG. 2 is a diagram illustrating the internal configuration of an HMD. [Figure 3]FIG. 1 is a diagram illustrating a hardware configuration of an image processing apparatus. [Figure 4] FIG. 1 is a diagram illustrating a functional configuration of an image processing apparatus according to a first embodiment. [Figure 5] 1 is a flowchart showing the overall flow of processing in the first embodiment. [Figure 6] FIG. 10 is a diagram illustrating the concept of a first play area determination process according to the first embodiment. [Figure 7] 10 is a flowchart of a first play area determination process according to the first embodiment. [Figure 8] FIG. 2 is a diagram showing a display example of a play area in the first embodiment. [Figure 9] FIG. 10 is a diagram showing another example of a play area determined in the first embodiment. [Figure 10] FIG. 10 is a diagram showing an example of a play area determined in Modification Example 1. [Figure 11] FIG. 10 is a diagram showing an example of a play area determined in Modification Example 2. [Figure 12] FIG. 10 is a diagram illustrating a functional configuration of an image processing apparatus according to a second embodiment. [Figure 13] FIG. 10 is a diagram illustrating the concept of a first play area determination process according to the second embodiment. [Figure 14] 10 is a flowchart of a first play area determination process according to the second embodiment. [Figure 15] FIG. 10 is a diagram showing another example of a play area determined in the second embodiment. [Figure 16] FIG. 10 is a diagram showing another example of a play area determined in the second embodiment. [Figure 17] FIG. 10 is a diagram illustrating a functional configuration of an image processing apparatus according to a third embodiment. [Figure 18] FIG. 11 is a diagram illustrating the concept of a first play area determination process according to the third embodiment. [Figure 19] 13 is a flowchart of a first play area determination process according to the third embodiment. [Figure 20] FIG. 11 is a diagram showing an example of a play area determined in a modified example of the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Note that the following embodiments do not limit the present disclosure, and not all of the combinations of features described in the embodiments are necessarily essential to the solutions of the present disclosure.

[0010] First Embodiment In the following embodiments, we will explain a situation in which two users in different rooms are wearing head-mounted displays (hereinafter referred to as HMDs) on their heads and are remotely communicating via a network. Multiple cameras and 3D sensors (not shown) are installed in each room, and a 3D model of the user is created based on the captured image data and displayed in real time on the HMD of the remote user. This creates an MR space in which the user feels as if the other user is in the same real space as the user.

[0011] The image processing system (HMD system 1) determines a play area for each user based on environmental information of the real space in which a first user (e.g., the user) exists and environmental information of the real space in which a second user (e.g., a communication partner) exists. In the first embodiment, an example will be described in which an image processing device connected to an HMD used by a first user determines the play area of ​​the first user based on the three-dimensional structure of the room in which the first user exists and the three-dimensional structure of the room in which the second user exists. The play area refers to the range in real space in which a real user can move. Note that the play area of ​​the second user is determined by executing the same process in an image processing device connected to an HMD used by a second user.

[0012] (Configuration of image processing device) 1 shows the configuration of an HMD system 1 as an image processing system in the present disclosure. The HMD system 1 includes an HMD 101 and an image processing device 102, which are connected via a transmission path 103 to communicate image data, control signals, and the like. The transmission path 103 includes a video signal line such as an HDMI (registered trademark) cable and a data signal line such as a USB cable. In addition, to receive input from a user, an input device such as a controller or keyboard (not shown) is communicatively connected to the image processing device 102. The communication connection between the HMD 101 and the image processing device 102 and the communication connection between the image processing device 102 and the input device may be a wired connection such as a USB cable, or a wireless connection such as Bluetooth (registered trademark).

[0013] FIG. 2 shows the internal configuration of the HMD 101. To achieve position tracking, the HMD 101 has multiple RGB cameras 201, 201, and an IMU (Inertial Measurement Unit) including a gyro sensor and an acceleration sensor (not shown). Furthermore, the HMD 101 has a distance sensor 202, such as a LiDAR, to acquire depth information. To display images, the HMD 101 also has displays 203, 203 corresponding to the left and right eyes. The displays 203, 203 are configured with display panels such as liquid crystal panels or organic EL panels. Furthermore, eyepieces 204, 204 corresponding to the left and right eyes are disposed in front of the displays 203, 203. A user of the HMD 101 can observe enlarged virtual images of the display images displayed on the displays 203, 203 through the eyepieces 204, 204. The HMD 101 is worn on the user's head by a band 104, and allows the user's left and right eyes to observe a left eye display image and a right eye display image (enlarged virtual image).

[0014] The image processing device 102 performs processing to generate a display image for the left eye and a display image for the right eye, and displays these images on the displays 203, 203 of the HMD 101. At this time, by providing an appropriate parallax between the display image for the left eye and the display image for the right eye, it is possible to give the user a visual perception with a sense of depth.

[0015] 2 indicate coordinate axes relating to the position and orientation of the HMD 101. In this embodiment, information relating to the position and orientation of the HMD 101 is obtained by setting the direction of gravity as the y direction, the direction of the user's line of sight as the z direction, and the direction perpendicular to the y direction and z direction as the x direction.

[0016] In this embodiment, the image processing device 102 is described as being an independent system configuration from the HMD 101, but the HMD 101 may be configured as an integrated HMD system including the image processing device 102 inside.

[0017] FIG. 3 is a diagram showing an example of the configuration of an image processing device 102 according to the present disclosure. The CPU 301 is a processor that performs overall control of each unit in the image processing device 102. The GPU 302 is a processor that performs image processing in response to instructions from the CPU 301. The GPU 302, for example, renders a virtual object, superimposes the virtual object on a real image acquired by the RGB camera 201 of the HMD 101, and creates a display image to be displayed on the HMD 101. The RAM 303 functions as the main memory, work area, etc. of the CPU 301. The ROM 304 stores a group of programs executed by the CPU 301. The HDD 305 stores applications executed by the CPU 301, data used in image processing, etc.

[0018] The general-purpose I / F 306 is a serial bus interface such as USB or IEEE1394, and is connected to an IMU or distance sensor provided in the HMD 101. This allows position and orientation information, a depth image of a target object, and the like to be acquired from the HMD 101. It is also used to acquire real-world images from the RGB camera 201 of the HMD 101. The output I / F 307 is an interface such as HDMI or DisplayPort, and is used to display images for display on the display 203 of the HMD 101. The network I / F 308 communicates with HMDs 101 used by others via a network such as a LAN or the Internet under the control of the CPU 301. The system bus 310 controls the flow of data within the device. The image processing device 102 may include components other than those described above.

[0019] (Functional configuration of image processing device) 4 is a diagram showing the functional configuration of the image processing device 102 in the first embodiment. The image processing device 102 has an acquisition unit 400, a play area determination unit 403, and a notification unit 404. The acquisition unit 400 includes a first environmental information acquisition unit 401 and a second environmental information acquisition unit 402. The play area determination unit 403 has a wall position detection unit 411, a wall distance calculation unit 412, and a determination unit 413. Each of these functional units is realized by the CPU 301 or GPU 302 executing processing based on a program stored in the ROM 304 or HDD 305.

[0020] The first environment information acquisition unit 401 acquires first environment information, which is information about the environment around a first user. The first user may be, for example, the user himself / herself.

[0021] The second environment information acquisition unit 402 acquires second environment information, which is information about the environment around the second user. The second user is, for example, a party with whom the first user is to perform remote communication, and exists in a different environment from the first user.

[0022] The environmental information is information for detecting the three-dimensional structure of the real space in which the user exists. In this embodiment, the environmental information acquisition units 401 and 402 acquire real images, depth images, and position and orientation information as the environmental information. The real images are images captured under visible light of the real space and are captured by the RGB cameras 201 and 201 of the HMD 101 used by the user. The real images are moving images captured at a predetermined frame rate. The depth images are images having distance information in the depth direction from the user's viewpoint position to an object and are acquired at a predetermined rate by the distance sensor 202 of the HMD 101. The position and orientation information is information detected at a predetermined rate by the IMU of the HMD 101 and is information regarding the position and orientation of the user. The image capture times of each frame of the real images and depth images correspond to the acquisition times of the position and orientation information.

[0023] The environmental information acquisition units 401 and 402 calculate a three-dimensional structure (environment map) around the user based on the acquired real image, depth image, and position and orientation information. This is obtained, for example, as three-dimensional point cloud data. One method for calculating the three-dimensional structure around the user is SLAM (Simultaneous Localization and Mapping; simultaneous execution of self-location estimation and environmental map creation). In this embodiment, as an example, a method called Visual SLAM is used, which acquires environmental information based on images acquired from a camera and an image sensor. Note that the method for acquiring environmental information is not limited to this, and SLAM technology using Lidar or other methods may also be used.

[0024] For example, the HMD 101 sets the origin (0,0,0) of the user's position to the position on the floor directly below the first detected position after startup. The HMD 101 also sets the direction of gravity as the height axis (y), the user's line of sight on a plane perpendicular to the height direction as the depth axis (z), and the direction perpendicular to the depth and height directions as the left-right axis (x). Regarding tilt (posture), the IMU detects the rotation angles in the roll, pitch, and yaw directions. When the HMD 101 starts the process of setting the play area, it instructs the user to look around. As a result, objects around the user, such as walls, floors, and ceilings, are captured as real images by the RGB cameras 201, 201. The IMU also acquires position and orientation information of the HMD 101 during image capture. Furthermore, the distance information between the object and the HMD 101 is acquired by the distance sensor 202.

[0025] The HMD 101 transmits the acquired real image, position and orientation information, and distance information (depth information) to the image processing device 102. The image processing device 102 calculates three-dimensional structural data of the user's surroundings using the above-mentioned Visual SLAM or the like based on the real image, position and orientation information, and distance information (depth information). The three-dimensional structural data is obtained as point cloud data indicating the three-dimensional positions of feature points of objects, including, for example, walls, floors, and ceilings.

[0026] The wall position detection unit 411 detects the positions of walls around the first user based on the first environmental information acquired by the first environmental information acquisition unit 401, i.e., the three-dimensional structural data around the first user. Also, the wall position detection unit 411 detects the positions of walls around the second user based on the second environmental information acquired by the second environmental information acquisition unit 402, i.e., the three-dimensional structural data around the second user. In this specification, a wall means a surface that stands approximately perpendicular to the floor.

[0027] The wall position detection unit 411 detects wall regions around the first user from the real image acquired by the first environment information acquisition unit 401 using an object detection algorithm such as YOLO. Then, the detected wall regions are associated with the 3D structure data calculated by Visual SLAM. This makes it possible to determine the 3D positions of multiple wall regions around the first user. Visual SLAM and YOLO are well-known technologies, so their description will be omitted. Note that the method for detecting wall regions is not limited to this, and any other method may be used. For example, CNN SLAM may be used to calculate the 3D structure data of a room and detect the wall regions. When the wall regions are detected, a room region in the 3D structure data is identified. In other words, the room region is the region surrounded by the wall regions.

[0028] Coordinates indicating the space of the room area (hereinafter simply referred to as the room) are held as data in a separate coordinate system that is independent of the viewpoint position or line of sight direction of the user (HMD 101). For example, the room coordinate system is set such that the center position of the room is the origin, the direction of gravity is the height (H) axis, the direction connecting the centers of a pair of opposing wall areas is the depth (D) axis, and the direction rotated 90 degrees horizontally from the depth direction is the left-right (W) axis. Note that the method of determining the origin and coordinate axes is not limited to this. For example, a marker or a predetermined feature point (e.g., one of the four corners of the room) previously installed in the real space of the room may be used as the origin. The coordinate axes may be determined depending on the application.

[0029] Similarly, for the second user's room, the wall position detection unit 411 detects the wall area around the second user based on the second environment information, thereby identifying the room area of ​​the second user.

[0030] The wall distance calculation unit 412 calculates the distance between opposing walls in at least two directions perpendicular to the height direction, based on the positions of the walls of the first user's room detected by the wall position detection unit 411. If the shape of the room is rectangular on the horizontal plane, pairs of opposing walls are detected in two perpendicular directions. These two directions are defined as the depth direction (first direction) and left-right direction (second direction) of the first user's room. The wall distance calculation unit 412 calculates the distance between the two opposing walls for each of the two directions. The size of the room is determined by the distances between the walls in these two directions.

[0031] Similarly, for the second user's room, the wall distance calculation unit 412 calculates the distance between opposing walls in at least two directions perpendicular to the height direction based on the positions of the walls of the second user's room detected by the wall position detection unit 411.

[0032] The determination unit 413 compares the distance between the opposing walls of the first user's room determined by the wall distance calculation unit 412 with the distance between the opposing walls of the second user's room in the two orthogonal directions. The determination unit 413 then determines the range of the play areas for the first user and the second user based on the smallest distance in each of the compared directions. The determination unit 413 places the play area having the determined range in the first user's room. This determines the first play area.

[0033] The first play area has a range in the height direction, a first direction perpendicular to the height direction (for example, the depth direction), and a second direction perpendicular to the height direction and the first direction (for example, the left-right direction). The process of determining the play area will be described in detail later.

[0034] The notification unit 404 notifies the first user of the first play area determined by the determination unit 413. An example of a notification method is displaying the first play area on a display. Specifically, the notification unit 404 displays a semi-transparent virtual object on the boundary between the inside and outside of the first play area. Note that the notification method is not limited to a visual method using a display, and other methods may also be used. For example, when the user approaches the boundary of the play area, the notification unit 404 may issue a warning (notification) by emitting a sound from a speaker (not shown) of the HMD 101 or emitting a vibration using a vibrator.

[0035] (Processing performed by the image processing device) FIG. 5 is a flowchart showing the overall processing flow of the first embodiment. The overall processing flow of the first embodiment executed by the image processing device 102 will be described using FIG. 5. The processing shown in this flowchart is written as a program readable by the CPU 301 and stored in the ROM 304 or HDD 305 of the image processing device 102. The program is called by the CPU 301, loaded into the RAM 303, and executed by the CPU 301. When the HMD 101 is started and communication is established between the HMD 101 and the image processing device 102, the CPU 301 starts this processing. In the following description, the symbol "S" means a step.

[0036] In S501, the CPU 301 (first environment information acquisition unit 401) acquires environment information (first environment information) around the first user. In this embodiment, the CPU 301 acquires, as the first environment information, a real image, a depth image, and position and orientation information captured and detected by the HMD 101 worn by the first user.

[0037] In S502, the CPU 301 (second environment information acquisition unit 402) acquires environment information (second environment information) around the second user. In this embodiment, the CPU 301 acquires, as the second environment information, a real image, a depth image, and position and orientation information captured and detected by the HMD 101 worn by the second user.

[0038] In S503, the CPU 301 (play area determination unit 403) determines a first play area, which is the play area of ​​the first user, based on environmental information about the surroundings of the first user and environmental information about the surroundings of the second user. In the first embodiment, the CPU 301 (play area determination unit 403) determines the first play area based on the size of the room of the first user and the size of the room of the second user. Details of the processing will be described later.

[0039] In S504, the CPU 301 (notification unit 404) notifies the first user of the first play area determined in S503. For example, the first play area is displayed on the displays 203, 203 of the HMD 101 used by the first user. As a method of displaying the first user, for example, a semi-transparent virtual object is displayed on the boundary surface between the inside and outside of the first play area.

[0040] (Details of the first play area determination process of the first embodiment) In the first play area determination process of the first embodiment, the first play area is determined based on the size of each user's room, which is determined from the first environmental information acquired in S501 and the second environmental information acquired in S502.

[0041] FIG. 6 is a diagram illustrating an overview of the first play area determination process of the first embodiment. The overview of the first play area determination process of the first embodiment will be described using this diagram. FIG. 6 shows a plan view of a room in which the direction perpendicular to the paper surface is the direction of gravity (the height direction of the room). A first user 601 and a second user 611 are in different rectangular rooms 600 and 610 in real space (FIGS. 6(a) and 6(b)). In this process, the CPU 301 first detects the positions of the walls of each room. Next, the CPU 301 calculates the distances W1 and D1 between opposing walls in the first user's room and the distances W2 and D2 between opposing walls in the second user's room. Since the user's rooms are assumed to be rectangular, the distances between opposing walls are calculated in two perpendicular directions. The size of the room is determined by the distances between the walls in these two directions. Finally, the CPU 101 compares the distances between the walls in the corresponding directions of each room and determines the range with the smallest value (distance) in each direction as the size of the first play area. In the case of FIG. 6, W1>W2 and D1>D2, and therefore W2 and D2 are the minimum values ​​in each direction, so the area inside the dashed line in FIG. 6(c) is determined as the range of the play area 605. In other words, the range P W =W2, and the depth range P D =D2. Note that this range is the maximum range of the first play area. CPU 301 may also determine any range below this maximum range as the range of the first play area, for example, by providing a further margin.

[0042] 7 is a flowchart showing the flow of the first play area determination process of the first embodiment. The flow of the first play area determination process of S503 will be described with reference to FIG.

[0043] In S701, the CPU 301 (wall position detection unit 411) detects the position of the wall of the first user's room based on the real image, depth image, and position and orientation information of the first user's room acquired in S501. In this embodiment, the wall position detection is performed as follows. First, the CPU 301 calculates the three-dimensional structure around the first user from the real image, depth image, and position and orientation information acquired from the HMD 101 of the first user using Visual SLAM. This is obtained, for example, as point cloud data indicating the three-dimensional positions of detected feature points.

[0044] The CPU 301 also detects wall regions from the real-world image using the object detection algorithm YOLO and associates them with the three-dimensional structure determined by Visual SLAM. As a result, data is obtained in which the positions of each feature point included in the point cloud data are linked to object identification labels (e.g., walls). This makes it possible to determine the three-dimensional positions of walls existing around the first user. Visual SLAM and YOLO are well-known technologies, so their explanation will be omitted.

[0045] In S702, the CPU 301 (wall position detection unit 411) detects the position of the wall of the second user's room based on the real image, depth image, and position and orientation information of the second user acquired in S502. The method for detecting the wall position is the same as in S701.

[0046] In S703, the CPU 301 (wall distance calculation unit 412) identifies a room area based on the positions of the walls of the first user's room detected in S701. Then, the CPU 301 calculates the distance between the walls in two perpendicular directions within a plane perpendicular to the height direction. The calculation of the distance between the walls is performed as follows. First, the walls detected in S701 are divided into planes. This can be done using a plane estimation algorithm such as RANSAC. Through this process, the positional relationship between the walls is detected.

[0047] Next, CPU 301 calculates the position of the center of gravity of each divided wall. Then, assuming that the position of a wall is equal to the position of the center of gravity of the wall, the distance between the wall farthest from the first user position and the wall opposite that wall is determined. For example, the distance between the walls in front of and behind the user is determined. This direction between the walls is defined as the first direction (depth direction). The distance between the walls in the first direction is calculated as the Euclidean distance between the positions in the depth direction and the left and right directions, ignoring the height direction position of the center of gravity of each wall. Furthermore, CPU 301 determines the distance between opposing walls in a direction rotated 90° from the first direction around the height direction as an axis, i.e., in a second direction (left and right direction) perpendicular to the first direction and the height direction. For example, the distance between the wall on the left side and the wall on the right side of the user position is defined as the distance between the walls in the second direction. The distance between the walls in the second direction is also calculated as the Euclidean distance between the positions in the depth direction and the left and right directions, ignoring the height direction position of the center of gravity of each wall.

[0048] The processing of S703 determines the distances D1 and W1 between the walls in the first and second directions, as shown in Fig. 6. In the above description, the distance between the walls in the first direction is D1, and the distance between the walls in the second direction is W1. Note that the left-right direction and the depth direction are expressions used for the purpose of explanation using the diagram, and do not necessarily have to coincide with the coordinate axes in the three-dimensional structure data (point cloud data) determined in S411.

[0049] In S704, the CPU 301 (wall distance calculation unit 412) calculates the distance between the walls in two perpendicular directions in a plane perpendicular to the height direction, based on the positions of the walls of the second user's room detected in S702. The method for calculating the distance between the walls in two perpendicular directions is the same as in S703. This process determines W2 and D2 in FIG. 6. W2 is the distance between the walls in the second direction, and D2 is the distance between the walls in the first direction.

[0050] In S705, the CPU 301 (determination unit 413) compares the distances W1 and D1 between the walls of the first user's room calculated in S703 with the distances W2 and D2 between the walls of the second user's room calculated in S704, and determines the ranges P W , P DSpecifically, the CPU 301 compares the distances between the walls in the first and second directions of the play area, and determines the smallest value as the range of the play area. That is, the range P of the play area in the first direction is D Let D1 be the minimum value of D2. Also, the range of the play area in the second direction P W Let be the minimum value of W1 and W2.

[0051] In S706, the CPU 301 (determination unit 413) determines the range P in the first direction and the second direction of the play area determined in S705. D , P W The first play area 605 is determined based on the above. This is determined as follows: The CPU 301 first determines the coordinates of the center of gravity of the centers of gravity of each wall determined in S703, i.e., the center of the first user's room. The CPU 301 then determines the center of the first user's room as the center position P0 of the first play area 605. The CPU 301 then determines a range of ±D2 / 2 in the first direction from the center position P0 and a range of ±W2 / 2 in the first direction as the range of the first play area 605. As a result, a rectangular area whose width in the first and second directions is the minimum value of the distance between the walls of the room in each direction is determined as the first play area 605. Note that the height range of the first play area 605 may be the entire range. The entire range refers to the entire height range in the data indicating the three-dimensional structure of the room, and includes at least the range from the floor to the ceiling of the first user's room 600.

[0052] 6, the minimum distance between the walls of the two rooms in the horizontal direction is W2, and the minimum distance in the depth direction is D2, and the size of the first play area 605 is determined as W2 × D2. If the center position of the first user's room 600 is set as the center position P0 of the first play area 605, the following range is determined as the first play area 605.

[0053] -W2 / 2 <W< +W2 / 2 -D2 / 2 <D< +D2 / 2

[0054] The rectangular area surrounded by points P1 (+W2 / 2, +D2 / 2), P2 (-W2 / 2, +D2 / 2), P3 (-W2 / 2, -D2 / 2), and P4 (+W2 / 2, -D2 / 2) shown in Figure 6(c) is the range of the first and second directions of the first play area 605.

[0055] The CPU 301 assigns label information identifying the first play area 605 to the coordinates corresponding to the first play area 605 in the data indicating the space of the room region of the first user's room 600.

[0056] Through the above processing, the first play area can be determined based on the environmental information surrounding the first user (i.e., the first user) as well as the environmental information surrounding the second user (i.e., the remote communication partner). Therefore, the first user's movable range, i.e., the first play area 605, can be determined so as to prevent the first user's 3D model from appearing embedded in the wall of the second user's room 610 when viewed from the second user's perspective. This prevents the first user's 3D model from appearing embedded in the wall of the other user's room even if the sizes of the first user's and the other user's rooms are different.

[0057] 8 shows an example of the first play area displayed on the HMD 101 used by the first user. A wall 800 existing in the real space is shown with a solid line, and a boundary surface 810 of the play area is shown with a dashed line. The boundary surface 810 of the play area is displayed semi-transparently in front of the wall 800 in the real space.

[0058] Note that by performing similar processing in the image processing device 102 connected to the HMD 101 used by the second user, the range in which the second user can move in the second user's room, i.e., the second play area, can be determined. According to the above processing, the range of the second play area becomes the same as the first play area. In the example shown in FIG. 6, since the size of the second user's room is smaller in both the depth direction and the left-right direction than the first user's room, the second play area is the entire second user's room.

[0059] Figure 9 shows another example of the room 900 of the first user 901 and the room 910 of the second user 911. As shown in Fig. 9(a), the size of the room 900 of the first user is W1 in the left - right direction and D1 in the depth direction. As shown in Fig. 9(b), the size of the room 910 of the second user is W2 in the left - right direction and D2 in the depth direction. Here, it is assumed that the room 910 of the second user is wider in the left - right direction (W1 < W2), and the room 900 of the first user is wider in the depth direction (D1 > D2). As a result of the processing of S703 described above, the range of the first play area 905 is P W = W1 in the left - right direction and P D = D2 in the depth direction. The range of the second play area 915 is also the same size as the size of the first play area 905, and is Q W = W1 in the left - right direction and Q D = D2 in the depth direction.

[0060] The first play area 905 in the room 900 of the first user, as shown in Fig. 9(c), when its center P O is set to the center of the room 900 of the first user, has the following range.

[0061] -W1 / 2 < W < +W1 / 2 -D2 / 2 < D < +D2 / 2

[0062] Therefore, the range within which the first user can move in the room 900 of the first user is the entire range of the room 900 of the first user in the left - right direction, and in the depth direction, it is a range narrower than the depth - direction range of the room 900 of the first user.

[0063] The second play area 915 in the room 9, as shown in Fig. 9(d), when its center Q O is set to the center of the room 910 of the second user, has the following range.

[0064] -W1 / 2 < W < +W1 / 2 -D2 / 2 < D < +D2 / 2

[0065] Therefore, the movable range of the second user in the second user's room 910 is the entire range of the second user's room 910 in the depth direction, and in the left - right direction, it is a range narrower than the left - right direction range of the second user's room.

[0066] <Modification Example 1 of the First Embodiment> FIG. 10 is a diagram for explaining Modification Example 1 of the first embodiment. As described above, in the first embodiment, the distances between the walls in the first direction and the second direction are compared, and the minimum value is used as the range of the first direction and the second direction of the play area. However, it is also possible to compare after swapping the directions. For example, in S705 of FIG. 7, the CPU 101 obtains the sum of differences diff0 when comparing the first direction with each other and the second direction with each other, and the sum of differences diff1 when comparing the first direction and the second direction alternately, as shown in the following (Equation 1) and (Equation 2). Then, the combination with the smaller sum of these differences is determined. And, as shown in (Equation 3) and (Equation 4), the first play area is determined based on the minimum value of the distance between the walls in the combination with the smaller sum of differences.

[0067] In the following (Equation 1) to (Equation 4), let the width of the first user's room in the left - right direction be W1, the depth of the first user's room be D1, the width of the second user's room in the left - right direction be W2, and the depth of the second user's room be D2. Also, let the range of the play area in the left - right direction be P W and the range in the depth direction be P D . Also, let min(a, b) be a function that determines the minimum value of a and b.

[0068] diff0 = |W1 - W2|+|D1 - D2| ··· (Equation 1) diff1 = |W1 - D2|+|D1 - W2| ··· (Equation 2) When diff0 < diff1, P W = min(W1, W2), P D = min(D1, D2) ··· (Equation 3) When diff0 ≧ diff1, P W = min(W1, D2), PD =min(D1,W2) (Equation 4)

[0069] FIG. 10 is a diagram illustrating the case where diff0≧diff1. As shown in FIG. 10(a), the size of the first user's room 1000 is W1 in the left-right direction and D1 in the depth direction, and as shown in FIG. 10(b), the size of the second user's room 1010 is W2 in the left-right direction and D2 in the depth direction. Also, let min(W1, D2)=D2 and min(D1, W2)=W2. In this case, the size P of the first play area 1005 is W , P D is determined as D2 in the left-right direction and W2 in the depth direction, as shown in Figure 10(c). The rectangular range surrounded by points P1 (+D2 / 2, +W2 / 2), P2 (-D2 / 2, +W2 / 2), P3 (-D2 / 2, -W2 / 2), and P4 (+D2 / 2, -W2 / 2) shown in Figure 10(c) is the first play area 1005.

[0070] <Modification 2 of the First Embodiment> FIG. 11 is a diagram illustrating a second modification of the first embodiment. As described above, in the first embodiment, the entire height range of the play area is considered. However, the heights of the first user's room and the second user's room may be compared, and the minimum value may be used as the height range of the play area. In this case, for example, in S701 and S702 of FIG. 7, the CPU 301 detects the floor and ceiling areas in addition to the wall areas from the three-dimensional structural data around each user. In S703 and S704, the CPU 301 calculates the height distance between the centers of gravity of the floor and ceiling areas. In S705, the CPU 301 determines the minimum height distance from the height H1 of the first user's room 1100 and the height H2 of the second user's room 1110. In S706, the CPU 301 determines the area from the floor of each room to the minimum height value min(H1,H2) determined in S705 as the height range of the first play area 1105.

[0071] As shown in FIG. 11(a), the size of the first user's room 1100 is W1 in the left-right direction, D1 in the depth direction, and H1 in the height direction. As shown in FIG. 11(b), the size of the second user's room 1110 is W2 in the left-right direction, D2 in the depth direction, and H2 in the height direction. The left-right and depth dimensions of the first play area 1105 are determined by the method of the first embodiment or any of its modifications. The height dimension of the first play area 1105 is determined to be the smaller of the height dimension H1 of the first user's room 1100 and the height dimension H2 of the second user's room 1110. In the example of FIG. 11, H1>H2, so the height dimension of the first play area 1105 is determined to be H2.

[0072] 11, the minimum value in the left-right direction is W2, the minimum value in the depth direction is D2, and the minimum value in the height direction is H2, so the size of the first play area 1105 is W2 x D2 x H2. The center position in the left-right and depth directions on the floor of the first user's room 1100 is set as the origin, and the upward direction is set as the positive direction in the height direction. In this case, the following range is determined as the first play area 1105.

[0073] -W2 / 2 <W< +W2 / 2 -D2 / 2 <D< +D2 / 2 0 <H< +H2

[0074] <Modification 3 of the First Embodiment> In the first embodiment, an example was shown in which the three-dimensional structural data of the second user's room and the distance between walls of the second user's room are calculated by the HMD system 1 used by the first user. However, this is not limiting. For example, in S502 of the flowchart in FIG. 5, the HMD system 1 used by the first user may acquire the distance between walls of the second user's room determined by the HMD system 1 of the second user as the second environment information. In this case, the processes of S702 and S704 in the flowchart in FIG. 7 are unnecessary. Alternatively, the HMD system 1 used by the first user may acquire three-dimensional structural data (point cloud data) of the second user's room determined by the HMD system 1 of the second user as the second environment information. This allows for efficient processing by omitting overlapping processes when the first user and the second user both execute the process of determining the play area.

[0075] <Second embodiment> In the first embodiment, a method for determining a play area for a first user based on the size of each user's room was described. However, the process described in the first embodiment is difficult to apply when the room shape is not rectangular. In the second embodiment, a method for determining a first play area that can be applied even when the room shape of each user is not rectangular will be described. Note that in the second embodiment, differences from the first embodiment will be mainly described, and similarities will not be described.

[0076] (Configuration of image processing device) The hardware configurations of the HMD system 1, HMD 101, and image processing device 102 according to the second embodiment are the same as those of the first embodiment, and therefore description thereof will be omitted.

[0077] (Functional configuration of image processing device) 12 is a diagram showing the functional configuration of an image processing device 102 according to the second embodiment. The image processing device 102 according to the second embodiment has an acquisition unit 1200, a play area determination unit 1203, and a notification unit 1204. The acquisition unit 1200 includes a first environmental information acquisition unit 1201 and a second environmental information acquisition unit 1202. The play area determination unit 1203 has a wall position detection unit 1211, a coordinate system conversion unit 1212, and a determination unit 1213.

[0078] The first environmental information acquisition unit 1201, the second environmental information acquisition unit 1202, the notification unit 1204, and the wall position detection unit 1211 are similar to the first environmental information acquisition unit 401, the second environmental information acquisition unit 402, the notification unit 404, and the wall position detection unit 411 of the first embodiment.

[0079] The coordinate system conversion unit 1212 converts the coordinates of the first user's room so that the first user's room and the second user's room overlap. The coordinate system conversion unit 1212 determines a coordinate system that maximizes the overlapping area between the first user's room and the second user's room.

[0080] The determination unit 1213 determines the overlapping area between the first user's room and the second user's room in the coordinate system determined by the coordinate system conversion unit 1212 as the first play area. Preferably, the first play area is determined so as to maximize the size of the overlapping area. The determination unit 1213 compares the size of the overlapping area in multiple states in which the first user's room area and the second user's room area are translated and rotated relative to each other while the heights of the floor surfaces of the first user's room area and the second user's room area are aligned. The size of the overlapping area is, for example, the area of ​​the overlapping area on the floor surface or a plane parallel to the floor surface.

[0081] The height range of the play area is the entire range, as in the first embodiment, or may be the shortest distance between the floor and ceiling in each user's room area.

[0082] (Processing performed by the image processing device) The overall processing flow in the second embodiment is the same as that in the first embodiment (FIG. 5). However, the details of the first play area determination process in S503 differ from those in the first embodiment. The details of the first play area determination process in the second embodiment will be described below.

[0083] (Details of the first play area determination process of the second embodiment) In the first play area determination process of the second embodiment, the CPU 301 (play area determination unit 1203) determines the first play area based on the overlapping area between the first user's room and the second user's room.

[0084] FIG. 13 is a diagram illustrating an overview of the first play area determination process in the second embodiment. In FIG. 13, the direction perpendicular to the paper surface indicates the direction of gravity (the height direction of the room). Assume that a first user 1301 and a second user 1311 are in different triangular rooms 1300 and 1310, respectively (FIGS. 13(a) and 13(b)). The CPU 301 first detects the positions of the walls of the rooms. Next, the CPU 301 calculates the coordinates of the center of the room based on the positions of the walls, sets the coordinate as the origin, and sets coordinate axes with the direction of gravity as the height direction (reference numerals 1302 and 1312 in FIG. 13). Thereafter, the CPU 301 overlaps the first user's room 1300 and the second user's room 1310 at the origin, finds the overlapping area between planes that pass through the origin and are parallel to the floor, and calculates the area of ​​the overlapping area. This process is repeated while rotating the first user's room 1300 by Δθ around the axis in the height direction to find the rotation angle that maximizes the area of ​​the overlapping region. In the example of FIG. 13, when the initial coordinate axis 1302 is rotated 180° as shown in FIG. 13(c), the first user's room 1300 and the second user's room 1310 completely overlap. Therefore, the area of ​​the overlapping region is maximized. The CPU 301 determines the overlapping region with the largest area as the first play area 1350. In other words, in the example of FIG. 13, the entire first user's room 1300 is determined as the first play area 1350. Note that this range is the maximum range of the first play area 1350. The CPU 301 may determine any range below this maximum range as the range of the play area, for example, to provide a margin.

[0085] 14 is a flowchart showing the flow of the first play area determination process in the second embodiment. The flow of the first play area determination process in the second embodiment will be described with reference to FIG.

[0086] In S1401, the CPU 301 (wall position detection unit 1211) detects the positions of the walls of the first user's room based on the real image, depth image, and position and orientation information of the first user's room acquired in S501. The method for detecting the positions of the walls is the same as in S701, and may use the above-mentioned Visual SLAM, the object detection algorithm YOLO, the plane estimation algorithm RANSAC, or the like. Through this process, the shape of the room is detected.

[0087] In S1402, the CPU 301 (wall position detection unit 1211) detects the position of the wall of the second user's room based on the real image, depth image, and position and orientation information of the second user's room acquired in S502. The method for detecting the wall position is the same as in S701.

[0088] In S1403, the CPU 301 (coordinate system conversion unit 1212) determines the initial values ​​of the coordinate systems for the first user's room and the second user's room. As the initial value, the center of gravity of the detected wall position, that is, the center of the room, is set as the origin of the coordinates. In addition, the initial coordinate axes are set such that the direction the user is facing at the time S1403 is started is the positive depth direction (+D), the direction rotated 90 degrees to the right from there is the positive left-right direction (+H), and the direction opposite to the direction of gravity is the positive height direction (+H). This process is executed for each of the first user's room and the second user's room.

[0089] In S1404, the CPU 301 (coordinate system conversion unit 1212) converts the coordinate system of the first user so as to maximize the overlapping area between the first user's room and the second user's room. In this embodiment, the rooms of the first user and the second user are first superimposed with the coordinate axes aligned with the origin set in S1403. Next, a plane of constant height passing through the origin, i.e., a plane parallel to the floor, is set, and the rotation angle Δθ, left-right movement amount ΔW, and depth movement amount ΔD that maximize the overlapping area of ​​each room on that plane are searched for. The search range is the range of the other user's room. That is, 0°≦Δθ≦360°, -W2 / 2≦ΔW≦+W2 / 2, and -D2 / 2≦ΔD≦+D2 / 2. The first user's room is then coordinate-converted into a coordinate system determined by the rotation angle Δθ, left-right movement amount ΔW, and depth movement amount ΔD that maximize the overlapping area of ​​each room.

[0090] In S1405, the CPU 301 (determination unit 1213) determines the overlapping area after the coordinate transformation in S1404 as the first play area. Note that, as for the height direction, the entire range may be determined as in the first embodiment, or the heights of the first user's room and the second user's room may be compared and the minimum value may be determined as the height direction range of the play area.

[0091] As described above, in the second embodiment, the first play area is determined based on the overlapping area between the first user's room and the second user's room, so the first play area can be determined even if the room shape is not rectangular. This prevents the player's 3D model from being displayed embedded in the wall of the other user's room, regardless of the shape of the room.

[0092] In the example of Figure 13, we have explained the case where the shapes and sizes of the rooms of the first user and the second user are the same, but according to the processing of the second embodiment, the play area can be set using the same processing method even if the shapes and sizes of the rooms of the first user and the second user are different.

[0093] FIG. 15 shows an example in which the shapes of the rooms of the first user and the second user are different and the overlapping area is determined by translation. As shown in FIG. 15(a), the size of the room of the first user is a triangle that fits within a rectangular frame with width W1 in the left-right direction and depth D1 in the depth direction. The initial coordinate axis 1502 has the center position of the floor surface as the origin 1503. Also, as shown in FIG. 15(b), the size of the room of the second user is a triangle that fits within a rectangular frame with width W2 in the left-right direction and depth D2 in the depth direction. Here, it is assumed that the maximum sizes in the left-right direction of each room are the same (W1 = W2), and the room of the second user is wider in the depth direction (D1 < D2). As a result of the processing of S1403 described above, when the room of the first user is translated by -ΔD, the overlapping area becomes the largest. In S1405, the CPU 301 determines the overlapping area when the room of the first user is translated (coordinate transformation) by -ΔD as the first play area 1550, as shown in FIG. 15(c). The origin 1504 of the coordinate axis after the coordinate transformation has moved by ΔD in the depth direction compared to the origin 1503 of the initial coordinate axis.

[0094] FIG. 16 shows an example in which the shapes of the rooms of the first user and the second user are different, and the overlapping area is determined by translational and rotational movements. As shown in FIG. 16(a), the size of the room of the first user is an ellipse that fits within a rectangular frame with width W1 in the left-right direction and depth D1 in the front-back direction. Also, as shown in FIG. 16(b), the size of the room of the second user is a triangle that fits within a rectangular frame with width W2 in the left-right direction and depth D2 in the front-back direction. Here, assume that the room of the first user is wider in the left-right direction (W1 > W2), and the room of the second user is wider in the front-back direction (D1 < D2). As a result of the processing of S1403 described above, when the room of the first user is rotated by Δθ from the initial coordinate axis 1602 and translated by ΔW in the left-right direction and ΔD in the front-back direction, it is assumed that the overlapping area becomes maximum. In S1405, the CPU 301 determines the overlapping area as the first play area 1650. According to the second embodiment, for example, even in rooms of various shapes such that the walls are curved surfaces, the overlapping area of the rooms of each user can be determined as the play area. The shape of the room is not limited to geometric shapes such as polygons, circles, and ellipses, and can be any shape. The present invention is also applicable to rooms having a shape with irregularities on a part of the wall.

[0095] <The Third Embodiment> In the third embodiment, a method for determining one's own (first) play area based on the obstacles in the room of the second user, who is the opponent, will be described.

[0096] (Configuration of the Image Processing Apparatus) The hardware configurations of the HMD system 1 and the image processing apparatus 102 according to the third embodiment are the same as those of the first embodiment.

[0097] (Functional Configuration of the Image Processing Apparatus) 17 is a diagram showing the functional configuration of an image processing device 102 according to the third embodiment. The image processing device 102 according to the third embodiment has an acquisition unit 1700, a play area determination unit 1703, a notification unit 1704, and an installation information acquisition unit 1706. The acquisition unit 1700 has a first environment information acquisition unit 1701, a second environment information acquisition unit 1702, and a determination criterion information acquisition unit 1705. The play area determination unit 1703 has an object detection unit 1711, a corresponding area calculation unit 1712, and a determination unit 1713.

[0098] The first environment information acquisition unit 1701, the second environment information acquisition unit 1702, the notification unit 1704, and the determination unit 1713 are similar to the first environment information acquisition unit 401, the second environment information acquisition unit 402, the notification unit 404, and the determination unit 413 of the first embodiment, and therefore description thereof will be omitted. In addition, in the third embodiment, the room areas of the first user and the second user are specified by the same method as in the first or second embodiment.

[0099] The determination criterion information acquisition unit 1705 acquires information that serves as a criterion for determining whether an object detected in the room of the second user is an obstacle. For example, when information about the size of the first user is used as the criterion, the determination criterion information acquisition unit 1705 receives input of the height and volume of the first user, that is, the first user himself / herself.

[0100] The installation information acquisition unit 1706 acquires information about the position where the 3D model of the first user is installed in the room of the second user.

[0101] The object detection unit 1711 detects an object around the second user based on the second environmental information acquired by the second environmental information acquisition unit 1702. The object detection unit 1711 also determines whether the detected object is an obstacle based on the determination criterion information acquired by the determination criterion information acquisition unit 1705. In this embodiment, information on the height of the first user is acquired as the determination criterion information. The object detection unit 1711 determines whether the detected object is an obstacle. For example, the object detection unit 1711 sets a value that is half the acquired height of the first user as a threshold, and determines that the detected object is an obstacle if the maximum value in the height direction of the detected object is greater than the threshold. The object detection unit 1711 assigns an identification label indicating that it is an obstacle area to the position of the obstacle area in the three-dimensional structure data of the second user's room.

[0102] The corresponding area calculation unit 1712 calculates an area in the first user's room that corresponds to an obstacle area in the second user's room. Hereinafter, an area in the first user's room that corresponds to an area in the second user's room is referred to as a corresponding area. The corresponding area of ​​the obstacle area in the second user's room is calculated based on the installation position of the 3D model of the first user in the second user's room and the actual position of the first user in the first user's room. More specifically, the corresponding area calculation unit 1712 determines a transformation matrix T that transforms the actual position (X1, Y1, Z1) of the first user into the installation position (X, Y, Z) acquired by the installation information acquisition unit 1706. Then, the transformation matrix T is used to perform coordinate transformation of the obstacle area in the second user's room. This makes it possible to calculate the corresponding area of ​​the obstacle area in the second user's room. Details will be described later.

[0103] The determination unit 1713 determines the first play area based on the corresponding area of ​​the obstacle area calculated by the corresponding area calculation unit 1712. Specifically, the determination unit 1713 determines the area obtained by excluding the corresponding area from the room area of ​​the first user as the first play area. Note that the corresponding area of ​​the obstacle area is not changed once determined. Regarding the height direction, the determination unit 1713 may determine the area obtained by excluding all areas in the height direction related to the corresponding area of ​​the obstacle area from the room area of ​​the first user as the play area of ​​the first user.

[0104] (Processing performed by the image processing device) The overall processing flow in the third embodiment is the same as that in the first embodiment (FIG. 5). However, the details of the first play area determination process in S503 differ from those in the first embodiment. The details of the first play area determination process in the third embodiment will be described below.

[0105] (Details of the first play area determination process of the third embodiment) In the third embodiment, the CPU 301 (play area determination unit 1703) determines the first play area to be the area after excluding the corresponding area of ​​the obstacle area in the second user's room detected from the second environmental information acquired in S502 from the first user's room.

[0106] Figure 18 is a diagram showing an overview of the first play area determination process in the third embodiment. In Figure 18, the direction perpendicular to the paper surface indicates the direction of gravity (the height direction of the room). It is assumed that a first user 1801 and a second user 1811 are in different rooms 1800 and 1810 in real space (Figures 18(a) and 18(b)). As shown in Figure 18, it is assumed that a coordinate system 1805 is set in the first user's room 1800, and a coordinate system 1815 is set in the second user's room 1810.

[0107] The CPU 301 (object detection unit 1711) first detects an object in the second user's room 1810 and determines whether the detected object is an obstacle. As shown in FIG. 18(c), it is assumed that an obstacle 1830 is determined to exist. Next, the CPU 301 (corresponding area calculation unit 1712) calculates an area of ​​the first user's room (corresponding area 1831) that corresponds to the obstacle area 1830 in the second user's room 1810. For example, it is assumed that the HMD 101 of the second user displays the 3D model 1820 of the first user at a predetermined position opposite the current position of the second user 1811. In this case, the corresponding area of ​​the obstacle 1830 in the first user's coordinate system is the position of the area 1831 in FIG. 18(d). Finally, the CPU 301 (determination unit 1713) determines the area 1802, which is the first user's room excluding the corresponding area, as the first play area.

[0108] 19 is a flowchart showing the flow of the first play area determination process in the third embodiment. The flow of the first play area determination process in the third embodiment will be described with reference to FIG.

[0109] In S1901, the CPU 301 (criterion information acquisition unit 1705) acquires criterion information for determining whether or not an object detected in the room of the second user is to be treated as an obstacle. In this embodiment, information on the height of the first user is acquired as the criterion information. The CPU 301 displays an input form for inputting the height of the first user on the display of the HMD 101 and accepts input.

[0110] In S1902, the CPU 301 aligns the coordinate systems of the first user's room and the second user's room. For example, the CPU 301 sets the origin at one of the four corners of the room. From there, along the wall of the room, the left-right direction is set as the X axis, the depth direction is set as the Y axis, and the height direction is set as the Z axis. Note that the method for setting the coordinate systems is arbitrary. For example, the origin may be the center of the room, or the position of a marker installed in advance in each room.

[0111] In S1903, the CPU 301 (object detection unit 1711) detects an obstacle present in the room of the second user based on the second environmental information acquired in S502. In this embodiment, the CPU 301 first acquires a three-dimensional structure around the second user using Visual SLAM, and then uses YOLO to distinguish and detect walls and object regions other than the walls. The CPU 301 then determines whether or not to treat the detected object region as an obstacle based on the determination criterion information. In this embodiment, the CPU 301 determines that an object region is an obstacle region if the length of the long side of the object region is equal to or greater than half the height of the first user acquired in S1901. Note that the method for determining the size of an object to be treated as an obstacle is not limited to this. For example, the size may be determined based on other information, such as the volume of the first user.

[0112] In S1904, the CPU 301 (installation information acquisition unit 1706) acquires information regarding the position where the second user installs the first user's 3D model (hereinafter referred to as installation position information). For example, the CPU 301 transmits a request to the HMD system 1 (image processing device 102) used by the second user to inquire about installation position information of the first user's 3D model. The HMD 101 used by the second user displays a message on the display prompting the second user to decide the installation position of the 3D model. Once the installation position of the first user's 3D model in the second user's room is decided by the second user, the HMD system 1 used by the second user transmits information about the installation position (X, Y, Z) to the HMD system 1 of the first user. Since the coordinate systems of the second user's room and the first user's room are aligned in S1902, the corresponding position where the first user's 3D model is installed in the first user's room is also acquired as (X, Y, Z).

[0113] In S1905, the CPU 301 calculates coordinate transformation to align the actual position (X1, Y1, Z1) of the first user in the room with the corresponding position (X, Y, Z) of the 3D model of the first user. Here, the posture is assumed to be a reference posture with no tilt. The position (X1, Y1, Z1) of the first user at a certain time t0 (for example, at the start) during execution of this flowchart is calculated by coordinate transformation of the position and posture information (x, y, z) detected by the HMD 101 to match it with the coordinate system set for the room. The CPU 301 determines a transformation matrix T to align the actual position (X1, Y1, Z1) of the first user with the corresponding position (X, Y, Z) of the 3D model.

[0114] In S1906, the CPU 301 (corresponding area calculation unit 1712) calculates a corresponding area in the first user's room that corresponds to the obstacle area in the second user's room. Specifically, the CPU 301 performs coordinate transformation on the obstacle area 1830 of the second user's room acquired in S1903 using the inverse transformation matrix T' of the transformation matrix T determined in S1905. As a result, a corresponding area 1831 of the obstacle area 1830 is calculated. The reference position of the obstacle area 1830 in the second user's room is set to (X2, Y2, Z2). The reference position (X3, Y3, Z3) of the corresponding area 1831 of the obstacle area 1830 relative to the actual position of the first user is calculated by coordinate transforming the reference position (X2, Y2, Z2) of the obstacle area 1830 using the inverse transformation matrix T'. An area of ​​the same size as the obstacle area 1830 based on the reference position (X3, Y3, Z3) is defined as a corresponding area 1831.

[0115] In S1907, the CPU 301 (determination unit 1713) determines, as the first play area, the area obtained by excluding the corresponding area 1831 calculated in S1906 from the first user's room 1800. Note that, in the height direction, the entire range of the corresponding area 1831 in the X direction and the Y direction may be excluded.

[0116] The above process allows the first user (the first user) to determine his / her own play area based on obstacles in the room of the second user (the other user). Therefore, the first user's movable range, i.e., the first play area, can be determined so that the second user does not see the first user's 3D model embedded in the wall of the second user's room 610. This prevents the first user's 3D model from being embedded in the wall of the other user's room even if there is an obstacle in the other user's room.

[0117] <Modification of the third embodiment> In the processing of the third embodiment described above, the image processing device 102 has shown an example in which the area in the first user's room corresponding to an obstacle in the second user's room is excluded from the first user's room. However, the excluded area may not only be the area corresponding to the obstacle, but also an area that is not visible to the second user, i.e., an area corresponding to a blind spot. In the HMD system 1 used by the first user, the play area determination unit 1703 identifies a blind spot that is not visible from the second user's viewpoint due to an object present in the second user's room area. Then, the play area determination unit 1703 determines the area corresponding to the blind spot in the first user's room area, and determines the area obtained by excluding the corresponding area from the first user's room area as the first user's play area.

[0118] Fig. 20 is a diagram illustrating blind spots and their corresponding areas. When an obstacle 2017 is detected in the room of a second user 2011 as shown in Fig. 20(a), a blind spot area 2016 is generated by the obstacle 2017 from the viewpoint of the second user 2011. Furthermore, even if the position of the obstacle 2017 remains the same, if the position or direction of the viewpoint of the second user changes, a blind spot area 2018 as seen by the second user 2011 will be an area different from the area 2016 shown in Fig. 20(a), as shown in Fig. 20(c).

[0119] After detecting an obstacle 2017 in the room 2010 of the second user, the CPU 301 (object detection unit 1711) determines a blind spot 2016 caused by the obstacle 2017 relative to the position and orientation information of the second user at a certain time t1. Similar to the processing of the third embodiment described above, the CPU 301 (corresponding area calculation unit 1712) determines a transformation matrix T for matching the actual position (X1, Y1, Z1) of the first user to the corresponding position (X, Y, Z) of the position where the 3D model 2020 of the first user is placed. Then, the CPU 301 performs coordinate transformation of the blind spot 2016 using the inverse transformation matrix T′. As a result, as shown in FIG. 20( b), a corresponding area 2006 in the room 2000 of the first user relative to the blind spot 2016 as seen by the second user is determined. Note that it is assumed that the coordinate systems of the room 2000 of the first user and the room 2010 of the second user are aligned.

[0120] Similar to the processing in the third embodiment described above, the CPU 301 (determination unit 1713) determines the first play area based on the corresponding area 2006 of the blind spot calculated by the corresponding area calculation unit 1712. Specifically, the CPU 301 determines the area of ​​the first user's room 2000 excluding the corresponding area 2006 as the first play area at time t1.

[0121] When the viewpoint position or line of sight of the second user changes, the CPU 301 determines the first play area at time t2. That is, the CPU 301 determines a blind spot 2018 (FIG. 20(c)) caused by an obstacle 2017 relative to the position and orientation information of the second user at time t2. As in the above-described process, the CPU 301 (corresponding area calculation unit 1712) determines a transformation matrix T2 for aligning the actual position of the first user with a corresponding position to the position where the 3D model 2020 of the first user is placed. Then, the CPU 301 performs coordinate transformation of the blind spot 2018 using the inverse transformation matrix T2′. As a result, a corresponding area 2008 in the first user's room relative to the blind spot 2018 as seen by the second user is determined, as shown in FIG. 20(d). The CPU 301 (determining unit 1713) determines the area obtained by excluding the blind spot corresponding area 2008 from the area of ​​the room area 2000 of the first user as the first play area at time t2.

[0122] Regarding the height direction, the determining unit 1713 may exclude all areas in the height direction that correspond to the blind spots from the room area of ​​the first user.

[0123] The above process determines the first user's movable range, i.e., the first play area, so that the first user's 3D model does not appear in a blind spot seen by the second user. This prevents the first user's 3D model from appearing in an area that is not visible to the other user (the second user).

[0124] As explained above, when users in different rooms are remotely communicating over a network, an appropriate play area can be set based on environmental information about not only the user's surroundings but also the other user's surroundings. This prevents problems such as the user's 3D model appearing embedded in the wall of the other user's room, and maintains the sense of realism in the MR space.

[0125] The present disclosure is not limited to the above-described embodiments, and may be implemented by combining the elements and concepts described in each embodiment. For example, as described in the first or second embodiment, the play area may be determined by excluding the area corresponding to the obstacle area in the other user's room from the play area determined by the wall distance and overlap area between the first user's room and the second user's room.

[0126] Furthermore, the location of the play area in the user's room is not limited to the center of the user's room, but may be any location within the user's room.

[0127] Furthermore, although the above-described embodiments illustrate examples of one-to-one communication in which one first user communicates with one second user, there may be two or more second users. For example, assume that three users A, B, and C, each in a different room, use the HMD system 1 and are connected to each other. In this case, the CPU 301 of the image processing device 102 of the HMD system 1 used by the first user A, obtains the environmental information of user B and the environmental information of user C as the environmental information of the second user. As described above, the environmental information may be a real image, a depth image, or position and orientation information, or it may be three-dimensional structural data of each room or distance between walls detected by the HMD system 1 used by each user. The CPU 301 calculates the positions of the walls in the rooms of users A, B, and C, and calculates the distances between the walls in the depth direction and the left and right directions. In this case, the CPU 301 determines the first play area using the following (Equation 5).

[0128] P W =min(W1,W2,W3), P D =min(D1,D2,D3) (Equation 5)

[0129] However, the size of the room of user A is W1 in the horizontal direction and D1 in the depth direction, the size of the room of user B is W2 in the horizontal direction and D2 in the depth direction, and the size of the room of user C is W3 in the horizontal direction and D3 in the depth direction. In this case, the horizontal size P of the first play areaW is the minimum value of each room in the horizontal direction, and the size in the depth direction P D is determined as the minimum value in the depth direction of each room.

[0130] As shown in the first modification of the first embodiment, the left-right direction and the depth direction of each room are interchanged to change the left-right size P W and the depth size P D In this case, it is preferable to determine the size of the play area so that it is the largest. Similarly, the second and third embodiments can also be applied to cases where there are two or more second users.

[0131] In the above-described embodiment, an example was shown in which the HMD system 1 used by the first user and the HMD system 1 used by the second user are connected for communication and perform the processes shown in the above-described flowcharts while transmitting and receiving information to each other, but this configuration is not limited to this. For example, a server that can be connected to the HMD systems 1 used by multiple users via a network may be provided, and the server may perform the processes shown in the above-described embodiments. The number of servers is not limited to one, and multiple servers may perform the processes in cooperation with each other.

[0132] Furthermore, although the above-described embodiment shows an example in which the first user and the second user each acquire surrounding environmental information using HMD101, the present disclosure is not limited to this, and the user may acquire surrounding environmental information using, for example, a camera, a smartphone, a tablet, or other device capable of capturing images.

[0133] <Other embodiments> The present disclosure can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0134] The disclosure of the above-described embodiment includes the following configurations.

[0135] (Configuration 1) An image processing device for remote communication between a first user and a second user who is in a different environment from the first user, comprising: an acquisition means for acquiring first environmental information, which is information for detecting a three-dimensional structure around the first user, and second environmental information, which is information for detecting a three-dimensional structure around the second user; a determination means for determining a play area for the remote communication of at least one of the first user and the second user based on the first environmental information and the second environmental information acquired by the acquisition means; An image processing device comprising:

[0136] (Configuration 2) The image processing device described in configuration 1, characterized in that the determination means determines the play area based on the size of the room in which the first user is present, which is determined based on the first environmental information, and the size of the room in which the second user is present, which is determined based on the second environmental information.

[0137] (Configuration 3) The determining means Detecting positions of a plurality of walls existing around the first user based on the first environmental information, and determining the size of a room in which the first user is present based on the distance between opposing walls; The image processing device according to configuration 2, characterized in that it detects the positions of multiple walls existing around the second user based on the second environmental information, and determines the size of the room in which the first user is present based on the distance between opposing walls.

[0138] (Configuration 4) The determining means The image processing device according to configuration 3, characterized in that the distance between the opposing walls is determined in at least two directions perpendicular to the height direction, and the distance between the opposing walls determined based on the first environmental information is compared with the distance between the opposing walls determined based on the second environmental information in at least two directions.

[0139] (Configuration 5) The determining means comparing the distance between the opposing walls determined based on the first environmental information with the distance between the opposing walls determined based on the second environmental information in a first direction, and determining the smallest distance as the range of the play area in the first direction; The image processing device according to configuration 3 or 4, characterized in that the distance between the opposing walls determined based on the first environmental information and the distance between the opposing walls determined based on the second environmental information are compared in a second direction perpendicular to the first direction, and the smallest distance is determined as the range of the second direction in the play area.

[0140] (Configuration 6) The determining means comparing a distance in a first direction between the opposing walls determined based on the first environmental information with a distance in a second direction perpendicular to the first direction between the opposing walls determined based on the second environmental information, and determining the smallest distance as the range in the first direction of the play area; The image processing device according to configuration 3 or 4, characterized in that the distance in the second direction between the opposing walls determined based on the first environmental information is compared with the distance in the first direction between the opposing walls determined based on the second environmental information, and the smallest distance is determined as the range in the second direction in the play area.

[0141] (Configuration 7) The determining means The image processing device according to configuration 1, characterized in that the play area is determined based on an overlapping area between the room of the first user determined based on the first environmental information and the room of the second user determined based on the second environmental information.

[0142] (Configuration 8) 8. The image processing device according to claim 7, wherein the determining means determines the area in which the overlapping area is the largest as the play area.

[0143] (Configuration 9) The determining means 9. The image processing device according to claim 7, wherein the size of the overlapping area is compared in a plurality of states in which the first user's room and the second user's room are moved parallel or rotated relative to each other.

[0144] (Configuration 10) The determining means The image processing device according to configuration 7 or 8, characterized in that, with the floor heights of the first user's room and the second user's room being the same, the size of the overlapping area is compared in a plurality of states in which the first user's room and the second user's room are moved parallel or rotated relative to each other.

[0145] (Configuration 11) 11. The image processing device according to any one of configurations 7 to 10, wherein the determining means compares the area of ​​the overlapping region on a floor surface or a plane parallel to the floor surface as the size of the overlapping region.

[0146] (Configuration 12) The image processing device according to any one of configurations 1 to 11, wherein the determining means sets the height direction range of the play area to a range that includes the entire height direction range of the room.

[0147] (Configuration 13) The image processing device according to any one of the first to eleventh embodiments, wherein the determining means determines the height direction range of the play area to be the shortest distance between the floor and ceiling of the room.

[0148] (Configuration 14) The determining means Identifying a room area of ​​the first user based on the first environment information; Identifying a room area of ​​the second user based on the second environment information and identifying an obstacle area in the room area of ​​the second user; determining a region in the room region of the first user that corresponds to the obstacle region; The image processing device according to any one of configurations 1 to 13, characterized in that an area after excluding an area corresponding to the obstacle area from the room area of ​​the first user is determined as the play area of ​​the first user.

[0149] (Configuration 15) The image processing device described in configuration 14, characterized in that the determination means determines an area corresponding to the obstacle area in the room area of ​​the first user based on information regarding an installation position of the 3D model of the first user in the room of the second user and actual position information of the first user in the room of the first user.

[0150] (Configuration 16) 16. The image processing device according to claim 14, wherein the determining means identifies the obstacle area in the room area of ​​the second user based on information about the size of the first user.

[0151] (Configuration 17) The image processing device described in configuration 14, characterized in that the determination means determines the excluding area obtained by excluding all vertical areas relating to the area corresponding to the obstacle area from the room area of ​​the first user as the play area of ​​the first user.

[0152] (Configuration 18) The determining means Identifying a room area of ​​the first user based on the first environment information; Identifying a room area of ​​the second user based on the second environment information, and identifying a blind spot that is obscured from a viewpoint of the second user by an object present in the room area of ​​the second user; determining an area corresponding to the blind spot in a room area of ​​the first user; The image processing device according to any one of configurations 1 to 13, characterized in that an area after excluding an area corresponding to the blind spot from the room area of ​​the first user is determined as the play area of ​​the first user.

[0153] (Configuration 19) The image processing device described in configuration 18, characterized in that the determination means determines the excluding area obtained by excluding all vertical areas corresponding to the blind spot from the room area of ​​the first user as the play area of ​​the first user.

[0154] (Configuration 20) 20. The image processing device according to any one of configurations 1 to 19, further comprising a display control means for displaying the play area determined by the determination means on a display visible to each user.

[0155] (Configuration 21) 21. The image processing device according to any one of configurations 1 to 20, further comprising a warning means for warning the user when the user approaches a boundary between the play area and the outside.

[0156] (Configuration 22) The image processing device according to any one of configurations 1 to 21, wherein the acquisition means acquires, as the first environmental information and the second environmental information, a real image captured from the viewpoint position of each user, a depth image, and position and orientation information of each user.

[0157] (Configuration 23) 23. The image processing device according to any one of configurations 1 to 22, wherein the first environmental information and the second environmental information are acquired by head-mounted displays worn by the user.

[0158] (Configuration 24) An image processing system including a head-mounted display and an image processing device that performs image processing for the head-mounted display, The image processing device includes: an acquisition means for acquiring first environmental information, which is information for detecting a three-dimensional structure around a first user acquired by the head-mounted display worn by the first user, and second environmental information, which is information for detecting a three-dimensional structure around the second user acquired by the head-mounted display worn by a second user who is in an environment different from that of the first user; a determination means for determining a play area of ​​at least one of the first user and the second user based on the first environmental information and the second environmental information acquired by the acquisition means; An image processing system comprising:

[0159] (Configuration 25) 1. An image processing method for remote communication between a first user and a second user who is in a different environment from the first user, comprising: acquiring first environmental information, which is information for detecting a three-dimensional structure around the first user, and second environmental information, which is information for detecting a three-dimensional structure around the second user; determining a play area for the remote communication of at least one of the first user and the second user based on the acquired first environment information and the acquired second environment information; An image processing method comprising:

[0160] (Configuration 26) A program for a computer to execute an image processing method for remote communication between a first user and a second user who is in an environment different from that of the first user, the program comprising: The image processing method includes: acquiring first environmental information, which is information for detecting a three-dimensional structure around the first user, and second environmental information, which is information for detecting a three-dimensional structure around the second user; determining a play area for the remote communication of at least one of the first user and the second user based on the acquired first environment information and the acquired second environment information; Programs including.

Claims

1. 1. An image processing device for remote communication between a first user and a second user who is in an environment different from that of the first user, comprising: an acquisition means for acquiring first environmental information, which is information for detecting a three-dimensional structure around the first user, and second environmental information, which is information for detecting a three-dimensional structure around the second user; a determination means for determining a play area for the remote communication of at least one of the first user and the second user based on the first environmental information and the second environmental information acquired by the acquisition means; An image processing device comprising:

2. The image processing device according to claim 1, characterized in that the determination means determines the play area based on the size of the room in which the first user is located, which is determined based on the first environmental information, and the size of the room in which the second user is located, which is determined based on the second environmental information.

3. The determining means Detecting positions of a plurality of walls present around the first user based on the first environmental information, and determining the size of a room in which the first user is present based on a distance between opposing walls; The image processing device according to claim 2, characterized in that the positions of multiple walls surrounding the second user are detected based on the second environmental information, and the size of the room in which the first user is present is determined based on the distance between opposing walls.

4. The determining means 4. The image processing device according to claim 3, wherein the distance between the opposing walls is determined in at least two directions perpendicular to the height direction, and the distance between the opposing walls determined based on the first environmental information is compared with the distance between the opposing walls determined based on the second environmental information in at least two directions.

5. The determining means comparing the distance between the opposing walls determined based on the first environmental information with the distance between the opposing walls determined based on the second environmental information in a first direction, and determining the smallest distance as the range of the play area in the first direction; The image processing device described in claim 3, characterized in that the distance between the opposing walls determined based on the first environmental information and the distance between the opposing walls determined based on the second environmental information are compared in a second direction perpendicular to the first direction, and the smallest distance is determined as the range of the second direction in the play area.

6. The determining means comparing a distance in a first direction between the opposing walls determined based on the first environmental information with a distance in a second direction perpendicular to the first direction between the opposing walls determined based on the second environmental information, and determining the smallest distance as the range in the first direction in the play area; The image processing device described in claim 3, characterized in that the distance in the second direction between the opposing walls determined based on the first environmental information is compared with the distance in the first direction between the opposing walls determined based on the second environmental information, and the smallest distance is determined as the range in the second direction in the play area.

7. The determining means The image processing device according to claim 1, characterized in that the play area is determined based on an overlapping area between the room of the first user determined based on the first environmental information and the room of the second user determined based on the second environmental information.

8. 8. The image processing device according to claim 7, wherein the determining means determines the area in which the overlapping area is largest as the play area.

9. The determining means 8. The image processing device according to claim 7, wherein the size of the overlapping area is compared in a plurality of states in which the first user's room and the second user's room are translated or rotated relatively to each other.

10. The determining means The image processing device according to claim 8, characterized in that the size of the overlapping area is compared in multiple states in which the first user's room and the second user's room are moved parallel or rotated relative to each other while the floor levels of the first user's room and the second user's room are aligned.

11. 11. The image processing apparatus according to claim 10, wherein the determining unit compares the area of ​​the overlapping region on the floor surface or on a plane parallel to the floor surface as the size of the overlapping region.

12. 8. The image processing device according to claim 2, wherein the determining means determines the range of the play area in the height direction to be a range that includes the entire range of the room in the height direction.

13. 8. The image processing device according to claim 2, wherein the determining means determines the height range of the play area to be the shortest distance between the floor and ceiling of the room.

14. The determining means Identifying a room area of ​​the first user based on the first environment information; Identifying a room area of ​​the second user based on the second environment information and identifying an obstacle area in the room area of ​​the second user; determining a region in the room region of the first user that corresponds to the obstacle region; The image processing device according to claim 1 , further comprising: determining a post-exclusion area, obtained by excluding an area corresponding to the obstacle area from the room area of ​​the first user, as the play area of ​​the first user.

15. The image processing device described in claim 14, characterized in that the determination means determines an area corresponding to the obstacle area in the room area of ​​the first user based on information regarding the installation position of the 3D model of the first user in the room of the second user and actual position information of the first user in the room of the first user.

16. The image processing apparatus according to claim 14 , wherein the determining means identifies the obstacle area in the room area of ​​the second user based on information about the size of the first user.

17. The image processing device described in claim 14, characterized in that the determination means determines the excluding area obtained by excluding all vertical areas relating to the area corresponding to the obstacle area from the room area of ​​the first user as the play area of ​​the first user.

18. The determining means Identifying a room area of ​​the first user based on the first environment information; Identifying a room area of ​​the second user based on the second environment information, and identifying a blind spot that is obscured from a viewpoint of the second user by an object present in the room area of ​​the second user; determining an area corresponding to the blind spot in the room area of ​​the first user; The image processing device according to claim 1 , wherein an area obtained by excluding an area corresponding to the blind spot from the room area of ​​the first user is determined as the play area of ​​the first user.

19. The image processing device described in claim 18, characterized in that the determination means determines an area after excluding all vertical areas corresponding to the blind spot from the room area of ​​the first user as the play area of ​​the first user.

20. 2. The image processing device according to claim 1, further comprising a display control means for displaying the play area determined by the determination means on a display that can be viewed by each user.

21. 2. The image processing device according to claim 1, further comprising a warning unit that warns the user when the user approaches a boundary between the play area and the outside.

22. The image processing device according to claim 1, characterized in that the acquisition means acquires, as the first environmental information and the second environmental information, a real image captured from each user's viewpoint position, a depth image, and position and orientation information of the user.

23. The image processing device according to claim 1 , wherein the first environmental information and the second environmental information are acquired by head-mounted displays worn by the user.

24. An image processing system including a head-mounted display and an image processing device that performs image processing for the head-mounted display, The image processing device includes: an acquisition means for acquiring first environmental information, which is information for detecting a three-dimensional structure around a first user acquired by the head-mounted display worn by the first user, and second environmental information, which is information for detecting a three-dimensional structure around the second user acquired by the head-mounted display worn by a second user who exists in an environment different from that of the first user; a determination means for determining a play area of ​​at least one of the first user and the second user based on the first environmental information and the second environmental information acquired by the acquisition means; An image processing system comprising:

25. 1. An image processing method for remote communication between a first user and a second user who is in an environment different from that of the first user, comprising: acquiring first environmental information, which is information for detecting a three-dimensional structure around the first user, and second environmental information, which is information for detecting a three-dimensional structure around the second user; determining a play area for the remote communication of at least one of the first user and the second user based on the acquired first environment information and the acquired second environment information; An image processing method comprising:

26. A program for a computer to execute an image processing method for remote communication between a first user and a second user who is in an environment different from that of the first user, the program comprising: The image processing method includes: acquiring first environmental information, which is information for detecting a three-dimensional structure around the first user, and second environmental information, which is information for detecting a three-dimensional structure around the second user; determining a play area for the remote communication of at least one of the first user and the second user based on the acquired first environment information and the acquired second environment information; Programs including.

Images