Information processing device, control method of information processing device, and program

Abstract

To provide a technology which can easily perform preparation of communication performed among a plurality of persons by using XR.SOLUTION: An information processing device comprises: setting means which displays a video of one or more persons in three dimensional space which a user sees to set a form of communication performed among the plurality of persons including the user; determination means which determines arrangement of the plurality of persons and photographing devices, on the basis of the set form of the communication, and a number of the photographing devices which photograph the user; and control means which controls to perform notification of the determined arrangement of the user and the photographing devices for the user.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to an information processing device, a control method for an information processing device, and a program, and more particularly to a system using technology such as virtual reality, augmented reality, or mixed reality. [Background technology]

[0002] There are technologies such as virtual reality (VR), mixed reality (MR), and augmented reality (AR). These technologies are called XR (Cross Reality). XR images are presented to the user using, for example, a head-mounted display (HMD), which is a head-mounted display device. A technology has also been proposed that captures in real time a three-dimensional image showing the appearance of a person in a remote location and presents it to the user. This technology can give the user the feeling that they are in the same space (three-dimensional space) as the person in the remote location, and enables spatial communication using the person's position, line of sight, and pointing. This technology is called Holoportation (Non-Patent Document 1).

[0003] When obtaining a 3D image showing the appearance of a person, in order to make the person visible from all angles and to allow the person to move around, a large number of sensors (imaging devices) are required to capture the person. However, depending on the person, it may not be possible to prepare a large number of sensors. Also, in places such as a home or an office, it may not be possible to install a large number of sensors.

[0004] On the other hand, depending on the form (type) of communication, it may be sufficient to communicate spatially using the person's position, gaze, and pointing, and it may not be necessary to make the person visible from all angles or to enable the person to move around. In this case, the number of sensors used can be reduced.

[0005] Patent Document 1 discloses a technique for visualizing the measurement range of a sensor. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2007-233971 A [Non-patent literature]

[0007] [Non-Patent Document 1] Holoportation: Virtual 3D Teleportation in Real-time, In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (pp. 741-754). ACM. Summary of the Invention [Problem to be solved by the invention]

[0008] However, the optimal placement of the sensor (imaging device) and multiple people in the three-dimensional space seen by the user varies depending on the form of communication. Therefore, even if the technology disclosed in Patent Document 1 is used, it is difficult for the user to smoothly and correctly set the placement of the sensor and multiple people.

[0009] The present invention aims to provide technology that makes it easy to prepare for communication between multiple people using XR. [Means for solving the problem]

[0010] A first aspect of the present invention is an information processing device characterized by having a setting means for setting a format of communication between a plurality of people including the user by displaying images of one or more people in a three-dimensional space viewed by the user, a determination means for determining an arrangement of the plurality of people and the imaging devices based on the set format of communication and the number of imaging devices that image the user, and a control means for controlling the device to notify the user of the determined arrangement of the user and the imaging devices.

[0011] A second aspect of the present invention is a method for controlling an information processing device, comprising the steps of: displaying images of one or more people in a three-dimensional space viewed by a user to set a format of communication between a plurality of people including the user; determining an arrangement of the plurality of people and the imaging devices based on the set format of communication and the number of imaging devices that image the user; and controlling the device to notify the user of the determined arrangement of the users and the imaging devices.

[0012] A third aspect of the present invention is a program for causing a computer to function as each of the means of the information processing device. Effect of the Invention

[0013] According to the present invention, it becomes easy to prepare for communication between multiple people using XR. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 is a diagram illustrating a system configuration. [Diagram 2] FIG. 2 is a block diagram showing a hardware configuration of the information processing device. [Diagram 3] 4 is a flowchart showing an operation of the information processing device. [Figure 4] FIG. 13 is a diagram showing a recommended arrangement of participants. [Diagram 5]FIG. 13 is a diagram showing a recommended placement of a 3D capture device and a method for determining the placement. [Figure 6] FIG. 13 is a diagram showing a recommended placement of a 3D capture device and a method for determining the placement. [Figure 7] 13 is a flowchart showing a placement adjustment process. [Figure 8] FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, an embodiment of the present invention will be described. In this embodiment, XR (Cross Reality) is used to easily prepare for communication between multiple people (multiple participants). Hereinafter, one of the multiple participants is called a user to distinguish it from the remaining participants. Each of the multiple participants can be the following user. The above communication is communication between multiple participants including the user by displaying images of one or more participants in a three-dimensional space seen by the user, and is, for example, a normal conversation, a meeting, or a business negotiation. The three-dimensional space seen by the user is, for example, a real space, a virtual space, or a space in which a real space and a virtual space are combined (mixed reality space or augmented reality space). In other words, the present invention is applicable to any of the cases where virtual reality (VR), mixed reality (MR), and augmented reality (AR) are used.

[0016] The preparation for communication may be interpreted as, for example, preparation for capturing an image of the user with an imaging device and transmitting the captured image to the outside, and includes the user setting up the imaging device and moving to a position where the user will communicate with other participants. In this embodiment, the information processing device determines the location of the user and the imaging device based on the form of communication, and This allows the user to easily understand the location of the user and the imaging device, and makes it easy to prepare for the above communication.

[0017] In this embodiment, an imaging unit provided in a head mounted display (HMD), which is a head mounted display device, captures a real space in the front direction of the HMD (a user wearing the HMD). Then, the HMD synthesizes images of other participants with the captured image of the real space, and presents (displays) the obtained composite image (an image in which the other participants have teleported, an MR image) to the user. This operation is performed with each participant as a user. However, the present invention is not limited to this configuration. For example, a VR image or an AR image may be presented to a certain participant instead of an MR image. A live-action image or an image of a virtual object such as an avatar may be used as the image of the participant. In addition, an XR image may not be presented to some participants. Note that although a plurality of participants are assumed to be in locations distant from each other, some participants (two or more participants) may be in the same location.

[0018] FIG. 1 is a diagram showing the configuration of a system according to this embodiment. The system shown in FIG. 1 has an information processing device 101, an HMD 112, and a three-dimensional capture device 102. The information processing device 101 may be a stationary computer such as a personal computer or a workstation, a smartphone or a wearable device, or a computer provided in the same housing as the HMD 112. Multiple functions of the information processing device 101 do not necessarily need to be executed in the same place (within one physical housing), and may be executed in different places. For example, some or all of the functions of the information processing device 101 may be executed on the cloud or in the HMD 112.

[0019] The 3D capture device 102 is an imaging device that captures an image of a user wearing the HMD 112. One or more 3D capture devices 102 are installed in the real space (location) where the user is present. The 3D capture device 102 is an imaging device capable of acquiring information on the three-dimensional shape of the user, and is, for example, a stereo camera. The 3D capture device 102 is not limited to a stereo camera, and may be, for example, a combination of a distance image sensor and a color image sensor.

[0020] The 3D reconstruction unit 103 performs 3D reconstruction based on information (information on the 3D shape of the user) obtained by the 3D capture device 102, and generates data of a reconstructed model (3D image) of the user. The method of 3D reconstruction is not particularly limited, and a known method may be used for 3D reconstruction. The format of the reconstructed model is not particularly limited, and for example, the reconstructed model may be formed of polygons, voxels, or a point cloud.

[0021] The transmission unit 104 transmits the data generated by the 3D reconstruction unit 103 (data of the reconstructed model (3D image) of the user) to the outside. The transmitted data is received, for example, by the devices of the other participants. In order to reduce the amount of data to be transmitted, the data may be compressed or thinned out.

[0022] The capture device position and orientation acquisition unit 105 acquires (calculates) information on the current location (e.g., position and orientation) of the 3D capture device 102. The location of the 3D capture device 102 may be estimated by an inside-out method or an outside-in method. An inside-out estimation method is, for example, a method using Simultaneous Localization and Mapping (SLAM). The location of the 3D capture device 102 is estimated based on the imaging results of the 3D capture device 102. In the outside-in method, for example, an external device such as a motion capture device (motion capture system) is used to estimate the 3D capture device 102. The position of the capture device 102 is estimated. The information on the position of the 3D capture device 102 may be information indicating the position in a world coordinate system with a predetermined position in the real space where the user is located as the origin, or may be information indicating the position in a coordinate system based on the user (HMD 112). The predetermined position may be, for example, the position of a marker placed in the real space, or the position of another object in the real space. Even if the information on the position of the 3D capture device 102 is not information on a coordinate system based on the user (HMD 112), when the information on the position is presented to the user, the information on the position is converted to information on a coordinate system based on the user.

[0023] The allocable area determination unit 106 acquires information about an area (guardian area) in which the user, the 3D capture device 102, and other participants (reconstructed models of other participants) can be arranged in the real space in which the user is located (the 3D space seen by the user). The acquisition method is not particularly limited, and a known method may be used to acquire the information. For example, information about the guardian area designated by the user may be stored in a memory, and the allocable area determination unit 106 may read the information from the memory. The 3D reconstruction unit 103 may generate a reconstructed model of the real space, and the allocable area determination unit 106 may calculate the guardian area based on the reconstructed model. For example, a floor area is calculated based on the reconstructed model of the real space, and the guardian area is calculated based on the floor area.

[0024] The recommended layout determination unit 107 determines a recommended layout of multiple participants including the user and the 3D capture devices 102 based on the type of communication to be performed and the number of 3D capture devices 102. The determination method will be described in detail later.

[0025] The CG generation unit 108 generates images (computer graphics (CG) images) of various objects. The CG generation unit 108 determines the arrangement (position and posture) of the objects to be generated based on information acquired by the user position and posture acquisition unit 110 (information on the current arrangement (position and posture) of the user (HMD 112)). For example, the CG generation unit 108 generates CG images (items) showing the recommended arrangement determined by the recommended arrangement determination unit 107.

[0026] The receiving unit 109 receives data of the reconstructed models (three-dimensional images) of the other participants.

[0027] The user position and orientation acquisition unit 110 acquires (calculates) information on the current position (position and orientation) of the user (HMD 112). The user's position may be estimated by an inside-out method or an outside-in method. An inside-out method is, for example, a method using SLAM, and the user's position is estimated based on the imaging results of an imaging unit 113 provided in the HMD 112. In the outside-in method, the user's position is estimated using an external device such as a motion capture device (motion capture system). The imaging unit 113 captures an image of the real space in front of the user (HMD 112).

[0028] The synthesis unit 111 synthesizes an image of real space captured by the imaging unit 113 with a CG image generated by the CG generation unit 108, a reconstructed model (3D image) of the other participants received by the receiving unit 109, or both, to generate a composite image. The composite image is displayed by the display unit 114 provided in the HMD 112 and presented to the user. The reconstructed models of the other participants may be synthesized in a recommended layout determined by the recommended layout determination unit 107, or may be synthesized in a layout specified by the user.

[0029] 2 is a block diagram showing the hardware configuration of the information processing device 101. A CPU 201 comprehensively controls each device connected via a bus 200. The information processing device 101 reads out and executes a program stored in a read-only memory (ROM) 202. Various programs such as an operating system, a processing program of a flowchart described later, and a device driver are stored in the ROM 202. These programs are temporarily stored in a random access memory (RAM) 203 and are executed by the CPU 201 as appropriate. The input I / F 204 inputs a signal from an external device (for example, an imaging device such as the 3D capture device 102, or an operation device) to the information processing device 101. The input I / F 204 may appropriately convert the signal from the external device into a signal in a format that can be processed by the information processing device 101. The output I / F 205 outputs a signal to an external device (for example, a display device such as the HMD 112). The signal output to the external device is, for example, a signal generated by the information processing device 101. The output I / F 205 may appropriately convert the signal output to the external device into a signal in a format that can be processed by the external device.

[0030] FIG. 3 is a flowchart showing the operation of the information processing device 101.

[0031] In step S301, the CPU 201 sets (determines) a communication format according to an instruction from a user (user operation). The communication format includes, for example, the number of participants, the composition of the participant groups, the presence or absence of content (shared content) shared between participants (persons), the size of the shared content, the type of the shared content, and some or all of the relationships between the participants. The method of setting the communication format is not particularly limited, and for example, the CPU 201 may receive information on the communication format specified by another participant via the input I / F 204 and set the communication format according to the information. The CPU 201 may display a dialogue that can be interacted with the user on the HMD 112 to allow the user to specify the communication format.

[0032] In step S302, the CPU 201 (recommended placement determination unit 107) determines a recommended placement of the multiple participants (and shared contents) based on the set communication format. The recommended placement of the multiple participants may be determined by any method based on the communication format.

[0033] 4(A) to 4(D) show an example of a recommended layout of multiple participants (and shared content). For simplicity, the recommended layout in a plane perpendicular to the height direction (a plane parallel to the floor) will be described here.

[0034] For example, when the number of participants is N (N is an integer of 2 or more), there is no group, and there is no shared content, the CPU 201 determines a recommended layout as shown in Fig. 4(A). An N-polygon is set, and the positions of N vertices of the N-polygon are determined as the positions of the N participants, respectively. Then, the orientations (postures) of the N participants are determined so that the N participants face the center (or center of gravity) of the N-polygon. In Fig. 4(A), the number of participants is three (participants 401 to 403), and the positions of three vertices of an equilateral triangle 404 with a side length of length L are determined as the positions of the three participants 401 to 403, respectively. Then, the orientations of the participants 401 to 403 are determined so that the participants 401 to 403 face the center of the equilateral triangle 404.

[0035] When there is a group including N participants and a group including M participants (M is an integer of 2 or more) and there is no shared content, the CPU 201 determines a recommended arrangement as shown in FIG. 4(B).

[0036] N-1 triangles are set. When multiple triangles are set as N-1 triangles, the multiple triangles share one vertex and adjacent triangles share one side. This sets up an N+1 polygon consisting of N-1 triangles. Similarly, M-1 triangles is set, and an M+1 polygon consisting of M-1 triangles is set. The center (or centroid) of the N+1 polygon, the vertex shared by the N-1 triangles, the center (or centroid) of the M+1 polygon, and the vertex shared by the M-1 triangles are located on a single line. The direction from the center (or centroid) of the N+1 polygon to the vertex shared by the N-1 triangles is opposite to the direction from the center (or centroid) of the M+1 polygon to the vertex shared by the M-1 triangles. The center (or centroid) of the N+1 polygon and the center (or centroid) of the M+1 polygon are the same.

[0037] Then, of the N+1 vertices of the N+1 polygon, the positions of N vertices excluding the vertex shared by the N-1 triangles are determined as the positions of the N participants. The orientations (postures) of the N participants are determined so that they face the vertex shared by the N-1 triangles. Similarly, of the M+1 vertices of the M+1 polygon, the positions of M vertices excluding the vertex shared by the M-1 triangles are determined as the positions of the M participants. The orientations (postures) of the M participants are determined so that they face the vertex shared by the M-1 triangles.

[0038] In FIG. 4B, there is a group including three participants 405 to 407 and a group including two participants 409 and 410. A quadrangle 411 is set, which is made up of two isosceles triangles having equal sides of length L1 and bases of length L2. The two isosceles triangles share a vertex of the apex angle and share one side. An isosceles triangle 412 is set, which has equal sides of length L1 and a base of length L2. The axis of line symmetry of the quadrangle 411 and the axis of line symmetry of the isosceles triangle 412 coincide with each other. The center of the range on the axis of line symmetry of the quadrangle 411 and the center of the range on the axis of line symmetry of the isosceles triangle 412 coincide with each other at a position 413. The direction from the position 413 to the vertex of the apex angle of the quadrangle 411 (two isosceles triangles) is opposite to the direction from the position 413 to the vertex of the apex angle of the isosceles triangle 412.

[0039] Of the four vertices of rectangle 411, the positions of three vertices excluding the vertex of the apex of rectangle 411 are determined as the positions of three participants 405 to 407. The orientations of participants 405 to 407 are determined so that they face the vertex of the apex of rectangle 411. Similarly, of the three vertices of isosceles triangle 412, the positions of two vertices excluding the vertex of the apex of isosceles triangle 412 are determined as the positions of two participants 409 and 410. The orientations of participants 409 and 410 are determined so that they face the vertex of the apex of isosceles triangle 412.

[0040] When the number of participants is N, there is no group, and there is shared content, the CPU 201 determines a recommended layout as shown in FIG. 4C. N-1 triangles are set. When multiple triangles are set as the N-1 triangles, the multiple triangles share one vertex, and adjacent triangles share one side. In this way, an N+1 polygon consisting of N-1 triangles is set. Then, of the N+1 vertices of the N+1 polygon, the position of the vertex shared by the N-1 triangles is determined as the position of the shared content, and the positions of the remaining N vertices are determined as the positions of the N participants. The orientation (posture) of the shared content is determined so that it faces the N vertices (N participants), and the orientation (posture) of the N participants is determined so that it faces the vertex (shared content) shared by the N-1 triangles.

[0041] In FIG. 4C, the number of participants is three (participants 401 to 403). A quadrangle 414 is set, which is made up of two isosceles triangles having equal sides of length L1 and bases of length L2. The two isosceles triangles share a vertex at the apex and one side. Of the four vertices of quadrangle 414, the position of the vertex at the apex of quadrangle 414 is determined as the position of shared content 415, and the positions of the remaining three vertices are determined as the positions of the three participants 401 to 403. The orientation of shared content 415 is determined so as to face the above three vertices (participants 401 to 403), and the orientation of participants 401 to 403 is determined so as to face the vertex at the apex of the apex of quadrangle 414 (shared content 415). The process of determining the position and orientation of the shared content 415 may be a process of determining the position and orientation of a bounding box 416 of the shared content 415. For example, the center position of the front rectangle of the bounding box 416 may be aligned with the vertices of the apex corners of the quadrangle 414. When the front rectangle of the bounding box 416 is oriented toward the above three vertices (participants 401 to 403), the central axis of the front rectangle (an axis passing through the center position of the front rectangle and perpendicular to the front rectangle) may be aligned with the linear symmetry axis of the quadrangle 414. The bottom surface of the bounding box 416 may be aligned with the floor surface.

[0042] When there is a group including N participants and a group including M participants, and there is shared content, the CPU 201 determines a recommended arrangement as shown in Fig. 4(D). As in the case of Fig. 4(B), the positions and orientations (postures) of the N participants and the positions and orientations (postures) of the M participants are determined. Then, a position at a distance L3 from the center or center of gravity of the N+1 polygon (center or center of gravity of the M+1 polygon) in a direction perpendicular to a line passing through a vertex shared by the N-1 triangles and a vertex shared by the M-1 triangles is determined as the position of the shared content. The orientation (posture) of the shared content is determined so that it faces the center or center of gravity of the N+1 polygon (center or center of gravity of the M+1 polygon).

[0043] In Fig. 4(D), there is a group including three participants 405 to 407, and a group including two participants 409 and 410. As in the case of Fig. 4(B), the positions and orientations of the three participants 405 to 407 and the positions and orientations of the two participants 409 and 410 are determined. Then, a position at a distance L3 from position 413 in a direction perpendicular to the axis of linear symmetry of rectangle 411 and isosceles triangle 412 is determined as the position of shared content 415, and the orientation of shared content 415 is determined to face position 413.

[0044] The distance between the participants may or may not be determined in advance. For example, the distances L and L1 may be a general social distance (for example, 1.2 to 3.5 m). The distance L2 may be a distance shorter than the general social distance (for example, 0.7 to 2 m). The distance between the participants may be determined according to the relationship between the participants. For example, when two participants are lovers or family members, a smaller distance may be determined as the distance between the two participants compared to when they are not lovers or family members. Similarly, the distance L3 may or may not be determined in advance. The distance L3 may be determined according to the size of the shared content. For example, when the size of the shared content is large, a shorter distance may be determined as the distance L3 compared to when the size of the shared content is small. Since the preferred arrangement (position and posture) of the shared content depends on the type of the shared content, the arrangement of the shared content may be determined taking the type of the shared content into consideration. The preferred arrangement of the shared content may change during communication. For example, when a participant who gives an explanation using the shared content changes, the preferred arrangement of the shared content also changes. Therefore, the arrangement of the shared content may be changed automatically or manually during communication.

[0045] Returning to the description of Fig. 3, in step S303, the CPU 201 (recommended placement determination unit 107) determines a recommended placement of the 3D capture devices 102 based on the set communication format and the number of 3D capture devices 102. For example, the CPU 201 determines a recommended placement of the 3D capture devices 102 based on the recommended placement of multiple participants (and shared content) and the number of 3D capture devices 102.

[0046] During communication, the positions of the participants may vary due to body movements such as gestures and hand movements, fluctuations in standing position, or both. In addition, there is a limit to the range that can be captured by the three-dimensional capture device 102. Therefore, when determining the arrangement of the multiple participants and the three-dimensional capture device 102, the characteristics of the positional fluctuations of the multiple participants, the imaging angle of the three-dimensional capture device 102, or both may be further taken into consideration. In this embodiment, It is assumed that the PU 201 determines the recommended placement of the three-dimensional capture device 102 by further taking into consideration the characteristics of the positional fluctuations of the multiple participants and the imaging angle of the three-dimensional capture device 102 .

[0047] In addition, it is preferable that the difference (perceptual difference) between how the user looks from other participants (recommended arrangement) and how the user looks from the three-dimensional capture device 102 (recommended arrangement) is small. In other words, it is preferable that the three-dimensional capture device 102 obtains an image with a high degree of reproducibility of how the user looks from other participants (recommended arrangement). The reproducibility of the view from a location different from the location of the other participants may be low. Therefore, in this embodiment, the CPU 201 determines the recommended arrangement of the three-dimensional capture device 102 (and multiple participants) so that the perceptual difference corresponding to the other participants is smaller than a threshold value (so that the reproducibility corresponding to the other participants is larger than a threshold value). If the number of other participants (participants who are not the user) is two or more, the CPU 201 calculates the perceptual difference (reproducibility) for each of the two or more other participants. Then, the CPU 201 determines the recommended arrangement of the three-dimensional capture device 102 so that a value obtained by combining the calculated perceptual differences is smaller than a threshold value (so that a value obtained by combining the calculated reproducibility is larger than a threshold value). Note that the determination method is not limited to this. For example, the CPU 201 may determine a recommended placement of the 3D capture device 102 for each of two or more other participants so that the perceptual difference is smaller than a threshold (so that the reproducibility is larger than a threshold).

[0048] A method for determining a recommended placement of the 3D capture device 102 will be described with reference to Figures 5(A) to 5(D). Figures 5(A) and 5(C) show an example of a recommended placement of multiple participants and the 3D capture device 102. Figures 5(B) and 5(D) show an example of a method for determining a recommended placement of the 3D capture device 102.

[0049] 5A, the number of participants is three (user 501, other participants 502, 503), there is no group, and there is no shared content. There is one three-dimensional capture device 102. In this case, the recommended placement of the three-dimensional capture device 102 is determined so that the three-dimensional capture device 102 faces the user 501 from a position between the participants 502 and 503.

[0050] The horizontal axis in Figure 5(B) indicates the angle from the recommended orientation of user 501. This angle can indicate the positions of other participants (the directions in which other participants are located) and the position of the 3D capture device 102 (the direction in which the 3D capture device 102 is placed). Assume that the 3D capture device 102 is pointed toward user 501. The recommended position of participant 502 is at an angle θ from the recommended position of user 501, and the recommended position of participant 503 is at an angle -θ from the recommended position of user 501.

[0051] First, a frequency model 512 is generated that indicates the frequency with which the participant 502 is present for each position (direction) taking into account the positional fluctuation of the participant 502. Similarly, a frequency model 513 is generated that indicates the frequency with which the participant 503 is present for each position taking into account the positional fluctuation of the participant 503. When looking at the frequency models 512 and 513, the vertical axis in FIG. 5(B) indicates frequency. In the frequency model 512, the frequency is maximum at an angle θ, and the frequency decreases linearly as it moves away from the angle θ. In the frequency model 513, the frequency is maximum at an angle -θ, and the frequency decreases linearly as it moves away from the angle -θ.

[0052] Next, a reproducibility model 514 is generated, which indicates the reproducibility corresponding to the imaging angle of view (imaging range) of the 3D capture device 102. When looking at the reproducibility model 514, the vertical axis of FIG. 5(B) indicates the reproducibility. In the image obtained by the 3D capture device 102, the reproducibility of the user's appearance from the position corresponding to the center of the imaging angle of view (optical axis) is the highest. In FIG. 5(B), the 3D capture device 102 is placed in the direction of an angle of 0 degrees. Reproducibility In the degree model 514, the reproducibility is maximum at an angle of 0 degrees, and decreases linearly as the angle moves away from 0 degrees.

[0053] Then, the size of the area where the frequency model 512 and the realism model 514 overlap is calculated as the realism of how the user appears to the participant 502. Similarly, the size of the area where the frequency model 513 and the realism model 514 overlap is calculated as the realism of how the user appears to the participant 503. A recommended placement of the 3D capture device 102 is determined so that a combined value of these realisms is greater than a threshold. For example, a recommended placement of the 3D capture device 102 is determined so that the sum of these realisms is maximized.

[0054] The size of the area of ​​the frequency model 512 that is not overlapped by the realism model 514 may be calculated as the perceptual difference corresponding to the participant 502 (the difference between how the user looks from the participant 502 and how the user looks from the 3D capture device 102). Similarly, the size of the area of ​​the frequency model 513 that is not overlapped by the realism model 514 may be calculated as the perceptual difference corresponding to the participant 503 (the difference between how the user looks from the participant 503 and how the user looks from the 3D capture device 102). Then, the recommended placement of the 3D capture device 102 may be determined so that the combined value of these perceptual differences is smaller than a threshold value.

[0055] The frequency model and the reproducibility model are not limited to those described above, for example, the frequency model, the reproducibility model, or both may have a normal distribution. Also, the accuracy of the frequency model, the reproducibility model, or both may be improved by taking into account some or all of the 3D reconstruction algorithm, the characteristics of the 3D reconstruction, and the shape of the participant.

[0056] As in FIG. 5(A), in FIG. 5(C), the number of participants is three (user 501, other participants 502, 503), there is no group, and there is no shared content. Unlike FIG. 5(A), in FIG. 5(C), the number of three-dimensional capture devices 102 is two. In this case, as shown in FIG. 5(D), reproducibility models 514-1 and 514-2 corresponding to the two three-dimensional capture devices 102 (three-dimensional capture devices 102-1 and 102-2), respectively, are generated. Then, the recommended placement of the three-dimensional capture devices 102-1 and 102-2 is determined in the same manner as described with reference to FIG. 5(B). In FIG. 5(C), the recommended placement of the three-dimensional capture device 102-1 is determined so that the device faces the user 501 from a position on a straight line passing through the user 501 and the participant 502. Similarly, the recommended placement of the three-dimensional capture device 102-2 is determined so that it faces the user 501 from a position on a straight line passing through the user 501 and the participant 503.

[0057] When the distance from the user to the three-dimensional capture device 102 is short, the user may be captured in high definition, but the range of the user captured may be narrow. When the distance from the user to the three-dimensional capture device 102 is long, the range of the user captured may be wide, but the user may not be captured in high definition. Therefore, in this embodiment, the CPU 201 determines the arrangement of the user and the three-dimensional capture device 102 so that a range obtained by adding a predetermined margin to a specific range of the user is captured. More specifically, the CPU 201 determines the shortest distance at which the specific range and the area to which a predetermined margin is added are captured as the distance from the user to the three-dimensional capture device 102. The specific range, the predetermined margin, or both may be determined according to the communication format. The specific range may be the entire range of the user, the range of the upper body, or the range of the face.

[0058] The reproducibility (perceptual difference) may be weighted. The CPU 201 assigns a weight corresponding to a specific participant among two or more other participants, and weights corresponding to the remaining participants. Alternatively, the CPU 201 may perform weighted synthesis of the reproducibility (perceptual difference) using a weight smaller than the weighting used for the weighted synthesis of the reproducibility. The CPU 201 may then determine a recommended arrangement of the multiple participants and the 3D capture device 102 such that the value obtained by the weighted synthesis of the reproducibility is greater than the threshold (the value obtained by the weighted synthesis of the perceptual difference is smaller than the threshold).

[0059] 6(A) and 6(B), a method for performing weighted synthesis to determine a recommended placement of the 3D capture device 102 will be described. Fig. 6(A) shows an example of a recommended placement of multiple participants and the 3D capture device 102. Fig. 6(B) shows an example of a method for determining a recommended placement of the 3D capture device 102.

[0060] In Fig. 6(A), there is a group including a user 601 and participants 602 and 603, and a group including participants 604 and 605, and there is no shared content. There are two 3D capture devices 102. In this case, the basic method for determining the recommended placement of the 3D capture devices 102 is similar to the method described using Figs. 5(B) and 5(D).

[0061] 6B, four frequency models 612 to 615 are generated corresponding to the four participants 602 to 605, respectively. Then, reproducibility models 616-1 and 616-2 are generated corresponding to the two 3D capture devices 102 (3D capture devices 102-1 and 102-2), respectively.

[0062] In communication between groups, it is preferable that participants can see participants of other groups more accurately than participants of the same group. In other words, it is preferable that the reproducibility of the user's appearance from participants of other groups is higher than the reproducibility of the user's appearance from participants of the same group as the user. Therefore, as shown in FIG. 6(B), weighting is performed on frequency models 612, 613 corresponding to participants 602, 603 of the same group as the user to reduce the frequency. After that, the recommended placement of the three-dimensional capture devices 102-1, 102-2 is determined by the same method as described using FIG. 5(B) and 5(D). By using the frequency models 612, 613 after weighting, the recommended placement of the three-dimensional capture devices 102-1, 102-2 is determined by prioritizing the reproducibility of the user's appearance from participants 604, 605 of other groups. In FIG. 6(B), the recommended placement of the three-dimensional capture device 102-1 is determined so that the three-dimensional capture device 102-1 faces the user 601 from a position on a straight line passing through the user 601 and the participant 604. Similarly, the recommended placement of the 3D capture device 102-2 is determined so that it faces the user 601 from a position on a line passing through the user 601 and the participant 605. Note that weighting may be performed to reduce the size (value) of the area where the frequency model 612 (before weighting) and the realism model overlap, and the size of the area where the frequency model 613 (before weighting) and the realism model overlap.

[0063] When there is shared content, the recommended placement of the three-dimensional capture device 102 can be determined in a manner similar to that described above. Weighting may be performed taking into consideration the type of shared content, whether or not the participant is a presenter, or both. For example, a small value may be set as the weight of the reproducibility (perceptual difference) for a participant who only views the shared content. A large value may be set as the weight of the reproducibility (perceptual difference) for a presenter.

[0064] Although the process of determining the recommended placement of the multiple participants (step S302) and the process of determining the recommended placement of the three-dimensional capture device 102 (step S303) are performed separately, the recommended placements may be determined collectively based on the various information described above. The multiple participants may be positioned such that the center (or center of gravity) of a polygon formed by the positions of the multiple participants and the three-dimensional capture device 102 approximately coincides with the center (or center of gravity) of the guardian area. and the 3D capture device 102 may be determined.

[0065] Returning to the description of Fig. 3, in step S304, the CPU 201 (recommended layout determination unit 107) performs layout adjustment processing. In the layout adjustment processing, the determined recommended layout (the recommended layout of the multiple participants and the 3D capture device 102) is adjusted in consideration of the size and shape of the room.

[0066] 7 is a flowchart showing the arrangement adjustment process. Here, it is assumed that the CPU 201 (arrangement possible area determination unit 106) has already acquired information on the guardian area.

[0067] In step S701, CPU201 determines whether all recommended layouts of multiple participants and 3D capture device 102 are within the guardian area. The method of determination is not particularly limited, and a known method may be used for the determination. If the recommended layouts are within the guardian area, the layout adjustment process ends, and if not (if at least one of multiple participants and 3D capture device 102 is placed outside the guardian area), proceed to step S702. When proceeding to step S702, CPU201 may obtain (calculate) information on the amount of protrusion from the guardian area for objects (reconstructed models of participants or 3D capture device 102) placed outside the guardian area.

[0068] In step S702, the CPU 201 adjusts the recommended arrangement of the multiple participants and the three-dimensional capture device 102 so that all of the multiple participants and the three-dimensional capture device 102 are within the guardian area (are arranged inside the guardian area). This process may use the information on the overhang amount described above. For example, when a participant is arranged outside the guardian area, the CPU 201 moves the participant closer to the other participants, and when the three-dimensional capture device 102 is arranged outside the guardian area, the CPU 201 moves the three-dimensional capture device 102 closer to the user. It is assumed that a lower limit is set for the distance between the participants and the distance from the user to the three-dimensional capture device 102. For example, the lower limit of the distance from the user to the three-dimensional capture device 102 is the shortest distance at which an important part of the user, such as the upper body or face, is captured. It is assumed that the CPU 201 adjusts the recommended arrangement so that the distance between the participants and the distance from the user to the three-dimensional capture device 102 do not fall below the lower limit. It is assumed that the recommended arrangement can be adjusted so that all of the multiple participants and the three-dimensional capture device 102 are within the guardian area. In this case, for example, the CPU 201 may adopt the arrangement that maximizes the overlap area between the polygon formed by the positions of the multiple participants and the 3D capture device 102 and the guardian area as the recommended arrangement after adjustment.

[0069] In step S703, CPU 201 determines whether the recommended layout has been adjusted so that all of the participants and the 3D capture device 102 fit within the guardian area. If the recommended layout has been adjusted, the layout adjustment process ends; if not, the process proceeds to step S704. For example, if a participant who is located outside the guardian area is not located inside the guardian area even when moved to a position that is the minimum distance from the other participants, the process proceeds to step S704. If a 3D capture device 102 who is located outside the guardian area is not located inside the guardian area even when moved to a position that is the minimum distance from the user, the process also proceeds to step S704.

[0070] In step S704, the CPU 201 notifies the user of the defect, such as some objects being placed outside the guardian area. For example, the CPU 201 may notify the user that other participants are displayed inside a wall, that another object is present in the recommended position of the 3D capture device 102, or that a part is missing from the user's reconstructed model. After that, for example, the recommended placement before the adjustment is adopted, Proceed to step S305 in FIG. 3. At this time, the reconstructed model of the user may not be transmitted to the outside. The start of communication may be canceled. Another communication (communication in another mode) such as an audio conference without video may be started. A dialogue that can be interacted with the user may be displayed on the HMD 112 to inquire the user about which of these operations to perform.

[0071] Returning to the description of Fig. 3, in step S305, the CPU 201 notifies the user of a recommended arrangement of the user and the 3D capture device 102. At this time, for example, the CPU 201 controls the HMD 112 to display an item indicating the recommended arrangement of the user and the 3D capture device 102 in the real space in which the user is located (the 3D space seen by the user). At this time, reconstructed models of other participants may be displayed in the recommended arrangement of the other participants. The arrangement of the other participants may be changeable by the user.

[0072] FIG. 8 shows an example of a displayed item. In FIG. 8, an item 801 (virtual object) indicating a recommended three-dimensional arrangement of the three-dimensional capture device 102 is displayed in the air. The item 801 is an item that imitates the shape of the three-dimensional capture device 102. Also, considering that a tripod is used to install the three-dimensional capture device 102, an item 802 (mark) indicating a recommended two-dimensional arrangement of the three-dimensional capture device 102 (recommended arrangement in a plane perpendicular to the height direction (in a plane parallel to the floor surface)) is displayed on the floor surface. The item 802 includes an arrow indicating the direction of the three-dimensional capture device 102. An item 803 (caption) that prompts the user to arrange the three-dimensional capture device 102 is displayed in association with the item 801. By looking at the items 801 to 803, the user can easily install the three-dimensional capture device 102 in the recommended arrangement.

[0073] 8, an item 804 (mark) indicating a two-dimensional recommended placement for the user is displayed on the floor. The item 804 includes an arrow indicating the user's direction. An item 805 (caption) encouraging the user to move is displayed in association with the item 804. By looking at the items 804 and 805, the user can easily adopt the recommended placement (easily move to the recommended position and face the recommended direction).

[0074] Note that the items are not limited to those described above. Item 802, item 803, or both may be omitted. Item 805 may be omitted.

[0075] CPU 201 may end the notification in step S305 when the arrangement of the user and three-dimensional capture device 102 substantially matches the recommended arrangement. For example, items 801 to 803 may be hidden when the arrangement of the three-dimensional capture device 102 substantially matches the recommended arrangement. Information acquired by capture device position and orientation acquisition unit 105 (information on the current arrangement of the three-dimensional capture device 102) is used to determine whether the arrangement of the three-dimensional capture device 102 substantially matches the recommended arrangement. Items 804 and 805 may be hidden when the arrangement of the user substantially matches the recommended arrangement. Information acquired by user position and orientation acquisition unit 110 (information on the user's current arrangement) is used to determine whether the arrangement of the user substantially matches the recommended arrangement.

[0076] As described above, according to this embodiment, a recommended arrangement of multiple participants and three-dimensional capture devices is determined based on the communication format and the number of three-dimensional capture devices (imaging devices). Then, the recommended arrangement of the user and the three-dimensional capture devices is notified to the user. This makes it easy to prepare for communication between multiple people using XR.

[0077] The number of participants may change (increase or decrease). When the number of participants changes, The processes of steps S301 to S305 may be re-executed if the number of participants has decreased, and steps S301 to S305 may not be re-executed if the number of participants has increased.

[0078] Changing the arrangement of the three-dimensional capture device takes time, and it is difficult to communicate during the change. Therefore. When re-determining the recommended arrangement of the multiple participants and the three-dimensional capture device after a change in the number of participants, it is preferable not to change the recommended arrangement of the three-dimensional capture device. If the processing of step S704 in FIG. 7 becomes necessary due to an increase in the number of participants, the addition of the participant may be canceled. For the new participant, communication only with voice may be started without displaying the reconstructed model. A small value may be set as the weight of the reproducibility (perceptual difference) for the new participant so that the reproducibility of the user's appearance from the original participants is prioritized. A dialogue that can be interacted with the user may be displayed on the HMD 112 to inquire the user about which of these actions to perform.

[0079] In addition, in communication, it is preferable that the arrangement of multiple participants is shared among the multiple participants. An example will be described with reference to FIG. 5(A). Assume that images of participants 502 and 503 are displayed to user 501 so that participants 502 and 503 are present at the positions shown in FIG. 5(A). In this case, images of user 501 and participant 503 may be displayed to participant 502 so that participants 501 and 503 are present at the positions shown in FIG. 5(A). Images of user 501 and participant 502 may be displayed to participant 503 so that participants 501 and 502 are present at the positions shown in FIG. 5(A). This enables spatial communication using people's positions, gazes, and pointing.

[0080] The above-described embodiment (including the modified examples) is merely an example, and the present invention also includes configurations obtained by appropriately modifying or changing the configurations of the above-described embodiment within the scope of the gist of the present invention. The present invention also includes configurations obtained by appropriately combining the configurations of the above-described embodiment.

[0081] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0082] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) A setting means for setting a form of communication between a plurality of people including the user by displaying images of one or more people in a three-dimensional space viewed by the user; a determination means for determining an arrangement of the plurality of persons and the imaging devices based on the set communication format and the number of imaging devices that image the user; a control means for controlling the notification of the determined arrangement of the user and the imaging device to the user; 13. An information processing device comprising: (Configuration 2) The imaging device is capable of acquiring information of a three-dimensional shape of the user, The information processing device includes: A generating means for generating data of a three-dimensional image of the user based on the information obtained by the imaging device; A transmitting means for transmitting the data of the three-dimensional image to the outside; Further having 2. The information processing device according to configuration 1. (Configuration 3) The control means controls to display, as the notification, an item indicating the determined arrangement of the user and the imaging device in the three-dimensional space. 3. The information processing device according to configuration 1 or 2. (Configuration 4) The form of communication includes at least one of the number of people, the composition of a group of people, the presence or absence of content shared between people, the size of the content, the type of the content, and the relationship between people. 4. The information processing device according to any one of configurations 1 to 3. (Configuration 5) The information processing device according to any one of configurations 1 to 4, wherein the determination means, in determining the positioning of the plurality of people and the imaging device, further takes into account at least one of the characteristics of positional fluctuations of the plurality of people and the imaging angle of view of the imaging device. (Configuration 6) an acquisition means for acquiring information regarding an area in the three-dimensional space in which the plurality of persons and the imaging device can be arranged; and When the determination means determines an arrangement of the plurality of people and the imaging device such that at least one of the plurality of people and the imaging device is located outside the area, the determination means adjusts the determined arrangement such that all of the plurality of people and the imaging device are located inside the area. 6. The information processing device according to any one of configurations 1 to 5. (Configuration 7) The control means performs control so as to give a second notification to the user when it is not possible to adjust the positions of the plurality of people and the imaging device so that all of the plurality of people and the imaging device are positioned inside the area. 7. The information processing device according to configuration 6. (Configuration 8) The one or more persons may be two or more persons, The determining means determines an arrangement of the plurality of persons and the imaging device such that a combined value of differences between how the user appears from each of the two or more persons and how the user appears from the imaging device is smaller than a threshold value. 8. The information processing device according to any one of configurations 1 to 7. (Configuration 9) The determination means determines an arrangement of the plurality of persons and the imaging device such that a weight corresponding to a specific person among the two or more persons is smaller than weights corresponding to the remaining people among the two or more persons, and the weighted combination value of a difference between how the user appears from each of the two or more persons and how the user appears from the imaging device is smaller than the threshold value. 9. The information processing device according to configuration 8. (Configuration 10) The specific person includes at least one of a person in the same group as the user and a person who has newly participated in the communication. 10. The information processing device according to configuration 9. (Configuration 11) The determining means determines, when a change occurs in the number of people who are communicating, an arrangement of the plurality of people who are communicating and the imaging device after the change. 11. The information processing device according to any one of configurations 1 to 10. (Configuration 12) The control means performs control so as to end the notification when an arrangement of the user and the imaging device substantially coincides with the determined arrangement. 12. The information processing device according to any one of configurations 1 to 11. (Configuration 13) The control means controls to display images of the one or more people in the three-dimensional space in the determined arrangement of the one or more people. 13. The information processing device according to any one of configurations 1 to 12. (Configuration 14) The determining means determines a position of the user and the imaging device so that an image of a specific range of the user plus a predetermined margin is captured. 14. The information processing device according to any one of configurations 1 to 13. (method) A step of displaying an image of one or more people in a three-dimensional space viewed by a user and setting a form of communication between a plurality of people including the user; determining an arrangement of the plurality of people and the imaging devices based on the set communication format and the number of imaging devices that image the user; a step of controlling the notification of the determined arrangement of the user and the imaging device to the user; 13. A method for controlling an information processing apparatus comprising the steps of: (program) A program for causing a computer to function as each of the means of the information processing device according to any one of configurations 1 to 14. [Explanation of symbols]

[0083] 101: Information processing device 201: CPU 107: Recommended layout determination unit 108: CG generation unit 111: Synthesis unit

Claims

1. a setting means for setting a form of communication between a plurality of people including the user by displaying images of one or more people in a three-dimensional space viewed by the user; a determination means for determining an arrangement of the plurality of persons and the imaging devices based on the set communication format and the number of imaging devices that image the user; a control means for controlling the notification of the determined arrangement of the user and the imaging device to the user; 13. An information processing device comprising:

2. the imaging device is capable of acquiring information of a three-dimensional shape of the user, The information processing device includes: A generating means for generating data of a three-dimensional image of the user based on the information obtained by the imaging device; A transmitting means for transmitting the three-dimensional image data to an outside. Further having 2. The information processing apparatus according to claim 1,

3. The control means controls to display, as the notification, an item indicating the determined arrangement of the user and the imaging device in the three-dimensional space.

2. The information processing apparatus according to claim 1,

4. The form of communication includes at least one of the number of people, the composition of a group of people, the presence or absence of content shared between people, the size of the content, the type of the content, and the relationship between people.

2. The information processing apparatus according to claim 1,

5. 2 . The information processing apparatus according to claim 1 , wherein the determining means, in determining the arrangement of the plurality of people and the imaging device, further takes into consideration at least one of a characteristic of positional fluctuation of the plurality of people and an imaging angle of view of the imaging device.

6. an acquisition means for acquiring information regarding an area in the three-dimensional space in which the plurality of persons and the imaging device can be arranged; and When the determination means determines an arrangement of the plurality of people and the imaging device such that at least one of the plurality of people and the imaging device is located outside the area, the determination means adjusts the determined arrangement such that all of the plurality of people and the imaging device are located inside the area.

2. The information processing apparatus according to claim 1,

7. The control means performs control so as to give a second notification to the user when it is not possible to adjust the arrangement of the plurality of persons and the imaging device so that all of the plurality of persons and the imaging device are arranged inside the area.

7. The information processing apparatus according to claim 6,

8. The one or more persons may be two or more persons; The determining means determines an arrangement of the plurality of persons and the imaging device such that a combined value of differences between how the user appears from each of the two or more persons and how the user appears from the imaging device is smaller than a threshold value.

2. The information processing apparatus according to claim 1,

9. The determination means determines an arrangement of the plurality of persons and the imaging device such that a weight corresponding to a specific person among the two or more persons is smaller than weights corresponding to the remaining people among the two or more persons, and the weighted combination value of a difference between how the user appears from each of the two or more persons and how the user appears from the imaging device is smaller than the threshold value.

9. The information processing apparatus according to claim 8,

10. The specific person includes at least one of a person in the same group as the user and a person who has newly participated in the communication.

10. The information processing apparatus according to claim 9,

11. The information processing apparatus according to claim 1 , wherein, when there is a change in the number of persons communicating, the determination means determines an arrangement of the plurality of persons communicating and the imaging device after the change.

12. The control means performs control so as to end the notification when an arrangement of the user and the imaging device substantially coincides with the determined arrangement.

2. The information processing apparatus according to claim 1,

13. The control means controls to display images of the one or more people in the three-dimensional space in the determined arrangement of the one or more people.

2. The information processing apparatus according to claim 1,

14. The determining means determines a position of the user and the imaging device so that an image of a specific range of the user plus a predetermined margin is captured.

2. The information processing apparatus according to claim 1,

15. A step of displaying an image of one or more people in a three-dimensional space viewed by a user and setting a form of communication between a plurality of people including the user; determining an arrangement of the plurality of people and the imaging devices based on the set communication format and the number of imaging devices that image the user; a step of controlling the notification of the determined arrangement of the user and the imaging device to the user; 13. A method for controlling an information processing apparatus comprising the steps of:

16. A program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 14.

Images