Information processing device, control method for information processing device, and program

Abstract

This will eliminate discrepancies in the relative positions of avatars, allowing for more natural-looking interactions between them. [Solution] Information on the difference in body size between the user and the avatar applied by the user is obtained, as well as the coordinate information of the user's joint points. Then, it is determined from the coordinate information of the user's joint points whether or not there is a contact action between the users, and if there is a contact action, the contact point between the users is calculated. At this time, if the difference in body size between the user who performed the contact action and their avatar is greater than or equal to a threshold, the contact point between the avatars is calculated taking the difference in body size into consideration, and the final avatar posture is determined by adjusting the avatar's posture.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, a control method for the information processing apparatus, and a program.

Background Art

[0002] In recent years, services for distributing live content in which a distributor appears as a virtual character (avatar) in a virtual space have become widespread. Furthermore, there has been proposed a service in which viewers themselves can enjoy a virtual space provided by this live content by appearing their own avatars.

[0003] On the other hand, when a plurality of avatars appear in a virtual space, unnaturalness may occur in actions such as contact between the avatars. Therefore, Patent Document 1 discloses a technique for controlling so as to increase the amount of information per unit time of distribution when a plurality of avatars appear in a virtual space and the distance between the avatars falls below a predetermined threshold value.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in the technique described in Patent Document 1, a deviation may occur in the positional relationship between the avatars, such as the hands, and an unnatural display may occur when the performers perform contact operations such as high-touch.

[0006] In view of the above problems, an object of the present invention is to eliminate the deviation in the positional relationship between the avatars and to display the contact operation between the avatars naturally.

Means for Solving the Problems

[0007] The information processing apparatus according to the present invention comprises: acquisition means for acquiring information on the difference in body size between a user and an avatar applied by the user; posture acquisition means for acquiring information on the user's posture; determination means for determining the posture of the avatar based on the user's posture information acquired by the posture acquisition means; and generation means for generating an image of the avatar according to the posture determined by the determination means. The determination means is characterized in that, when there is a contact action between users, it switches the method for determining the avatar's posture according to the difference in body size acquired by the acquisition means. [Effects of the Invention]

[0008] According to the present invention, discrepancies in the relative positions of avatars can be eliminated, allowing for a more natural display of contact between avatars. [Brief explanation of the drawing]

[0009] [Figure 1] This is a block diagram showing an example of the internal configuration of an avatar display system. [Figure 2] This flowchart shows an example of a processing procedure by the information processing device of the first embodiment. [Figure 3] This is a diagram illustrating the rendering of the avatar. [Figure 4] This flowchart shows an example of a processing procedure by the information processing device of the second embodiment. [Figure 5] This is a diagram illustrating the overview of contact pattern information. [Modes for carrying out the invention]

[0010] Embodiments of the present invention will be described below with reference to the drawings. However, the present invention is not limited to the embodiments described below, and various forms that do not depart from the spirit of the invention are also included. Furthermore, each embodiment described below is merely one embodiment of the present invention, and it is possible to combine each embodiment as appropriate.

[0011] (First Embodiment) Figure 1 is a block diagram showing an example of the internal configuration of the avatar display system 10 according to this embodiment. As shown in Figure 1, the avatar display system 10 consists of an information processing device 100 and a motion capture device 150. In this embodiment, the information processing device 100 is described as a personal computer (PC), but it may be a portable smartphone or an HMD (head-mounted display). The motion capture device 150 is a device worn by the user and transmits coordinate information of the user's joint points to the information processing device 100. If there are multiple users, each user is fitted with a motion capture device 150. In this case, the avatar display system 10 consists of the information processing device 100 and motion capture devices 150 equal to the number of users.

[0012] First, the internal configuration of the information processing device 100 will be described. The information processing device 100 includes a control unit 101, a non-volatile memory 102, a working memory 103, an operation unit 104, a display unit 105, a recording medium 106, and a communication unit 107.

[0013] The control unit 101 controls each part of the information processing device 100. Alternatively, instead of the control unit 101 controlling the entire device, multiple hardware components may share the processing to control the entire device.

[0014] The non-volatile memory 102 stores programs and other data executed by the control unit 101. The working memory 103 is used to temporarily store data generated by the control unit 101, or as image display memory for the display unit 105, or as a working area for the control unit 101, etc.

[0015] The operation unit 104 includes, for example, the power button of the information processing device 100, a keyboard, a mouse, etc., and receives instructions for the information processing device 100 from the user's operation. The display unit 105 displays data generated by the control unit 101 and displays a GUI (Graphical User Interface) for interactive operation. Note that the display unit 105 does not necessarily have to be built into the information processing device 100, and the information processing device 100 may be configured to have only a display control function that displays on an external display device.

[0016] The recording medium 106 stores data generated by the control unit 101, as well as user body information and avatar information, which will be described later. The recording medium 106 may be configured to be detachable from the information processing device 100, or it may be built into the information processing device 100. In either case, the information processing device 100 shall have at least means for accessing the recording medium 106.

[0017] The communication unit 107 is an interface that performs wireless LAN communication compliant with standards such as IEEE 802.11, enabling wireless communication with the access point. Furthermore, the communication unit 107 transmits and receives data to and from devices connected to the cloud network via the access point using higher-level protocols such as TCP / IP. Specifically, the communication unit 107 communicates with the motion capture device 150.

[0018] Next, the configuration of the motion capture device 150 will be described. The motion capture device 150 includes a control unit 151, a non-volatile memory 152, a working memory 153, an operation unit 154, a posture acquisition unit 155, and a communication unit 157.

[0019] The control unit 151 controls each part of the motion capture device 150. Note that instead of the control unit 151 controlling the entire device, multiple hardware components may share the processing to control the entire device. The non-volatile memory 152 stores programs and the like executed by the control unit 151. The working memory 153 is used as a buffer area for temporarily expanding the avatar's posture information and as a working area for the control unit 151. The operation unit 154 includes, for example, the power button and operation buttons of the motion capture device 150, and receives instructions for the motion capture device 150 from the user's operations.

[0020] The posture acquisition unit 155 acquires the coordinate information of the user's joint points. Specifically, the posture acquisition unit 155 uses an acceleration sensor, a gyro sensor, etc., not shown in the figure, to acquire the coordinate information of the joint points of the user wearing the motion capture device 150.

[0021] The communication unit 157 is an interface that performs wireless LAN communication compliant with, for example, the IEEE802.11 standard, and enables wireless communication with an access point. Also, the communication unit 157 transmits and receives data to and from devices connected to a cloud network via an access point using a higher-level protocol such as TCP / IP. In this embodiment, the communication unit 157 is connected to the information processing device 100 and transmits the coordinate information of the user's joint points acquired by the posture acquisition unit 155 to the information processing device 100. Note that in this embodiment, it is assumed that the coordinate information of the user's joint points is transmitted to the information processing device 100 at a constant period from the start of communication.

[0022] Next, referring to FIG. 2, the detailed operation of the information processing device 100 in this embodiment will be described. FIG. 2 is a flowchart showing an example of the processing procedure by the information processing device 100 of this embodiment. The processing shown in FIG. 2 is realized by the control unit 101 of the information processing device 100 executing a program stored in the non-volatile memory 102 and controlling each part of the information processing device 100.

[0023] First, in step S201, the control unit 101 acquires the physical information of the users participating as avatars. If there are multiple users participating as avatars, the control unit 101 acquires the physical information of all of them. Here, the user's physical information is assumed to include at least the user's actual height. The method for acquiring the user's physical information is that the user's physical information is first input from the operation unit 104 or received via the communication unit 107, and the user's physical information is recorded on the recording medium 106. The control unit 101 acquires the physical information of the users participating as avatars from the recording medium 106.

[0024] Next, in step S202, the control unit 101 acquires avatar information of users participating as avatars. If there are multiple users participating as avatars, the control unit 101 acquires avatar information for all of them. The avatar information includes information on the correspondence between the avatar and the user, and information on the avatar's physique. The avatar's physique information shall include at least the avatar's height. Similar to the user's physical information, the avatar information is either input from the operation unit 104 or received via the communication unit 107, and the avatar information is recorded on the recording medium 106. The control unit 101 acquires the avatar information of users participating as avatars from the recording medium 106.

[0025] Next, in step S203, the control unit 101 calculates the difference in body size between the user and the avatar to be used by the user, based on the user's physical information acquired in step S201 and the avatar information acquired in step S202. Specifically, the control unit 101 calculates the difference in body size as the difference between the user's actual height and the avatar's height, as the acquisition means.

[0026] Next, in step S204, the control unit 101 determines whether or not there are any avatars whose body size difference calculated in step S203 is greater than or equal to a threshold. If the control unit 101 determines that there are avatars whose body size difference is greater than or equal to a threshold, the process proceeds to step S205. On the other hand, if the control unit 101 determines that there are no avatars whose body size difference is greater than or equal to a threshold, the process proceeds to step S206.

[0027] In step S205, the control unit 101 sets a correction flag for avatars whose size difference exceeds a threshold and stores it in the working memory 103. Next, in step S206, the control unit 101, as a posture acquisition means, acquires coordinate information of the current joint points of each user from the motion capture device 150 via the communication unit 107, as information regarding the user's posture. Furthermore, the control unit 101, as a determination means, determines whether or not the users are in contact with each other based on the coordinate information of each joint point obtained from the motion capture device 150, and if they are in contact, calculates the coordinates of the contact point.

[0028] Next, in step S207, the control unit 101 identifies the avatar's posture from the avatar information acquired in step S202 and the user's joint point coordinate information acquired in step S206. At this time, the control unit 101 identifies the user for whom joint point coordinate information has been obtained based on the user correspondence information included in the avatar information, and identifies the avatar's posture for each user (avatar).

[0029] Next, in step S208, the control unit 101 determines from the coordinate information of the user's joint points obtained in step S206 whether or not there are any combinations of users that are in contact with each other. If the control unit 101 determines that there are combinations of users that are in contact with each other, the process proceeds to step S209. On the other hand, if the control unit 101 determines that there are no combinations of users that are in contact with each other, the process proceeds to step S211.

[0030] In step S209, the control unit 101 determines whether a correction flag has been set for at least one avatar corresponding to a user in a combination of users who are in contact with each other. If the control unit 101 determines that a correction flag has been set, the process proceeds to step S210. On the other hand, if the control unit 101 determines that a correction flag has not been set, the process proceeds to step S211.

[0031] In step S210, the control unit 101 determines the final avatar pose to be used for rendering as a determination means. At this time, if a user with a height difference exceeding a threshold is included in a combination of users who are in contact with each other, the avatar's physique will be deformed, or the image will appear as if the avatars are not in contact with each other despite the users being in contact. Therefore, the control unit 101 calculates the contact points between avatars, taking into account the avatars' height differences. Then, it adjusts the coordinates of each joint of the avatars involved according to the contact points and determines the final avatar pose. At this time, the avatar whose pose is changed is not necessarily the avatar with the "needs correction" flag set. For example, if an avatar with a height difference exceeding a threshold is taller than the other avatars it is in contact with, the pose of the avatar with the "needs correction" flag set is changed to a posture that makes it bend at the waist. On the other hand, if, for example, an avatar with a height difference exceeding a threshold is shorter than the other avatars it is in contact with, the pose of the avatar with no height difference is changed to a posture that makes it bend at the waist. Furthermore, for other avatars whose posture is not changed, the posture of the avatar identified in step S207 is determined as the final avatar posture to be used for rendering.

[0032] On the other hand, in step S211, the control unit 101 determines, as a determination means, that the avatar's pose identified in step S207 will be used as the avatar's pose for rendering.

[0033] In step S212, the control unit 101, acting as a generation means, renders the avatar according to the avatar's pose determined in step S210 or step S211, and stores the rendered image in the working memory 103. At this time, based on the final avatar's pose, each avatar is generated not as a wireframe consisting only of coordinate information, but as image information that has undergone texture processing, etc. Then, image data that has undergone various image processing for viewing is generated from the rendered avatar and stored in the working memory 103.

[0034] Then, in step S213, the control unit 101 determines whether or not to terminate the rendering of the avatar. If the control unit 101 determines to terminate the rendering of the avatar, the process shown in Figure 2 is terminated. On the other hand, if the control unit 101 determines not to terminate the rendering of the avatar, the process returns to step S206, and the same process is repeated at regular intervals based on the coordinate information of the user's joint points that has been newly acquired.

[0035] As described above, the image data including the avatar stored in the working memory 103 is displayed on the display unit 105 or distributed to the cloud network via the communication unit 107.

[0036] Next, with reference to Figure 3, an overview of the avatar rendered by the avatar display system in this embodiment will be described. Figure 3(a) shows user 301 and user 302 high-fiving. Here, there is no significant difference in physique between the two users 301 and 302, but the avatar used by user 302 is considerably shorter in height than the avatar used by user 301.

[0037] Figure 3(b) shows an example of an image that appears when the avatar's posture is determined and rendered using the user's joint point coordinate information. In Figure 3(b), avatar 303 is the avatar of user 301, and avatar 304 is the avatar of user 302. If the avatar's posture is determined using the user's joint point coordinate information, the avatars will appear not to be touching each other, even though the users are in a high-five motion.

[0038] Figure 3(c) shows an example image when avatars are rendered according to the procedure in Figure 2. In Figure 3(c), avatar 305 is the avatar of user 301, and avatar 306 is the avatar of user 302. In this embodiment, by calculating the contact points 307 between avatars considering the physique of the avatars and rendering avatar 305 in a crouching position, the avatars can be displayed in a way that allows them to naturally come into contact with each other.

[0039] As described above, according to this embodiment, when there is a difference in physique between the user and the avatar, and when the avatar performs contact actions with other avatars, the avatar's shape is corrected to match the avatar's physique. This makes it possible to display more natural contact actions between avatars.

[0040] In this embodiment, an example was described in which the user wears the motion capture device 150 itself. However, other methods may be used as long as the coordinate information of the user's joint points can be obtained. For example, the motion capture device 150 itself may be contactless from the user, and the coordinate information of the user's joint points may be obtained by estimating the user's joint points from video footage of the user.

[0041] Furthermore, in this embodiment, the contact points between avatars are calculated considering the physique of the avatars, and the coordinates of each joint of the avatars involved are adjusted according to the contact points. As shown in the example in Figure 3(c), when adjusting to a posture such as bending at the waist, the lower body joints do not need to be adjusted. In this case, only the coordinates of the upper body joints are adjusted, and the lower body posture is left as is, using the avatar posture identified in step S207 based on the user's joint point coordinate information. In this way, the final avatar posture may be determined by limiting the parts adjusted based on the user-to-user contact pattern (contact points) to only a portion of the parts.

[0042] (Second embodiment) In the first embodiment, an example was described in which contact points were calculated considering the avatar's physique, and the coordinates of the relevant joints of the avatar were adjusted accordingly. In contrast, this embodiment describes an example in which rendering is performed using multiple contact pattern information of avatars that have been registered in advance. Note that the explanation of parts common to the first embodiment will be omitted, and only the differences from the first embodiment will be described.

[0043] The configuration of the avatar display system in this embodiment is basically the same as in Figure 1. However, in this embodiment, the recording medium 106 also records multiple contact pattern information corresponding to multiple patterns of contact between avatars, such as "high-five" and "handshake." This contact pattern information is generated based on the user's physical information and the avatar's physique information, which are recorded in advance on the recording medium 106. Specifically, contact point information between avatars corresponding to multiple patterns, such as "high-five" and "handshake," is included in each contact pattern information.

[0044] Next, the detailed operation of the information processing device 100 in this embodiment will be described with reference to Figure 4. Figure 4 is a flowchart showing an example of a processing procedure by the information processing device 100 in this embodiment. The processing shown in Figure 4 is realized by the control unit 101 of the information processing device 100 executing a program stored in the non-volatile memory 102 and controlling each part of the information processing device 100. The same step numbers are used for processing that is the same as in Figure 2, and the explanation of these processes will be omitted. Below, only the processing that differs from that in Figure 2 will be described.

[0045] In step S401, the control unit 101 determines the final pose of the avatar to be used for rendering. At this time, the control unit 101 uses the contact point information between users obtained from the user's joint point coordinate information acquired in step S206 to identify which contact pattern information among the contact pattern information pre-recorded in the recording medium 106 corresponds to the contact action. Then, it selects the corresponding contact pattern information from among the multiple contact pattern information recorded in the recording medium 106, adjusts the coordinates of each joint of the avatar using the contact point information between avatars included in that contact pattern information, and determines the final pose of the avatar.

[0046] Next, with reference to Figure 5, an overview of the avatar rendered by the avatar display system in this embodiment will be described. Figure 5(a) shows user 501 and user 502 high-fiving. Here, there is no significant difference in physique between the two users 501 and 502, but the avatar used by user 502 is considerably shorter than the avatar used by user 501.

[0047] Figure 5(b) shows an example of a "high-five" contact pattern, and Figure 5(c) shows an example of a "handshake" contact pattern. In Figure 5(b), avatar 503 is the avatar of user 501, and avatar 504 is the avatar of user 502. Similarly, in Figure 5(c), avatar 505 is the avatar of user 501, and avatar 506 is the avatar of user 502. In this embodiment, the contact pattern information recorded on the recording medium 106 is pre-adjusted to ensure a natural representation, taking into account the differences in the avatars' body sizes. For example, when coordinate information of the user's joint points is acquired in the state shown in Figure 5(c), the control unit 101 identifies the contact pattern in Figure 5(b) from the information of the contact points between the users. Then, using the contact point information between the avatars in the contact pattern shown in Figure 5(b), the final avatar posture is determined.

[0048] As described above, according to this embodiment, by using contact point information of multiple avatars that have been registered in advance as contact pattern information, the process of calculating contact points that take into account the avatars' body sizes can be omitted. This makes it easy to adjust the coordinates of each joint of the avatars to determine the final avatar posture and display natural contact movements between avatars.

[0049] (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0050] This embodiment includes the following configurations, methods, and programs.

[0051] (Composition 1) A means for acquiring information on the difference in body size between a user and the avatar that the user applies to, A posture acquisition means for acquiring information about the user's posture, A determination means for determining the avatar's posture based on the user's posture information acquired by the posture acquisition means, A generation means for generating an image of the avatar according to the posture determined by the determination means, It has, The information processing apparatus is characterized in that, when users interact with each other, the determination means switches the method for determining the avatar's posture according to the difference in body size acquired by the acquisition means.

[0052] (Configuration 2) The information processing device according to Configuration 1, characterized in that the determination means determines the posture of the avatar by adjusting the contact point in the contact action according to the difference in body size when the difference in body size obtained by the acquisition means is greater than or equal to a threshold. (Composition 3) The information processing apparatus according to configuration 2, characterized in that the determination means determines the avatar's posture by adjusting a part of the avatar's posture based on the user's posture information acquired by the posture acquisition means when the difference in body size acquired by the acquisition means is greater than or equal to a threshold. (Composition 4) The system further includes a determination means that determines whether or not there is a contact action between users based on the user's posture information acquired by the posture acquisition means. The information processing apparatus according to any one of configurations 1 to 3, characterized in that the determination means switches the method for determining the avatar's posture according to the difference in body size when the determination means determines that there is a contact action between users.

[0053] (Composition 5) The posture acquisition means is characterized by acquiring coordinate information of the user's joint points as information relating to the user's posture, as an information processing device according to any one of configurations 1 to 4. (Composition 6) The information processing device according to any one of configurations 1 to 5, characterized in that the acquisition means inputs the user's physical information and the information of the avatar to be used by the user, and calculates the difference in physique based on the input information. (Composition 7) The system further includes a memory means for storing information on multiple contact patterns when avatars come into contact with each other. An information processing device according to any one of configurations 1 to 6, characterized in that, when there is a contact action between users, the determination means identifies a contact pattern between the users based on the user's posture information acquired by the posture acquisition means, selects the identified contact pattern information from among a plurality of contact pattern information stored in the storage means, and determines the avatar's posture using the selected contact pattern information.

[0054] (method) A process for acquiring information on the difference in body size between the user and the avatar that the user applies to, A posture acquisition step for acquiring information about the user's posture, A determination step in which the avatar's posture is determined based on the user's posture information obtained in the posture acquisition step, A generation step which generates an image of the avatar according to the posture determined in the determination step, It has, A control method for an information processing device, characterized in that, when there is contact between users, the determination step switches the method for determining the avatar's posture according to the difference in body size acquired in the acquisition step.

[0055] (program) A program for causing a computer to function as one of the means of the information processing device described in any of Configurations 1 to 7. [Explanation of symbols]

[0056] 101: Control Unit, 107: Communication Unit

Claims

1. A means for acquiring information on the difference in body size between a user and the avatar that the user applies to, A posture acquisition means for acquiring information about the user's posture, A determination means for determining the avatar's posture based on the user's posture information acquired by the posture acquisition means, A generation means for generating an image of the avatar according to the posture determined by the determination means, It has, The information processing apparatus is characterized in that, when users interact with each other, the determination means switches the method for determining the avatar's posture according to the difference in body size acquired by the acquisition means.

2. The information processing apparatus according to claim 1, characterized in that the determination means determines the posture of the avatar by adjusting the contact point in the contact action according to the difference in body size when the difference in body size obtained by the acquisition means is greater than or equal to a threshold.

3. The information processing apparatus according to claim 2, characterized in that the determination means determines the avatar's posture by adjusting a part of the avatar's posture based on the user's posture information acquired by the posture acquisition means when the difference in body size acquired by the acquisition means is greater than or equal to a threshold.

4. The system further includes a determination means that determines whether or not there is a contact action between users based on the user's posture information acquired by the posture acquisition means. The information processing apparatus according to claim 1, characterized in that the determination means switches the method for determining the avatar's posture according to the difference in body size when the determination means determines that there is a contact action between users.

5. The information processing device according to claim 1, characterized in that the posture acquisition means acquires coordinate information of the user's joint points as information relating to the user's posture.

6. The information processing device according to claim 1, characterized in that the acquisition means inputs the user's physical information and the information of the avatar to be used by the user, and calculates the difference in physique based on the input information.

7. The system further includes a memory means for storing information on multiple contact patterns when avatars come into contact with each other. The information processing device according to claim 1, characterized in that, when there is a contact action between users, the determination means identifies a contact pattern between the users based on the user's posture information acquired by the posture acquisition means, selects the identified contact pattern information from among a plurality of contact pattern information stored in the storage means, and determines the avatar's posture using the selected contact pattern information.

8. A process for acquiring information on the difference in body size between the user and the avatar that the user applies to, A posture acquisition step for acquiring information about the user's posture, A determination step in which the avatar's posture is determined based on the user's posture information obtained in the posture acquisition step, A generation step which generates an image of the avatar according to the posture determined in the determination step, It has, A control method for an information processing device, characterized in that, when there is contact between users, the determination step switches the method for determining the avatar's posture according to the difference in body size acquired in the acquisition step.

9. A program for causing a computer to function as each means of the information processing apparatus described in claim 1.

Images