Distribution device, distribution method, and program

Abstract

The present technology relates to a distribution device, a distribution method, and a program with which a communication amount can be reduced while maintaining the natural movement of an avatar. This distribution device comprises: a receiving unit that receives first movement information indicating the movement of a user; a distribution unit that distributes, to a terminal which displays an avatar, second movement information indicating the movement of the avatar corresponding to the user; and an adjustment unit that adjusts, according to the first movement information, an information amount of the second movement information to be distributed by the distribution unit. This technology can be applied to, for example, an information processing system that provides an application which causes the movement of a user using a terminal connected via a network to be reflected on an avatar in a virtual space.

Description

[[Technical Field]]

[0001] The present technology relates to a distribution device, a distribution method, and a program, and particularly relates to a distribution device, a distribution method, and a program that can reduce communication traffic while maintaining natural movement of an avatar. [[Background Art]]

[0002] There are applications that use a plurality of terminals connected via a server on a network to reflect the movements of a plurality of users on respective avatars arranged in a virtual space.

[0003] In order to move an avatar naturally, a plurality of terminals need to mutually transmit movement information indicating the movement of the avatar. As the number of users increases, the amount of movement information transmitted and received by the server increases explosively.

[0004] Patent Literature 1 describes a technology for reducing the load on a system by varying the amount of information per unit time of distribution performed by a distribution unit according to the distance between avatars in a virtual space.

[0005] Further, for example when the movement of an avatar is indicated by joint positions of the avatar or the like, the amount of movement information can be reduced by decreasing the number of joints for which movement information is distributed. In this case, by using technologies called FK (Forward Kinematics) and IK (Inverse Kinematics), movement information for other joints is estimated based on the distributed movement information of the joints. [[Prior Art Literature]] [[Patent Literature]]

[0006] [[Patent Literature 1]] Japanese Unexamined Patent Publication No. 2020-91504 [[Summary of the Invention]] [[Problem to be Solved by the Invention]]

[0007] In the technology described in Patent Document 1, the amount of information regarding the movement of avatars located far away in the virtual space is reduced, which can result in unnatural behavior of those avatars. As the avatar's entire body is displayed the further away it is located, users of the terminal may see avatars behaving unnaturally.

[0008] Furthermore, in situations such as a concert venue where other avatars are positioned behind one, the method of varying the amount of motion information according to distance may not be effective.

[0009] In technologies such as FK and IK, the estimation of joint movement information that is not transmitted can fail, resulting in unnatural avatar behavior. Furthermore, if the frequency of motion information transmission is further reduced to decrease the amount of information transmitted to each device, these technologies may become more prone to failing to estimate joint movement information that is not transmitted.

[0010] Therefore, it was not easy to reduce the amount of motion information transmitted and received via the server while making the avatar move naturally.

[0011] This technology was developed in light of these circumstances, and aims to reduce data usage while maintaining the natural movement of the avatar. [Means for solving the problem]

[0012] One aspect of this technology is a distribution device comprising: a receiving unit that receives first motion information indicating the movements of a user; a distribution unit that distributes second motion information indicating the movements of an avatar corresponding to the user to a terminal that displays the avatar; and an adjustment unit that adjusts the amount of information of the second motion information distributed by the distribution unit according to the first motion information.

[0013] One aspect of this technology is a distribution method in which a distribution device receives first motion information indicating the user's movements, adjusts the amount of information in second motion information indicating the avatar's movements, which is distributed to a terminal that displays an avatar corresponding to the user, and distributes the second motion information to the terminal.

[0014] One aspect of this technology involves a program that causes a computer to receive first motion information indicating the user's movements, adjust the amount of second motion information indicating the avatar's movements which is distributed to a terminal that displays an avatar corresponding to the user, and then distribute the second motion information to the terminal.

[0015] In one aspect of this technology, first motion information indicating the user's movements is received and distributed to a terminal that displays an avatar corresponding to the user. The amount of information in second motion information indicating the avatar's movements is adjusted and the second motion information is distributed to the terminal. [Brief explanation of the drawing]

[0016] [Figure 1] This figure shows an example configuration of an information processing system according to one embodiment of this technology. [Figure 2] This is a block diagram showing an example configuration of a head-mounted display terminal. [Figure 3] This is a block diagram showing an example server configuration. [Figure 4] This figure shows an example of the range of motion of the joints in an avatar. [Figure 5] This is a flowchart illustrating the application execution process for a head-mounted display terminal. [Figure 6] This is a flowchart explaining the server's distribution process. [Figure 7] This is a block diagram showing an example configuration of a head-mounted display terminal related to the first modified example. [Figure 8] This is a block diagram showing an example of the server configuration related to the first modified example. [Figure 9] This diagram shows the movements of the user and their avatar. [Figure 10] It is a block diagram showing a configuration example of a server according to a second modification. [Figure 11] It is a diagram showing an example of the importance of each part of a body. [Figure 12] It is a block diagram showing a configuration example of a server according to a third modification. [Figure 13] It is a block diagram showing a configuration example of a head-mounted display terminal according to a third modification. [Figure 14] It is a diagram showing an example of a method for obtaining support information indicating the presence or absence of an obstruction between a viewer's viewpoint and each joint of an avatar. [Figure 15] It is a diagram showing an example of a method for obtaining support information indicating the amount of light reaching an avatar. [Figure 16] It is a diagram showing an example of a method for obtaining support information indicating the display area of an avatar. [Figure 17] It is a block diagram showing a configuration example of a head-mounted display terminal according to a fourth modification. [Figure 18] It is a block diagram showing a configuration example of a server according to a fourth modification. [Figure 19] It is a diagram showing an example of a degree-of-attention estimation method. [Figure 20] It is a diagram showing another example of a degree-of-attention estimation method. [Figure 21] It is a diagram showing still another example of a degree-of-attention estimation method. [Figure 22] It is a block diagram showing a configuration example of computer hardware. DETAILED DESCRIPTION OF EMBODIMENTS

[0017] Hereinafter, embodiments for carrying out the present technology will be described. The description will be given in the following order. 1. Configuration of Information Processing System 2. Operation of Each Device 3. Modifications

[0018] <1. Configuration of Information Processing System> Figure 1 shows an example of the configuration of an information processing system according to one embodiment of this technology.

[0019] The information processing system shown in Figure 1 is a system that realizes an application that reflects the movements of multiple users onto each avatar placed in a virtual space.

[0020] As shown in Figure 1, the information processing system is configured by connecting n head-mounted display terminals 1-1 to 1-n to a server 2 via a network.

[0021] Head-mounted display terminals 1-1 to 1-n acquire motion information indicating the movements of the user wearing the head-mounted display terminals 1-1 to 1-n and transmit it to server 2. Head-mounted display terminals 1-1 to 1-n also receive motion information indicating the movements of avatars corresponding to other users, which has been distributed from server 2, and apply the avatar motion information to the avatar's 3D model.

[0022] As viewers, users can move their viewpoint within the virtual space and view images of a virtual space containing avatars that represent the movements of other users, using head-mounted display terminals 1-1 to 1-n.

[0023] Server 2 receives and stores user movement information transmitted from head-mounted display terminals 1-1 to 1-n. Server 2 functions as a distribution device that adjusts the amount of avatar movement information based on the corresponding user movement information and then distributes it to head-mounted display terminals 1-1 to 1-n.

[0024] In the following, when it is not necessary to distinguish between head-mounted display terminals 1-1 through 1-n individually, they will simply be referred to as head-mounted display terminal 1.

[0025] Figure 2 is a block diagram showing an example configuration of the head-mounted display terminal 1.

[0026] As shown in Figure 2, the head-mounted display terminal 1 consists of an input processing unit 11, a transmission data processing unit 12, a receiving unit 13, a program processing unit 14, a design data storage unit 15, and a display unit 16.

[0027] The input processing unit 11 processes data received from controllers and sensors. The input processing unit 11 includes a motion information processing unit 21 and an operation unit 22.

[0028] The motion information processing unit 21 acquires bone position information, which indicates the user's posture in terms of joint positions and joint angles, as user motion information. For example, the motion information processing unit 21 estimates bone position information for the user's upper body based on the positions of the controllers held by the user in both hands and the head-mounted display terminal 1 worn on the user's head. The positions of the controllers and the head-mounted display terminal 1 are detected, for example, by sensors provided on each device.

[0029] Furthermore, the motion information processing unit 21 can also acquire bone position information based on captured images obtained by, for example, photographing the user's entire body with a camera.

[0030] The control unit 22 accepts input from operations such as the directional keys and control buttons provided on the controller. The direction and amount of movement of the avatar corresponding to the user in the virtual space are input using the directional keys and control buttons.

[0031] The transmission data processing unit 12 includes a bone position holding unit 31 and a user position acquisition unit 32.

[0032] The bone position holding unit 31 holds bone position information as user movement information acquired by the motion information processing unit 21.

[0033] The user position acquisition unit 32 acquires position information indicating the position of the avatar corresponding to the user in the virtual space, based on the operation input received by the operation unit 22. Alternatively, the user position acquisition unit 32 may acquire position information based on the amount of user movement calculated from the movement information acquired by the movement information processing unit 21. This position in the virtual space also becomes the viewer's point of view.

[0034] The transmission data processing unit 12 compresses and encrypts the user's movement information and location information, converts it into a data format for transmission over the network, and generates transmission data. The transmission data processing unit 12 then sends the transmission data to the server 2 over the network.

[0035] The receiving unit 13 receives the distribution data delivered from the server 2. The receiving unit 13 decodes and decompresses the distribution data and obtains movement information and location information of avatars corresponding to other users.

[0036] The program processing unit 14 includes a rendering unit 41. The rendering unit 41 applies the avatar movement information acquired by the receiving unit 13 to each 3D model. The rendering unit 41 performs rendering and generates a rendered image that shows the virtual space as seen from the user's perspective.

[0037] In this virtual space, each avatar is placed based on the positional information of the avatars corresponding to other users. The design data of the objects in the virtual space and the avatars corresponding to other users, as well as the 3D models of the avatars, are read from the design data storage unit 15 and used when rendering.

[0038] The design data storage unit 15 stores 3D models of avatars and objects, as well as their design data.

[0039] The display unit 16 displays the rendered image generated by the rendering unit 41.

[0040] Figure 3 is a block diagram showing an example configuration of Server 2.

[0041] As shown in Figure 3, the server 2 consists of a receiving unit 51, a user management unit 52, a program processing unit 53, a bone position holding unit 54, and a distribution unit 55.

[0042] The receiving unit 51 receives the transmitted data sent from the head-mounted display terminal 1. The receiving unit 51 decodes and decompresses the transmitted data to obtain the user's movement information and location information.

[0043] The user management unit 52 holds user login information and manages the scene (world) that each user is experiencing by viewing the rendered video. For example, in an information processing system, movement information of the avatar corresponding to each user is mutually distributed to head-mounted display terminals 1 used by users experiencing the same scene.

[0044] The program processing unit 53 includes an adjustment unit 61. The adjustment unit 61 sets a distribution priority (value criterion) indicating the priority for distributing avatar movement information corresponding to the user, based at least on the user's movement information, and adjusts the amount of avatar movement information distributed to the head-mounted display terminal 1 by the distribution unit 55 according to the set distribution priority. The avatar movement information is bone position information that indicates the avatar's posture by the position and angle of the avatar's joints corresponding to the user's joints.

[0045] For example, the adjustment unit 61 adjusts the frequency of distribution of avatar movement information for each joint of the avatar according to the distribution priority set based on the amount of joint movement of the user indicated by the user's movement information. The amount of joint movement includes the amount of change in joint position and the amount of change in joint angle. The amount of joint movement of the user is calculated, for example, based on the history of user movement information held in the bone position holding unit 54.

[0046] Figure 4 shows an example of the range of motion of the joints in Avata.

[0047] As shown on the left side of Figure 4, an avatar with its right arm raised will receive movement information that causes it to move its right arm closer to its head, as shown on the right side of Figure 4.

[0048] In this case, for example, the change in the position of the neck joint is small, and the change in the joint angle is also small. Therefore, the adjustment unit 61 sets the distribution priority of the neck joint to a low value and sets the frame rate for distributing the bone position information of the neck joint to a value lower than the reference value.

[0049] The change in the position of the right shoulder joint is small, while the change in the joint angle is large. Therefore, the adjustment unit 61 sets a high priority for the distribution of the right shoulder joint and sets the frame rate for distributing the bone position information of the right shoulder joint to a value higher than the reference value.

[0050] The change in the joint position of the right elbow is moderate, and the change in the joint angle is small. Therefore, the adjustment unit 61 sets the distribution priority of the right elbow joint to moderate and sets the frame rate for distributing bone position information of the right elbow joint to a reference value.

[0051] The change in the joint position of the right hand is large, and the change in joint angle is also large. Therefore, the adjustment unit 61 sets a high priority for the distribution of the right hand joints and sets the frame rate for distributing the bone position information of the right hand joints to a value higher than the reference value.

[0052] Thus, for example, if at least one of the changes in joint position or joint angle is large, the delivery priority for that joint is set to high; if at least one of the changes in joint position or joint angle is moderate, the delivery priority for that joint is set to moderate; and if both the changes in joint position and joint angle are small, the delivery priority for that joint is set to low.

[0053] In the above explanation, the amount of change in joint position and joint angle was divided into three levels: large, medium, and small. However, it is also possible to set the frame rate for distributing bone position information in proportion to the amount of change in joint position and joint angle.

[0054] Returning to Figure 3, the bone position holding unit 54 stores the history of bone position information as movement information for each user acquired by the receiving unit 51.

[0055] The distribution unit 55 compresses, encrypts, and converts the movement and location information of the avatar corresponding to each user into a data format for transmission over the network, thereby generating distribution data. The distribution unit 55 then distributes the distribution data to the head-mounted display terminal 1 via the network.

[0056] <2. Operation of each device> The application execution process of the head-mounted display terminal 1 will be explained with reference to the flowchart in Figure 5.

[0057] In step S1, the motion information processing unit 21 acquires bone position information as user motion information.

[0058] In step S2, the user position acquisition unit 32 acquires the position information of the avatar corresponding to the user based on the operation input received by the operation unit 22.

[0059] In step S3, the transmission data processing unit 12 generates transmission data by compressing, encrypting, and converting the user's movement information and location information, and then sends it to the server 2 via the network.

[0060] In step S4, the receiving unit 13 acquires avatar movement information corresponding to other users. The avatar movement information is obtained by decrypting and decompressing the distribution data delivered from the server 2.

[0061] In step S5, the rendering unit 41 applies the movement information of avatars corresponding to other users to the 3D model.

[0062] In step S6, the rendering unit 41 performs rendering and generates a rendered image that shows the virtual space as seen from the user's perspective.

[0063] In step S7, the display unit 16 displays the rendered image.

[0064] Next, we will explain the distribution process of Server 2, referring to the flowchart in Figure 6.

[0065] In step S11, the receiving unit 51 receives the transmitted data from the head-mounted display terminal 1 and acquires user movement information by decoding and decompressing the transmitted data.

[0066] In step S12, the bone position holding unit 54 holds a history of bone position information as user movement information.

[0067] In step S13, the adjustment unit 61 adjusts the amount of information in the avatar movement information distributed by the distribution unit 55.

[0068] In step S14, the distribution unit 55 generates distribution data by compressing, encrypting, and converting the avatar's movement information and location information, and distributes it to the head-mounted display terminal 1 via the network.

[0069] In the distribution process of server 2 described above, the amount of avatar movement information is adjusted according to the distribution priority set based on user movement information, and then distributed to head-mounted display terminal 1.

[0070] For example, by setting a distribution priority based on the amount of joint movement, only the bone position information of joints with large amounts of movement will be distributed at a high frame rate. This reduces the amount of avatar movement information distributed to the head-mounted display terminal 1, thereby reducing the load on the information processing system.

[0071] Generally, parts of the body that move a large amount tend to exhibit unnatural avatar behavior. Even if the overall amount of motion information is reduced, the bone position information for parts of the body that tend to exhibit unnatural avatar behavior is delivered at a high frame rate, allowing the head-mounted display terminal 1 to naturally represent the user's movements using the avatar.

[0072] <3. Variant> • First variation (an example of distributing specific information that allows identification of the type of avatar movement) Figure 7 is a block diagram showing an example configuration of the head-mounted display terminal 1 according to the first modified example. In Figure 7, components identical to those in Figure 2 are denoted by the same reference numerals. Repetitive explanations are omitted as appropriate.

[0073] The configuration of the head-mounted display terminal 1 shown in Figure 7 differs from the configuration of the head-mounted display terminal 1 shown in Figure 2 in that an animation storage unit 101 is provided.

[0074] The receiving unit 13 decodes and decompresses the data distributed from the server 2, and obtains avatar movement information and location information corresponding to other users. Here, the avatar movement information is specific information that can identify the type of avatar movement.

[0075] The rendering unit 41 retrieves animation data associated with specific information acquired by the receiving unit 13 from the animation storage unit 101 and applies it to the avatar's 3D model.

[0076] The animation memory unit 101 stores specific information and animation data that has been created in advance to represent the movements indicated by the specific information using an avatar, with these data being associated with each other.

[0077] Figure 8 is a block diagram showing an example configuration of Server 2 according to the first modified example. In Figure 8, components identical to those in Figure 3 are denoted by the same reference numerals. Repetitive explanations are omitted as appropriate.

[0078] The configuration of Server 2 shown in Figure 8 differs from the configuration of Server 2 in Figure 3 in that it includes a motion matching database 111.

[0079] The motion matching database 111 is pre-created using machine learning, and for example, predefined types of motion such as jumping, walking, running, and waving arms are associated with the feature quantities of motion information that represent those motions and registered accordingly.

[0080] The adjustment unit 61 compares the user's movement information acquired by the receiving unit 51 with the feature quantities of the movement information registered in the movement matching database 111, and estimates the type of movement indicated by the user's movement information. The adjustment unit 61 sets the specific information assigned to the type of movement corresponding to the feature quantities that matched the user's movement information as the avatar's movement information distributed by the distribution unit 55.

[0081] Alternatively, a motion matching database 111 may be provided on the head-mounted display terminal 1, and specific information as user motion information may be transmitted from the head-mounted display terminal 1 to the server 2.

[0082] Figure 9 shows the movements of the user and the avatar.

[0083] As shown on the left side of Figure 9, assume that the user is jumping with both arms raised. In this case, specific information assigned to the jumping action is delivered from server 2 to head-mounted display terminal 1.

[0084] In the head-mounted display terminal 1, which receives the distributed data containing specific information, animation data to make the avatar perform a jumping action is applied to the 3D model. As a result, as shown on the right side of Figure 9, the avatar performs a jumping action in the virtual space, for example, with both arms lowered.

[0085] Because avatars perform actions based on pre-created action data, they cannot accurately represent the user's movements, but viewers can get a general idea of ​​what actions other users have taken.

[0086] As can be seen from the fact that the amount of information in avatar movement information is determined by the product of the number of joints, the type of movement information (joint position and joint angle), and the distribution frequency, if the bone position information of each joint is distributed as avatar movement information, the amount of information in avatar movement information becomes bloated. By distributing only specific information such as an ID that can identify the type of avatar movement as avatar movement information, server 2 can significantly reduce the amount of information in avatar movement information distributed to head-mounted display terminal 1, thereby reducing the load on the information processing system.

[0087] If the type of movement corresponding to the user's movement information is not registered in the movement matching database 111, the avatar's movement information, adjusted according to the distribution priority set based on the user's movement information, will be distributed by the distribution unit 55.

[0088] • A second variation (an example of adjusting the amount of motion information depending on the type of user movement) Figure 10 is a block diagram showing an example configuration of server 2 according to the second modified example. In Figure 10, components identical to those in Figure 8 are denoted by the same reference numerals. Repetitive explanations are omitted as appropriate.

[0089] The configuration of Server 2 shown in Figure 10 differs from the configuration of Server 2 shown in Figure 8 in that it includes a site importance database 151.

[0090] The body part importance database 151 registers the importance of each body part in the user's movements.

[0091] The adjustment unit 61 compares the user's movement information acquired by the receiving unit 51 with the feature quantities of the movement information registered in the movement matching database 111, and estimates the type of movement indicated by the user's movement information.

[0092] The adjustment unit 61 obtains the importance of each body part in the type of movement corresponding to the feature quantity that matches the user's movement information from the body part importance database 151. The adjustment unit 61 sets a distribution priority according to the importance of each body part and adjusts the distribution frequency of the avatar's movement information for each joint of the avatar according to the set distribution priority.

[0093] Figure 11 shows examples of the importance of different parts of the body.

[0094] As shown in Figure 11, assume the user is jumping. For example, the lower body enclosed by ellipse #1 is defined as an important part for recreating the jumping action, while the upper body enclosed by ellipse #2 is defined as an important part for recreating the jumping action.

[0095] In this case, the adjustment unit 61 sets a high distribution priority for the joints included in the lower body of the avatar, and the distribution unit 55 distributes the bone position information of those joints at a high frame rate. Conversely, the adjustment unit 61 sets a low distribution priority for the joints included in the upper body of the avatar, and the distribution unit 55 distributes the bone position information of those joints at a low frame rate.

[0096] As described above, by setting a distribution priority for each joint of the avatar based on the type of user movement, it becomes possible to reduce the amount of motion information distributed to the head-mounted display terminal 1 and reduce the load on the information processing system.

[0097] Since movement information for joints, which are important parts for reproducing the user's movements with an avatar, is delivered at a high frame rate, the head-mounted display terminal 1 can naturally represent the user's movements using the avatar.

[0098] • Third variation (an example of adjusting the amount of motion information according to the display mode of the avatar) Figure 12 is a block diagram showing an example configuration of Server 2 according to the third modified example. In Figure 12, components identical to those in Figure 3 are denoted by the same reference numerals. Repetitive explanations are omitted as appropriate.

[0099] The configuration of Server 2 shown in Figure 12 differs from that of Server 2 in Figure 3 in that it includes a rendering unit 201, a design data storage unit 202, and a support information holding unit 203.

[0100] The rendering unit 201 applies the user movement information acquired by the receiving unit 13 to the 3D model of the avatar read from the design data storage unit 202. The rendering unit 201 performs rendering to obtain support information that assists (reduces computational load) the rendering calculation on the head-mounted display terminal 1. The support information is obtained as information indicating the display manner of the avatar placed in the virtual space based on the avatar's position information, specifically, information indicating the visibility of the avatar in the virtual space.

[0101] For example, the rendering unit 201 obtains one of the following as support information: information indicating the presence or absence of an obstruction between the viewer's viewpoint and each joint of the avatar; information indicating the amount of light reaching the avatar; and information indicating the display area of ​​the avatar displayed on the head-mounted display terminal 1. The support information obtained by the rendering unit 201 is stored in the support information holding unit 203.

[0102] The adjustment unit 61 reads support information from the support information holding unit 203 and sets the distribution priority based on the support information. In other words, the adjustment unit 61 sets the distribution priority based on the display manner of each avatar from the viewpoint of the viewer using the head-mounted display terminal 1. Then, the adjustment unit 61 adjusts the amount of information in the avatar's movement information according to the set distribution priority. For example, based on the support information, the adjustment unit 61 sets only the bone position information of joints that are easily visible to the viewer as the avatar's movement information distributed by the distribution unit 55. Also, for example, the adjustment unit 61 sets a lower frame rate for distributing bone position information of joints that are easily visible to the viewer.

[0103] Figure 13 is a block diagram showing an example configuration of the head-mounted display terminal 1 according to the third modified example. In Figure 13, components identical to those in Figure 2 are denoted by the same reference numerals. Repetitive explanations are omitted as appropriate.

[0104] The configuration of the head-mounted display terminal 1 shown in Figure 13 differs from the configuration of the head-mounted display terminal 1 shown in Figure 2 in that a complementary unit 211 is provided in the program processing unit 14.

[0105] If only bone position information for joints that are easily visible to the viewer is delivered from server 2 as avatar movement information, the interpolation unit 211 interpolates the bone position information of other joints based on the bone position information of those joints.

[0106] Figure 14 shows an example of a method for obtaining supporting information indicating the presence or absence of obstructions between the viewer's viewpoint and each joint of the avatar.

[0107] Supporting information indicating the presence or absence of obstructions between the viewer's viewpoint and each joint of the avatar can be obtained, for example, as a result of a Z-test for the position of each joint of the avatar. The Z-test determines the front-to-back relationship of the 3D models of the avatar or objects.

[0108] First, the rendering unit 201 places all 3D models of avatars and objects to which the user's movement information has been applied into the virtual space. Next, as shown in Figure 14, the rendering unit 201 determines whether or not a 3D model of an avatar other than avatar A1 is placed between the viewer's viewpoint C1, which serves as the rendering reference, and the joint position indicated by a circle on avatar A1 corresponding to another user.

[0109] In the example shown in Figure 14, an obstruction, Sh1, is placed between the joints of avatar A1 (excluding the joints of the right hand and left foot) and the viewpoint C1. In this case, the adjustment unit 61 sets a higher distribution priority for the right hand and left foot, where there is no obstruction Sh1 between them and the viewpoint C1, and the distribution unit 55 distributes only the bone position information for the right hand and left foot.

[0110] The rendering unit 201 will perform Z-tests for the viewpoint of one viewer, targeting the joints of all other users' avatars, and seek supporting information.

[0111] If there is an obstruction between the viewer's viewpoint and the avatar, the avatar's movement information will not be used in the rendering performed by the viewer's head-mounted display terminal 1, thus wasting the distributed avatar movement information. By setting a higher distribution priority for joints that are not obstructed between the viewer's viewpoint and the avatar, and distributing only the bone position information of those joints, it is possible to reduce the amount of avatar movement information distributed to the head-mounted display terminal 1.

[0112] Figure 15 shows an example of a method for obtaining support information indicating the amount of light reaching the avatar.

[0113] Supporting information indicating the amount of light reaching the avatar can be obtained, for example, as a result of ray tracing, which determines whether light rays from a light source pass through a rectangular prism representing the region where the avatar exists.

[0114] First, the rendering unit 201 places all 3D models of avatars and objects to which the user's movement information has been applied in the virtual space and performs lighting processing. Next, as shown in Figure 15, the rendering unit 201 projects light from lights L1 and L2 provided in the virtual space toward the rectangular parallelepiped B1 that surrounds the entire avatar A1, and determines whether or not the light reaches the rectangular parallelepiped B1.

[0115] In the example shown in Figure 15, light from light source L1 is blocked by the shield Sh1 and does not reach the rectangular prism B1, while light from light source L2 reaches the rectangular prism B1.

[0116] The rendering unit 201 calculates the amount of light reaching the cuboid B1 as the sum of the light amounts reaching the avatar. If the amount of light reaching the avatar is small, the adjustment unit 61 sets a lower distribution priority for that avatar, reduces the number of joints to which bone position information is distributed, or sets a lower bitrate for distributing the movement information of that avatar.

[0117] Viewers have difficulty seeing avatars that are not illuminated by light. Reducing the movement information of avatars that are difficult to see will result in less unnaturalness for viewers compared to reducing the movement information of avatars that are easy to see.

[0118] Therefore, by setting a lower distribution priority for avatars that receive less light, and reducing the amount of information about the movement of those avatars, it is possible to reduce the amount of information about the movement of the avatars distributed to the head-mounted display terminal 1 while preventing viewers from perceiving unnaturalness.

[0119] In Figure 15, a single rectangular prism B1 surrounding avatar A1 is shown to represent the region where avatar A1 exists. However, it is also possible to have multiple rectangular prisms surrounding different parts of avatar A1 represent the regions where avatar A1 exists.

[0120] Figure 16 shows an example of a method for obtaining support information indicating the display area of ​​an avatar.

[0121] Support information indicating the display area of ​​the avatar shown on the head-mounted display terminal 1 is obtained, for example, as a result of a process that calculates the area when a rectangular prism representing the region where the avatar exists is projected onto the image space.

[0122] First, the rendering unit 201 places a rectangular prism in the virtual space that encloses each of the 3D models of all the avatars to which the user's movement information has been applied. On the left side of Figure 16, the rectangular prism B1 enclosing avatar A1 is placed in the virtual space.

[0123] The rendering unit 201 converts the virtual space in which the rectangular prism B1 is placed into a normalized projection space, as shown at the tip of arrow #11. Then, the rendering unit 201 converts the normalized projection space into an image space as seen from the viewer's viewpoint C1, as shown at the tip of arrow #12. The rendering unit 201 uses the area of ​​the rectangular prism B1 projected into this image space as supporting information.

[0124] The adjustment unit 61, when the display area of ​​an avatar is small, sets a lower distribution priority for that avatar, reduces the number of joints to which bone position information is distributed, and sets a lower bitrate for distributing the movement information of that avatar.

[0125] Viewers find it harder to see avatars when the display area is small. Reducing the movement information of avatars that are difficult to see will result in less unnaturalness for viewers compared to reducing the movement information of avatars that are easy to see.

[0126] The display area of ​​an avatar is determined by its size and distance. Therefore, if an avatar is large, it may appear large in the rendered image even if it is far away. For example, an avatar measuring 50m and placed 50m away from the viewer's viewpoint will appear the same size in the rendered image as an avatar measuring 2m and placed 2m away from the viewer's viewpoint.

[0127] Patent Document 1 describes varying the amount of motion information depending on the distance between the viewer's viewpoint and the avatar. However, in rendered images, the amount of motion information for the aforementioned 2m and 50m avatars, which are displayed at the same size, should be the same.

[0128] The server 2 of this technology reduces the amount of motion information distributed to the head-mounted display terminal 1 while preventing viewers from perceiving unnaturalness, by reducing the amount of motion information information distributed according to the distribution priority set based on the display area.

[0129] In Figure 16, a single rectangular prism B1 surrounding avatar A1 was explained as representing the region where avatar A1 exists. However, it is also possible for multiple rectangular prisms surrounding different parts of avatar A1 to represent the regions where avatar A1 exists.

[0130] • Fourth variation (an example of adjusting the amount of motion information according to the relationship between the viewer and the user) Figure 17 is a block diagram showing an example configuration of the head-mounted display terminal 1 according to the fourth modified example. In Figure 17, components identical to those in Figure 13 are denoted by the same reference numerals. Repetitive explanations are omitted as appropriate.

[0131] The configuration of the head-mounted display terminal 1 shown in Figure 17 differs from the configuration of the head-mounted display terminal 1 shown in Figure 13 in that the transmission data processing unit 12 is equipped with a chat information acquisition unit 251.

[0132] The chat information acquisition unit 251 acquires chat information indicating the user's statements. When voice chat takes place between the user and another user, audio data is acquired as chat information from the microphone provided on the head-mounted display terminal 1. When text chat takes place between the user and another user, text data based on the operation input received by the operation unit 22 is acquired as chat information.

[0133] The transmission data processing unit 12 generates transmission data by compressing, encrypting, and converting user movement information, location information, and chat information into a data format for transmission over the network.

[0134] The receiving unit 13 decodes and decompresses the distributed data and obtains movement information, location information, and chat information of avatars corresponding to other users.

[0135] The program processing unit 14 presents the chat information acquired by the receiving unit 13 to the user. When voice chat takes place between the user and another user, audio based on the voice data is output from the speaker provided in the head-mounted display terminal 1. When text chat takes place between the user and another user, text based on the text data is displayed in the display unit 16.

[0136] Figure 18 is a block diagram showing an example configuration of Server 2 according to the fourth modified example. In Figure 18, components identical to those in Figure 3 are denoted by the same reference numerals. Repetitive explanations are omitted as appropriate.

[0137] The configuration of Server 2 shown in Figure 18 differs from the configuration of Server 2 in Figure 3 in that it includes a user relationship maintenance unit 261.

[0138] The receiving unit 51 decodes and decompresses the transmitted data to obtain user movement information, location information, and chat information.

[0139] In multiplayer applications, multiple users can form groups and act together. Additionally, users who communicate with each other or form groups are added to a friends list.

[0140] The user relationship maintenance unit 261 stores the degree of relevance between users, which is determined by the server 2 based on the groups to which each user belongs and the friend list that shows the connections between users.

[0141] Furthermore, in the virtual space, the number of times users have come into contact with or been in close proximity to each other, or the time elapsed since users came into contact with each other, is determined by Server 2 and stored by the User Relationship Maintenance Unit 261. The number of times users have come into contact with or been in close proximity to each other, and the time elapsed since users came into contact with each other, are indicators of the relationship between users.

[0142] By storing this information in the user relationship retention unit 261 for a predetermined period, it becomes possible to eliminate the need to calculate the degree of relevance between users for each frame.

[0143] The adjustment unit 61 reads from the user relationship maintenance unit 261 the degree of relevance between users, the number of times users have come into contact or been in close proximity to each other, and the time elapsed since the users came into contact with each other, and sets the distribution priority based on this information. The adjustment unit 61 adjusts the amount of avatar movement information according to the set distribution priority.

[0144] Specifically, since avatars corresponding to users with a strong relationship to the viewer are likely to be located within the viewer's field of view, the adjustment unit 61 increases the frame rate at which it distributes movement information for avatars corresponding to users with a strong relationship to the viewer. On the other hand, the adjustment unit 61 reduces the amount of movement information for avatars corresponding to users with a weak relationship to the viewer.

[0145] For example, avatars corresponding to users belonging to the same group as the viewer are likely to be within the viewer's field of view, and the viewer and those users are likely to communicate frequently. Therefore, the adjustment unit 61 sets a higher distribution priority for avatars corresponding to users belonging to the same group as the viewer.

[0146] The adjustment unit 61 can also adjust the amount of information in the avatar's movement information according to the distribution priority set based on the level of attention each user receives from multiple viewers. For example, the level of attention a user receives is estimated based on the number of viewers who have the avatar corresponding to that user within their field of view.

[0147] Figure 19 shows an example of a method for estimating attention levels.

[0148] As shown in Figure 19, it is assumed that there are avatars A11 to A15, corresponding to five users, in the virtual space. In Figure 19, the field of view of each avatar A11 to A15 is indicated by a dashed triangle.

[0149] In the example shown in Figure 19, since avatars A11 to A13 are not within the field of view of any of the users, it is estimated that avatars A11 to A13 have low attention levels. Therefore, the adjustment unit 61 sets a lower distribution priority for avatars A11 to A13, which have low attention levels.

[0150] Furthermore, since avatar A14 is within the field of view of the user corresponding to avatar A13, it is estimated that avatar A14 has a moderate level of attention. Therefore, the adjustment unit 61 sets the distribution priority of avatar A14, which has a moderate level of attention, to a moderate level.

[0151] Since Avatar A15 is within the field of view of the users corresponding to Avatars A11, A12, and A14, it is estimated that Avatar A15 will attract a lot of attention. Therefore, the adjustment unit 61 sets a higher distribution priority for Avatar A15, which has a high level of attention.

[0152] Figure 20 shows another example of a method for estimating attention.

[0153] User engagement may be estimated based on whether or not the user has made a statement using chat.

[0154] As shown in Figure 20, it is assumed that there are avatars A21 to A25 corresponding to five users in the virtual space.

[0155] In the example in Figure 20, avatars A21 to A23 have not made any statements using chat, so they are unlikely to be seen by other users, and their level of attention is estimated to be low. Therefore, the adjustment unit 61 sets a lower distribution priority for avatars A21 to A23, which have low levels of attention.

[0156] Furthermore, since avatars A24 and A25 are making statements using chat, there is a high possibility that they will be seen by other users, and it is estimated that avatars A24 and A25 are attracting a lot of attention. Therefore, the adjustment unit 61 sets a higher distribution priority for avatars A24 and A25, which are attracting a lot of attention.

[0157] Figure 21 shows yet another example of a method for estimating attention.

[0158] The user's level of attention may be estimated based on the paths or directions in which multiple avatars move within the virtual space.

[0159] When multiple users act with a common objective, such as going to a specific location, searching for a specific object, or chasing a specific object, it is reasonable to assume that some degree of collective behavior will occur, regardless of the group to which each user belongs.

[0160] As shown in Figure 21, six avatars, A31 to A36, corresponding to six users, are assumed to be moving within the virtual space. The arrows pointing to avatars A31 to A36 indicate the movement path of each avatar.

[0161] Server 2 stores the movement paths of avatars A31 to A36 over a certain period of time and calculates the average movement path and direction of the group consisting of avatars A31 to A36. In the example in Figure 21, arrow #21 indicates the direction of movement of the group.

[0162] The adjustment unit 61 estimates that avatar A31 is positioned at the front of the group based on the group's movement path and direction. Here, avatar A31, being at the front of the group, is more likely to be within the field of view of other users, i.e., it is considered to attract more attention. Therefore, the adjustment unit 61 sets a high distribution priority for avatar A31.

[0163] Furthermore, if multiple avatars are located near the front of a group, the distribution priority of each of the multiple avatars may be set to be higher. If the average direction of movement of the group of avatars does not converge in one direction, such as when each user is acting independently, the adjustment unit 61 will determine that there are no users of high attention and will not adjust the amount of movement information.

[0164] As described above, in the server 2 of this technology, the amount of avatar movement information delivered to the head-mounted display terminal 1 is adjusted according to the relationship between users and the level of user attention.

[0165] Viewers are less likely to see avatars that correspond to users with low value to them, such as users with weak relationships or users who are not attracting attention. By reducing the amount of information in the movement data of avatars corresponding to users with low value to the viewer, it is possible to reduce the amount of information in the avatar movement data delivered to the head-mounted display terminal 1 while preventing viewers from feeling that it is unnatural.

[0166] ·others The above describes an example of VR (Virtual Reality) display in which a rendered image showing a virtual space with avatars reflecting the movements of other users is displayed on the head-mounted display terminal 1. Alternatively, the head-mounted display terminal 1 may also be configured to display AR (Augmented Reality) display in which avatars corresponding to other users are superimposed on a captured image obtained by photographing the space in which the viewer is actually located.

[0167] About computers The series of processes described above can be executed by hardware or by software. When the series of processes are executed by software, the programs that make up the software are installed from a program storage medium onto a computer that is built into dedicated hardware, or a general-purpose personal computer.

[0168] Figure 22 is a block diagram showing an example of the hardware configuration of a computer that executes the series of processes described above by a program. Server 2 is composed of, for example, a PC having a configuration similar to that shown in Figure 22.

[0169] The CPU (Central Processing Unit) 501, ROM (Read Only Memory) 502, and RAM (Random Access Memory) 503 are interconnected by a bus 504.

[0170] An input / output interface 505 is further connected to the bus 504. An input unit 506 consisting of a keyboard, mouse, etc., and an output unit 507 consisting of a display, speakers, etc. are connected to the input / output interface 505. In addition, a storage unit 508 consisting of a hard disk, non-volatile memory, etc., a communication unit 509 consisting of a network interface, etc., and a drive 510 that drives removable media 511 are connected to the input / output interface 505.

[0171] In a computer configured as described above, the CPU 501 performs the aforementioned series of processes by loading, for example, a program stored in the memory unit 508 into the RAM 503 via the input / output interface 505 and the bus 504 and executing it.

[0172] The program executed by the CPU 501 is recorded on removable media 511, for example, or provided via a wired or wireless transmission medium such as a local area network, the internet, or digital broadcasting, and installed in the storage unit 508.

[0173] The programs executed by the computer may be programs that are processed chronologically in the order described herein, or they may be programs that are processed in parallel or at necessary times, such as when they are called.

[0174] In this specification, a system refers to a collection of multiple components (devices, modules (parts), etc.), regardless of whether all components are located in the same enclosure. Therefore, multiple devices housed in separate enclosures and connected via a network, and a single device containing multiple modules within a single enclosure, are both considered systems.

[0175] Furthermore, the effects described herein are merely illustrative and not limiting, and other effects may also occur.

[0176] The embodiments of this technology are not limited to those described above, and various modifications are possible without departing from the spirit of this technology.

[0177] For example, this technology can be configured as cloud computing, where a single function is shared and processed collaboratively by multiple devices via a network.

[0178] Furthermore, each step described in the flowchart above can be performed by a single device, or it can be divided and performed by multiple devices.

[0179] Furthermore, if a single step includes multiple processes, those processes can be executed by a single device or shared among multiple devices.

[0180] <Examples of configuration combinations> This technology can also be configured as follows:

[0181] (1) A receiving unit that receives first motion information indicating the user's movements, A distribution unit distributes second movement information, which indicates the movement of the avatar corresponding to the user, to a terminal that displays the avatar. An adjustment unit adjusts the amount of information of the second motion information distributed by the distribution unit according to the first motion information. A distribution device equipped with the following features. (2) The adjustment unit adjusts the amount of information in the second motion information according to the distribution priority set based on at least the first motion information. The distribution device described in (1) above. (3) The first motion information is information about the position and angle of the user's joints, The second motion information includes information indicating the position and angle of the avatar's joints corresponding to the user's joints. The distribution device described in (2) above. (4) The adjustment unit adjusts the amount of information in the second motion information according to the distribution priority set based on the amount of change in the position and angle of the user's joints indicated by the first motion information. The distribution device according to (3). (5) The adjustment unit adjusts the frequency of distribution of the second motion information for each joint of the avatar according to the distribution priority. The distribution device described in (4) above. (6) If the type of user movement indicated by the first movement information is a predefined type, the adjustment unit sets the type of avatar movement corresponding to the type of user movement as the second movement information distributed by the distribution unit, using specific information that the terminal can identify. The distribution device according to any one of (2) to (5) above. (7) The adjustment unit adjusts the frequency of delivery of the second motion information for each joint of the avatar according to the delivery priority set based on the type of user movement indicated by the first motion information. The distribution device described in any of (3) to (5) above. (8) The adjustment unit adjusts the amount of information in the second motion information according to the distribution priority set based on the display mode of the avatar from the viewer's perspective using the terminal. The distribution device described in any of (2) to (7) above. (9) The adjustment unit adjusts the amount of information in the second motion information according to the distribution priority set based on the presence or absence of obstructions between it and the viewer's viewpoint. The distribution device described in (8) above. (10) The adjustment unit adjusts the amount of information in the second motion information according to the distribution priority set based on the amount of light reaching the avatar. The distribution device described in (8) or (9) above. (11) The adjustment unit adjusts the amount of information in the second motion information according to the distribution priority set based on the display area of ​​the avatar displayed on the terminal. The distribution device described in any of (8) to (10) above. (12) The adjustment unit adjusts the amount of information in the second motion information according to the distribution priority set based on the relationship between the viewer using the terminal and the user. The distribution device described in any of (2) to (11) above. (13) The adjustment unit adjusts the amount of information in the second motion information according to the distribution priority set based on the group to which the viewer belongs. The distribution device described in (12) above. (14) The adjustment unit adjusts the amount of information in the second motion information according to the distribution priority set based on the list indicating the connection between the user and the viewer. The distribution device described in (12) or (13) above. (15) The adjustment unit adjusts the amount of information in the second motion information according to the distribution priority set based on the number of times the user and the viewer have come into contact or come into proximity in the virtual space, or the time elapsed since the user and the viewer came into contact. The distribution device described in any of (12) to (14) above. (16) The adjustment unit adjusts the amount of information in the second motion information according to the distribution priority set based on the number of viewers who include the avatar in their field of view. The distribution device described in any of (12) to (15) above. (17) The adjustment unit adjusts the amount of information in the second motion information according to the distribution priority set based on whether or not the user has made a statement. The distribution device described in any of (12) to (16) above. (18) The adjustment unit adjusts the amount of information in the second motion information according to the distribution priority set based on the path or direction in which the multiple avatars move in the virtual space. The distribution device described in any of (12) to (17) above. (19) The distribution device, Upon receiving the first motion information indicating the user's movements, The amount of information of the second motion information indicating the movement of the avatar, which is distributed to the terminal that displays the avatar corresponding to the user, is adjusted. The second motion information is distributed to the terminal. Distribution method. (20) On the computer, Upon receiving the first motion information indicating the user's movements, The amount of information of the second motion information indicating the movement of the avatar, which is distributed to the terminal that displays the avatar corresponding to the user, is adjusted. The second motion information is distributed to the terminal. A program to execute a process. [Explanation of Symbols]

[0182] 1 Head-mounted display terminal, 2 Server, 11 Input processing unit, 12 Transmission data processing unit, 13 Receiving unit, 14 Program processing unit, 15 Design data storage unit, 16 Display unit, 21 Motion information processing unit, 22 Operation unit, 31 Bone position holding unit, 32 User position acquisition unit, 41 Rendering unit, 51 Receiving unit, 52 User management unit, 53 Program processing unit, 54 Bone position holding unit, 55 Distribution unit, 101 Animation storage unit, 111 Motion matching database, 151 Part importance database, 201 Rendering unit, 202 Design data storage unit, 203 Support information holding unit, 211 Complementary unit, 251 Chat information acquisition unit, 261 User relationship holding unit

Claims

1. A receiving unit that receives first motion information indicating the user's movements, A distribution unit distributes second movement information, which indicates the movement of the avatar corresponding to the user, to a terminal that displays the avatar. An adjustment unit adjusts the amount of information of the second motion information distributed by the distribution unit according to the distribution priority set based on at least the first motion information. A distribution device equipped with the following features.

2. The first motion information is information indicating the position and angle of the user's joints, The second motion information includes information indicating the position and angle of the avatar's joints corresponding to the user's joints. The distribution device according to claim 1.

3. The adjustment unit adjusts the amount of information in the second motion information according to the distribution priority set based on the amount of change in the position and angle of the user's joints indicated by the first motion information. The distribution device according to claim 2.

4. The adjustment unit adjusts the frequency of distribution of the second motion information for each joint of the avatar according to the distribution priority. The distribution device according to claim 3.

5. If the type of user movement indicated by the first movement information is a predefined type, the adjustment unit sets specific information that allows the terminal to identify the type of avatar movement corresponding to the type of user movement as the second movement information distributed by the distribution unit. A distribution device according to any one of claims 1 to 4.

6. The adjustment unit adjusts the frequency of distribution of the second motion information for each joint of the avatar according to the distribution priority set based on the type of user movement indicated by the first motion information. A distribution device according to any one of claims 2 to 4.

7. The adjustment unit adjusts the amount of information in the second motion information according to the distribution priority set based on the display mode of the avatar from the viewer's perspective using the terminal. A distribution device according to any one of claims 1 to 6.

8. The adjustment unit adjusts the amount of information in the second motion information according to the distribution priority set based on the presence or absence of obstructions between it and the viewer's viewpoint. The distribution device according to claim 7.

9. The adjustment unit adjusts the amount of information in the second motion information according to the distribution priority set based on the amount of light reaching the avatar. The distribution device according to claim 7 or 8.

10. The adjustment unit adjusts the amount of information in the second motion information according to the distribution priority set based on the display area of ​​the avatar displayed on the terminal. A distribution device according to any one of claims 7 to 9.

11. The adjustment unit adjusts the amount of information in the second motion information according to the distribution priority set based on the relationship between the viewer using the terminal and the user. A distribution device according to any one of claims 1 to 10.

12. The adjustment unit adjusts the amount of information in the second motion information according to the distribution priority set based on the group to which the viewer belongs. The distribution device according to claim 11.

13. The adjustment unit adjusts the amount of information in the second motion information according to the distribution priority set based on the list indicating the connection between the user and the viewer. The distribution device according to claim 11 or 12.

14. The adjustment unit adjusts the amount of information in the second motion information according to the distribution priority set based on the number of times the user and the viewer have come into contact or come into proximity in the virtual space, or the time elapsed since the user and the viewer came into contact. A distribution device according to any one of claims 11 to 13.

15. The adjustment unit adjusts the amount of information in the second motion information according to the distribution priority set based on the number of viewers who include the avatar in their field of view. A distribution device according to any one of claims 11 to 14.

16. The adjustment unit adjusts the amount of information in the second motion information according to the distribution priority set based on whether or not the user has made a statement. A distribution device according to any one of claims 11 to 15.

17. The adjustment unit adjusts the amount of information in the second motion information according to the distribution priority set based on the path or direction in which the multiple avatars move in the virtual space. A distribution device according to any one of claims 11 to 16.

18. The distribution device, Receiving first motion information that indicates the user's movements, The amount of information of the second motion information indicating the movement of the avatar, which is distributed to the terminal that displays the avatar corresponding to the user, is adjusted according to the distribution priority set based on at least the first motion information. Distributing the second motion information to the terminal Distribution methods including those mentioned.

19. On the computer, Receiving first motion information that indicates the user's movements, The amount of information of the second motion information indicating the movement of the avatar, which is distributed to the terminal that displays the avatar corresponding to the user, is adjusted according to the distribution priority set based on at least the first motion information. Distributing the second motion information to the terminal A program that performs a process that includes the following.

Images