Data transmission device, data reception device, data processing method and program

Abstract

The present invention provides a device and a method for processing that allows a user's motion to be reflected in almost real-time in an avatar displayed on a display unit. The data transmission apparatus includes: a motion analysis unit that analyzes a motion of a user based on a detection value from a sensor attached to the user; and a transmission information generation unit that receives a result of the exercise analysis and generates transmission information that stores data representing the exercise of the user. A transmission information generation unit generates frame information which is a combination of a plurality of data blocks each storing data indicating movement of each body part of a user. The data receiving apparatus obtains data representing a motion of each body part of the user from a data block in the received frame information, and performs avatar motion control to reflect the motion of the user in the avatar displayed on the display unit.

Description

Technical Field

[0001] The present disclosure relates to a data transmitting device, a data receiving device, a data processing method, and a program. More specifically, the present disclosure relates to a data transmitting device, a data receiving device, a data processing method, and a program capable of analyzing a user's movements based on detection values from a sensor (motion sensor) worn on the user's body, and causing a character displayed on a PC, etc., to perform movements similar to those of the user in substantially real time. Background Art

[0002] In recent years, many users use various user devices such as PCs, smartphones, game equipment, or VR glasses that allow viewing of virtual reality images (VR images) to use game applications and virtual reality space applications (metaverse applications).

[0003] When using a game application or a virtual reality space application (metaverse application), various characters are displayed on a display unit of a user equipment, for example, and the user can operate the user equipment to cause the displayed characters to perform actions.

[0004] It should be noted that the character includes, for example, a virtual character in an application, an avatar that is a virtual image representing the user himself, and the like.

[0005] By operating the user interface (UI), physical buttons, switches, etc. displayed on the display power of the user equipment, the user can make the character displayed on the display unit perform various actions, such as jumping, walking, running, punching or dancing.

[0006] However, in many cases, the actions that a user can cause a character to perform are limited to actions predefined in an application such as a game application, and, for example, the original action considered by the user cannot be implemented.

[0007] It should be noted that a method of attaching a sensor to the user's body and analyzing the user's movement based on the sensor detection value is described in, for example, Patent Document 1 (WO 2018 / 123293 A).

[0008] When the sensor detection value is applied, the original action assumed by the user can be analyzed. However, it is difficult to reflect such user actions in real time on the actions of a character such as an avatar displayed on a PC, for example.

[0009] The configuration described in patent document 1 is a configuration in which motion data (dominant data) representing the action of a model pitching posture is recorded in advance, and then a user wearing a motion sensor is caused to perform a pitching motion to generate tactile stimulation for a body part of the user in which a difference from the dominant data occurs, thereby teaching the user a correct pitching posture.

[0010] In this patent document, motion data (leading data) representing a pitching posture action serving as a model is pre-recorded data, and this data is compared with the motion being performed by the user. This patent document does not disclose a process for reflecting the motion being performed by the user on a character displayed on a PC or the like in real time.

[0011] In order to reflect the user's movements analyzed based on the detection values of the sensors worn by the user on the character on the PC substantially in real time, many processes must be continuously performed in a short period of time, such as the user's movement analysis processing, analysis data transmission processing, and processing of applying the analysis data to the character.

[0012] (a) Analysis Processing of User's Movement Based on Detection Values of Sensors Worn by the User

[0013] (b) Recording Process of User's Motion Data Based on Analysis Results of Sensor Detection Values

[0014] (c) Role Application Processing of Recorded User Movement Data

[0015] In order to continuously perform all the processes of (a) to (c) so that the user's movements are performed as the character's movements substantially in real time, high-speed processing of a large amount of data is required, and it has been difficult to perform the processing on a general PC or smartphone so far.

[0016] One of the main factors in this regard is believed to be the lack of definition of a data format that allows the following processes to be performed continuously in a short period of time: analysis processing of motion data from wearable motion sensors, transmission processing of analysis results, and processing of applying the analysis results to characters.

[0017] Reference List

[0018] Patent Literature

[0019] Patent Document 1: WO 2018 / 123293 A Summary of the Invention

[0020] Problems to be solved by the present invention

[0021] The present disclosure is proposed, for example, in view of the aforementioned problems, and provides a data sending device, a data receiving device, a data processing method, and a program capable of implementing the following processing: analyzing the user's movements based on the detection values of a sensor (motion sensor) worn on the user's body, and reflecting movements similar to the user on a character displayed on a PC, etc., substantially in real time.

[0022] Solutions to Problems

[0023] A first aspect of the present disclosure is a data transmission device, comprising:

[0024] a motion analysis unit that analyzes the user's motion based on sensor detection values of a motion sensor worn on the user's body;

[0025] a transmission information generating unit that receives the analysis result of the motion analyzing unit and generates transmission information storing data indicating the motion of the user; and

[0026] The communication unit sends the transmission information generated by the transmission information generation unit to the data receiving device, wherein

[0027] The transmission information generation unit generates, as the transmission information, frame information in which a plurality of data blocks each storing data representing a motion in units of a body part of a user are combined.

[0028] Furthermore, a second aspect of the present disclosure is a data receiving device, comprising:

[0029] a received data analyzing unit for analyzing received data from the data sending device; and

[0030] a display control unit that receives the analysis result of the received data analysis unit and performs motion control of the avatar displayed on the display unit, wherein

[0031] The reception data analyzing unit acquires a plurality of data blocks in which motion data in units of body parts of the user on the data transmitting device side are respectively stored from frame information included in the reception data, and extracts the motion data in units of body parts of the user from the plurality of acquired data blocks, and

[0032] The display control unit performs avatar motion control by reflecting motion in units of user's body parts as motion in units of avatar parts using motion data in units of user's body parts extracted from the plurality of data blocks by the received data analysis unit.

[0033] In addition, a third aspect of the present disclosure is a data processing method executed in a data sending device, the data processing method comprising:

[0034] a motion analysis step, wherein the motion analysis unit analyzes the user's motion based on sensor detection values of a motion sensor worn on the user's body;

[0035] a transmission information generating step, wherein a transmission information generating unit receives the analysis result of the motion analyzing step and generates transmission information storing data indicating the motion of the user; and

[0036] The communication step, wherein the communication unit transmits the transmission information generated in the transmission information generating step to the data receiving device, wherein

[0037] The steps for generating the sending information include:

[0038] Frame information in which a plurality of data blocks each storing data representing motion in units of a body part of the user are combined is generated as the transmission information.

[0039] In addition, a fourth aspect of the present disclosure is a data processing method executed in a data receiving device, the data processing method comprising:

[0040] a received data analyzing step, wherein the received data analyzing unit analyzes the received data from the data transmitting device; and

[0041] A display control step, wherein a display control unit receives the analysis result in the received data analysis step and performs motion control of an avatar displayed on a display unit, wherein

[0042] The steps for receiving data analysis include:

[0043] a step of acquiring a plurality of data blocks in which motion data in units of body parts of the user on the data transmitting device side are respectively stored from frame information included in the received data; and

[0044] extracting motion data in units of user's body parts from the plurality of acquired data blocks, and

[0045] The display control steps include:

[0046] Avatar motion control is performed, wherein motion in units of user's body parts is reflected as motion in units of avatar parts using the motion data in units of user's body parts extracted from the plurality of data blocks in the received data analysis step.

[0047] Furthermore, a fifth aspect of the present disclosure is a program for causing a data transmission device to execute data processing, the data processing including:

[0048] a motion analysis step, causing the motion analysis unit to analyze the user's motion based on sensor detection values of a motion sensor worn on the user's body;

[0049] a transmission information generating step of causing the transmission information generating unit to receive the analysis result of the motion analyzing step and generate transmission information storing data indicating the motion of the user; and

[0050] The communication step causes the communication unit to transmit the transmission information generated in the transmission information generation step to the data receiving device, wherein

[0051] In the Send Information Generation step, perform the following steps:

[0052] Frame information in which a plurality of data blocks each storing data representing motion in units of a body part of the user are combined is generated as the transmission information.

[0053] Furthermore, a sixth aspect of the present disclosure is a program for causing a data receiving device to execute data processing, the data processing including:

[0054] a received data analyzing step, causing the received data analyzing unit to analyze the received data from the data transmitting device; and

[0055] a display control step of causing a display control unit to receive the analysis result in the received data analysis step and perform motion control of an avatar displayed on a display unit, wherein

[0056] In the Receive Data Analysis step, perform the following steps:

[0057] acquiring a plurality of data blocks in which motion data in units of body parts of the user on the data transmitting device side are respectively stored from frame information included in the received data, and

[0058] Extracting motion data in units of user's body parts from the plurality of acquired data blocks, and

[0059] In the Display Control step:

[0060] Avatar motion control is performed, wherein motion in units of user's body parts is reflected as motion in units of avatar parts using the motion data in units of user's body parts extracted from the plurality of data blocks in the received data analysis step.

[0061] It should be noted that the program of the present disclosure is a program that can be provided via, for example, a storage medium or a communication medium that provides various program codes in a computer-readable format to an information processing device or a computer system that can execute the program codes. By providing such a program in a computer-readable format, processing corresponding to the program is implemented on the information processing device or computer system.

[0062] Other objects, features, and advantages of the present disclosure will become apparent from a more detailed description based on the examples and drawings of the present disclosure described later. It should be noted that in this specification, a system is a logical set configuration of multiple devices and is not limited to a case in which the devices of each configuration are in the same housing.

[0063] According to the configuration of the example of the present disclosure, an apparatus and method are realized that allow a process of reflecting a user's motion in an avatar displayed on a display unit substantially in real time.

[0064] Specifically, for example, the system includes a motion analysis unit that analyzes the user's motion based on detection values from sensors worn by the user, and a transmission information generation unit that receives the motion analysis results and generates transmission information storing data representing the user's motion. The transmission information generation unit generates frame information that combines multiple data blocks that each store data representing the motion of each body part of the user. The data receiving device obtains the data representing the motion of each body part from the data blocks in the received frame information and performs avatar motion control to reflect the user's motion in the avatar displayed on the display unit.

[0065] According to the present configuration, an apparatus and method are realized that allow processing to reflect the user's motion in the avatar displayed on the display unit substantially in real time.

[0066] It should be noted that the effects described herein are merely examples and not limitations, and additional effects may also be provided. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 It is a diagram for describing the configuration of an information processing system and an information processing device, and an overview of processing to which the processing of the present disclosure is applicable.

[0068] Figure 2 It is a diagram explaining a processing example of the configuration to which the present disclosure is applied.

[0069] Figure 3 is a diagram explaining an example of attaching a sensor for acquiring motion information of a user.

[0070] Figure 4 is a diagram showing skeleton information and frame information constituting avatar control information transmitted from a data transmitting device to a data receiving device.

[0071] Figure 5 It is a diagram explaining a configuration example of skeleton information constituting avatar control information and data blocks constituting frame information.

[0072] Figure 6 is a diagram explaining the format of the data block.

[0073] Figure 7 is a diagram explaining a configuration example of a data block.

[0074] Figure 8 is a diagram explaining a configuration example of a data block.

[0075] Figure 9 is a diagram explaining a configuration example of data blocks constituting skeleton information.

[0076] Figure 10 is a diagram explaining a configuration example of data blocks constituting skeleton information.

[0077] Figure 11 is a diagram explaining a configuration example of a skeleton data array block constituting skeleton information.

[0078] Figure 12 is a diagram explaining an example of a skeletal configuration of an avatar.

[0079] Figure 13 is a diagram explaining a configuration example of data blocks constituting frame information.

[0080] Figure 14 is a diagram explaining a configuration example of data blocks constituting frame information.

[0081] Figure 15 is a diagram explaining a configuration example of frame data blocks constituting frame information.

[0082] Figure 16 It is a diagram explaining a configuration example of data blocks constituting a skeleton transformation data array block of frame information.

[0083] Figure 17 is a diagram explaining an example of conversion data generation processing.

[0084] Figure 18 is a diagram explaining avatar control information transmitted from a data transmitting device to a data receiving device.

[0085] Figure 19 It is a diagram explaining a configuration example in which a cross-dressing instruction data block is added as a data block of frame information.

[0086] Figure 20 It is a diagram explaining an example of avatar display using a change of dress instruction data block performed in a data receiving device.

[0087] Figure 21 It is a diagram explaining an example of avatar display using a change of dress instruction data block performed in a data receiving device.

[0088] Figure 22 1 is a flowchart showing the generation and transmission process of the cross-dressing instruction data block executed by the data transmission device.

[0089] Figure 23 is a flowchart showing an example of avatar display control processing using a change of dress instruction data block, which is performed by the data receiving device.

[0090] Figure 24is a diagram explaining an example of a process in which each of a plurality of data transmitting devices transmits avatar control information.

[0091] Figure 25 is a diagram explaining the configuration of two items of avatar control information transmitted from a data transmitting device to a data receiving device.

[0092] Figure 26 is a diagram explaining the configuration of two items of avatar control information transmitted from a data transmitting device to a data receiving device.

[0093] Figure 27 It is a diagram explaining an example of classification processing of avatar control information by IP address performed by the data receiving device.

[0094] Figure 28 is a diagram explaining an example of a process in which each of a plurality of data transmitting devices transmits avatar control information.

[0095] Figure 29 It is a diagram showing a flowchart for explaining the sequence of avatar display control processing in the case where a plurality of items of avatar control information executed by a data receiving apparatus are received.

[0096] Figure 30 It is a diagram explaining a specific example of the switching control process of multiple avatars in time units.

[0097] Figure 31 It is a diagram explaining a specific example of the switching control process of multiple avatars in time units.

[0098] Figure 32 It is a diagram explaining a data block configuration example of frame information including a priority information data block used when performing switching control processing of a plurality of avatars in time units.

[0099] Figure 33 is a diagram showing a flowchart for describing a processing sequence performed by the data transmission device in the case of performing a selection application process according to avatar priority levels.

[0100] Figure 34 : is a flowchart showing a processing sequence performed by the data receiving apparatus in the case of performing selection application processing according to avatar priority.

[0101] Figure 35 : is a flowchart showing a processing sequence performed by the data receiving apparatus in the case of performing selection application processing according to avatar priority.

[0102] Figure 36 It is a diagram explaining a specific example of realizing a process of causing an avatar to perform an action that a user wants the avatar to perform instead of the user's own motion.

[0103] Figure 37 It is a diagram explaining a data block configuration example to which frame information of a posture / posture information storage block that implements processing for causing an avatar to perform an action that a user wishes to cause the avatar to perform is added.

[0104] Figure 38 : is a flowchart showing a processing sequence of a data transmission device that generates and transmits frame information to which a posture / posture information storage block is added.

[0105] Figure 39 is a diagram showing a flowchart for explaining a processing sequence of a data receiving apparatus that receives and uses frame information to which a posture / posture information storage block is added.

[0106] Figure 40 is a diagram explaining a specific example of a usage example of an image data storage block that stores image data captured by a camera, such as a user's facial expression.

[0107] Figure 41 It is a diagram explaining a data block configuration example to which frame information of an image data storage block storing image data captured by a camera, such as a user's facial expression, is added.

[0108] Figure 42 It is a diagram explaining a specific example of a usage example of a voice data storage block that stores voice data acquired by a microphone, such as a user's spoken voice.

[0109] Figure 43 It is a diagram explaining a data block configuration example to which frame information of a voice data storage block storing voice data (such as a user's spoken voice) acquired by a microphone is added.

[0110] Figure 44 is a diagram explaining an example of performing motion control of an avatar by allocating motion of each user to an avatar based on the user by using motion data of a plurality of users.

[0111] Figure 45 is a diagram explaining a configuration example of storing frame information of data blocks representing motions of a plurality of users.

[0112] Figure 46 is a diagram explaining a configuration example of storing frame information of data blocks representing motions of a plurality of users.

[0113] Figure 47 is a diagram explaining a specific example of a usage example of a touch sensor analysis data storage block that stores detection data and analysis data of a touch sensor worn by a user.

[0114] Figure 48is a diagram explaining a configuration example to which frame information of a touch sensor analysis data storage block storing detection data and analysis data of a touch sensor worn by a user is added.

[0115] Figure 49 is a diagram explaining an example of detecting data and analyzing the data using a motion sensor mounted on a ball.

[0116] Figure 50 It is a diagram explaining a specific example of a usage example of a lighting control information storage block that stores lighting control information (lighting control program) in addition to the user's motion information.

[0117] Figure 51 It is a diagram explaining a configuration example to which frame information of a lighting control information storage block storing lighting control information (lighting control program) is added.

[0118] Figure 52 is a diagram explaining a configuration example of an information processing device such as a data transmitting device and a data receiving device.

[0119] Figure 53 is a diagram explaining a hardware configuration example of the information processing device according to the present disclosure. DETAILED DESCRIPTION

[0120] The details of the data transmission device, data reception device, data transmission method, and program according to the present disclosure will be described below with reference to the accompanying drawings. It should be noted that the description will be made based on the following items.

[0121] 1. Overview of Information Processing Systems to Which the Processing of the Present Disclosure is Applicable

[0122] 2. Configuration example of avatar control information and data block

[0123] 3. Data blocks that constitute skeleton information

[0124] 4. Data blocks that constitute frame information

[0125] 5. Sending and processing of avatar control information configured by skeleton information and frame information

[0126] 6. Added the generation, sending and application of frame information of avatar change instruction data block

[0127] 7. Example of processing when multiple data transmission devices transmit avatar control information

[0128] 8. Example of switching between multiple avatar controls based on multiple avatar control information in time units

[0129] 9. Example of using data blocks storing pose and posture information

[0130] 10. Other Examples

[0131] 11. Application Examples of the Configuration and Processing of the Present Disclosure

[0132] 12. Configuration Examples of Information Processing Equipment

[0133] 13. Hardware configuration examples of information processing equipment

[0134] 14. Summary of the Configuration of the Present Disclosure

[0135] [1. Configuration Example of Information Processing System to Which the Processing of the Present Disclosure is Applicable]

[0136] First, refer to Figure 1 and subsequent drawings describe a configuration example of an information processing system to which the processes of the present disclosure are applicable.

[0137] Figure 1 is a diagram showing a configuration example of an information processing system to which the processing of the present disclosure is applied.

[0138] Figure 1 A user 10 , a data sending device (eg, a smartphone) 30 , and a data receiving device (eg, a PC) 50 are shown.

[0139] It should be noted that the data transmission device 30 is not limited to a smartphone, and may be a tablet terminal, a PC, VR glasses having functions such as data processing and communication, or the like.

[0140] Furthermore, the data receiving device 50 is not limited to a PC, and may be a tablet terminal, a smartphone, VR glasses, or the like having functions such as data processing and communication.

[0141] The user 10 wears the motion sensors 20 on each body part of the user, such as the head, arms, and feet.

[0142] The motion sensor 20 is, for example, an acceleration sensor or an inertial measurement unit (IMU).

[0143] An IMU is a motion sensor that can simultaneously measure acceleration in the xyz three-axis directions and angular velocity around the three axes.

[0144] The detection value of the motion sensor 20 attached to each body part of the user 10 is input to a data transmission device (eg, a smartphone) 30 through proximity communication such as Bluetooth (registered trademark).

[0145] The data transmission device 30 analyzes the detection value of the motion sensor 20 of each body part of the user 10 and generates avatar control information 40 including motion data representing the user's motion and the like.

[0146] The avatar control information 40 generated by the data transmitting apparatus 30 is control information for controlling the movement of an avatar (character) displayed on the display unit of the data receiving apparatus 50 .

[0147] The avatar control information 40 generated by the data sending device 30 includes, for example, frame information (motion data) and skeleton information, wherein the frame information represents the user's movement (three-dimensional movement) every (1 / 50) second corresponding to the frame interval of the reproduced image frame (posture data frame) in the data receiving device 50 that implements the avatar reproduction processing, and the skeleton information represents the skeleton structure (three-dimensional structure) of the avatar.

[0148] It should be noted that capturing the user's motion (three-dimensional motion) every (1 / 50) second is an example, and other settings may be used. Furthermore, the interval at which the motion sensor 20 captures data for each body part of the user 10 and the frame interval of the reproduced image frames (gesture data frames) in the data receiving device 50 do not necessarily need to match. The screen update frequency is determined by the refresh rate of the display or system configuration, and even when the screen update frequency is 60 fps, the data acquisition interval from the motion sensor 20 can be set to (1 / 50) second.

[0149] The detailed configuration of the avatar control information 40 will be described later.

[0150] The data transmitting device 30 transmits the avatar control information 40 generated based on the detection value of the motion sensor 20 to the data receiving device (eg, PC) 50 , the avatar control information 40 including frame information (motion data) representing the user's motion and skeleton information.

[0151] The data receiving device 50 executes, for example, a game application, a virtual reality space application (metaverse application), or the like, and displays an avatar (virtual character) 51 on the display unit of the data receiving device 50. It should be noted that the avatar (virtual character) 51 is a virtual character that performs actions reflecting the movements of the user 10.

[0152] The data receiving device 50 applies the avatar control information 40 received from the data transmitting device 30 to the avatar 51 displayed on the display unit, and causes the avatar 51 to perform motions similar to those of the user 10 substantially in real time.

[0153] For example, in Figure 1 In the illustrated example, the user 10 dances, and the avatar 51 displayed on the display unit of the data receiving apparatus 50 dances similarly to the user 10 .

[0154] Will refer to Figure 2An example of the correspondence relationship between the motion of the user 10 and the avatar 51 displayed on the display unit of the data receiving device 50 is described.

[0155] exist Figure 2 , the time-sequential movement of the user 10 is shown in the upper portion. That is, the time (t1)→(t2)→(t3) represents the movement of the user 10 over time.

[0156] The lower portion shows the movement of the avatar 51 displayed on the display unit of the data receiving device 50. The avatar 51 is also displayed as performing similar dance movements as the user 10 over time (t1)→(t2)→(t3).

[0157] As described above, the information processing system of the present disclosure can reflect the movement of the user 10 substantially in real time on the movement of a character such as an avatar displayed on a display device such as a PC.

[0158] In the system of the present disclosure, in order to realize real-time processing, the data format of the avatar control information transmitted from the data transmitting device 30 to the data receiving device is defined in a predetermined format.

[0159] The data format of the avatar control information will be described in detail later.

[0160] Will refer to Figure 3 An example in which a sensor (motion sensor) is attached to the user 10 is described.

[0161] exist Figure 3 In the example shown, user 10 has the following eight motion sensors attached to various body parts:

[0162] (1) Right hand motion sensor S1;

[0163] (2) Left hand motion sensor S2;

[0164] (3) right arm motion sensor S3;

[0165] (4) Left arm motion sensor S4;

[0166] (5) right ankle motion sensor S5;

[0167] (6) Left ankle motion sensor S6;

[0168] (7) head motion sensor S7; and

[0169] (8) Waist motion sensor S8.

[0170] These eight motion sensors detect the movements of eight parts of the user's body, namely the right hand, left hand, right arm, left arm, right foot, left foot, head and waist.

[0171] like Figure 3 As shown in the figure, the motion sensors (IMU) attached to the user's right hand, left hand, right arm, left arm, right foot, left foot, head and waist respectively output the acceleration in the three-axis direction and the angular velocity of the three axes around the user's right hand, left hand, right arm, left arm, right foot, left foot, head and waist.

[0172] It should be noted that Figure 3 The example of attaching sensors shown in the figure is an example of attaching sensors to eight parts of the user's body. However, this is for example only, and there can be various modes of attaching sensors, such as attaching sensors only to the hands, attaching sensors only to the hands and feet, and attaching more sensors to various parts of the body.

[0173] The detection value of the motion sensor worn by the user 10 is first transmitted to the data transmission device 30, such as a smartphone of the user 10. The transmission process of the sensor detection value is performed by proximity communication such as Bluetooth (registered trademark).

[0174] The data sending device 30 that receives the sensor detection value analyzes the received sensor detection value, generates avatar control information 40 in a predetermined format representing the movement of the user 10, and sends the generated avatar control information 40 to the data receiving device 50, such as a PC that performs character and avatar reproduction processing.

[0175] As described above, the data transmission device 30 analyzes the detection values of the motion sensor 20 for each body part of the user 10 to generate the avatar control information 40 representing the user's motion. The avatar control information 40 includes, for example, frame information corresponding to motion data representing the user's motion every (1 / 50) second, skeleton information representing the skeleton structure of the avatar, and the like.

[0176] [2. Configuration example of avatar control information and data block]

[0177] Next, configuration examples of avatar control information and data blocks will be described.

[0178] Will refer to Figure 4 A configuration example of the avatar control information 40 as transmission data from the data transmission device (smartphone, etc.) 30 to the data reception device (PC, etc.) 50 is described.

[0179] like Figure 4 As shown in , the avatar control information 40 which is transmission data from the data transmission device (smartphone etc.) 30 to the data reception device (PC etc.) 50 includes the following two pieces of information.

[0180] (a) Skeleton information 41

[0181] (b) Frame information 42

[0182] like Figure 4 As shown in the lower left portion of , the skeleton information 41 is information indicating the structure of the avatar 51 reflecting the movement of the user 10 .

[0183] Specifically, the skeleton information 41 is structural information of the avatar indicating the arrangement of bones and joints at which the bones are rotatably connected.

[0184] Each bone is connected to another bone through a joint, and the bones are configured to be rotatable about the joint.

[0185] Skeleton information 41 , which is structural information indicating the structure of the avatar 51 reflecting the motion of the user 10 , is transmitted from the data transmitting device (smartphone, etc.) 30 to the data receiving device (PC, etc.) 50 .

[0186] It should be noted that the skeleton information 41 is transmitted from the data transmitting device (smartphone, etc.) 30 to the data receiving device (PC, etc.) 50 at intervals of 3 seconds, for example.

[0187] In principle, if the avatar 51 reflecting the movement of the user 10 does not change, the skeleton information 41 can be sent only once, but considering the occurrence of reception errors in the data receiving device (PC, etc.) 50, avatar change processing, etc., the skeleton information 41 can be sent at regular intervals (for example, 3 seconds intervals).

[0188] For example, in the case of changing an avatar reflecting the motion of the user 10 , the data transmitting device (smartphone, etc.) 30 transmits a request to change the avatar together with skeleton information of the new avatar to be changed to the data receiving device (PC, etc.) 50 .

[0189] Another item of data included in the transmission data from the data transmitting device (smartphone, etc.) 30 to the data receiving device (PC, etc.) 50 is Figure 4 Frame information 42 is shown in the lower right portion.

[0190] like Figure 4 As shown in the lower right portion of , the frame information 42 corresponds to motion data representing the motion of the user 10. That is, the frame information is motion information (motion data) in units of reproduced image frames as avatar reproduced images in the data receiving device (PC, etc.) 50.

[0191] The frame information 42 is configured by data expressing motion in units of avatar configuration units (skeletons).

[0192] For example, the frame information 42 is transmitted from the data transmitting device (smartphone, etc.) 30 to the data receiving device (PC, etc.) 50 for each reproduced image frame in the data receiving device (PC, etc.) 50 .

[0193] Specifically, for example, when image reproduction is performed at 50 fps (frames per second) in the data receiving device (PC, etc.), 50 items of frame information 42 are transmitted from the data transmitting device (smartphone, etc.) 30 to the data receiving device (PC, etc.) 50 per second.

[0194] That is, the data transmitting device (smartphone, etc.) 30 sequentially generates new frame information 42 reflecting the user's motion at (1 / 50) second intervals, and transmits the generated new frame information 42 to the data receiving device (PC, etc.) 50 at (1 / 50) second intervals.

[0195] It should be noted that, as described above, the example of acquiring and transmitting the user's motion (three-dimensional motion) every (1 / 50) second is for example only, and other settings may be used. In addition, the data acquisition interval from the motion sensor 20 for each body part of the user 10 and the frame interval of the reproduced image frame (gesture data frame) in the data receiving device 50 do not necessarily need to match.

[0196] As reference Figure 4 As described, the avatar control information 40 , which is transmission data from the data transmission device (smartphone, etc.) 30 to the data reception device (PC, etc.) 50 , includes the following two pieces of information.

[0197] (a) Skeleton information 41

[0198] (b) Frame information 42

[0199] Both pieces of information are configured as "data blocks" having a predetermined data format.

[0200] Will refer to Figure 5 A configuration example of data blocks constituting the skeleton information 41 and the frame information 42 is described.

[0201] like Figure 5 As shown in FIG, each item of the skeleton information 41 and the frame information 42 includes one data block.

[0202] A data block has areas for storing the following three items of data:

[0203] (a) Data size storage area

[0204] (b) Data type storage area

[0205] (c) Actual data storage area

[0206] Furthermore, various data blocks may be stored in the data portion of the data blocks constituting the skeleton information 41 and the frame information 42 .

[0207] That is to say, if Figure 5 As shown in , the data blocks constituting the skeleton information 41 and the frame information 42 have a configuration capable of storing another data block therein.

[0208] like Figure 5 As shown in , the data blocks stored in the skeleton information 41 and the frame information 42 can have various configurations.

[0209] There are possible various configurations, such as a configuration in which a plurality of data blocks are stored in parallel in one data block, and a multi-level data block storage configuration in which another data block is stored in a storage data block in one data block.

[0210] Will refer to Figure 6 Describes the data format of a data block.

[0211] like Figure 6 As shown in , a data block consists of the following three data sections:

[0212] (a) Data size storage area

[0213] (b) Data type storage area

[0214] (c) Actual data storage area

[0215] The "(a) data size storage area" is an area for storing the data size (data capacity (bytes, etc.)) of a data block.

[0216] The "(b) data type storage area" is an area for storing identification data such as the type of data block, that is, a block storage area of some kind. For example, type identification information indicating the type of data stored in the (c) data area is stored, such as whether the block storage data is header information, transmission source information, skeleton information, frame information, etc.

[0217] The "(c) actual data storage area" is a storage area for actual data of target data as a data block. For example, header information, transmission source information, skeleton data, and entity data such as bone position and rotation angle information are stored.

[0218] Figure 7 and 8 Shows the correspondence between various data block configurations and data formats.

[0219] (1) shows an example of the data format of one data block.

[0220] (2) shows an example of data blocks in a hierarchical configuration, in which another data block is stored inside the data portion of one data block.

[0221] (3) shows an example of data blocks having a two-level hierarchical configuration, in which another data block is stored inside the data portion of one data block, and a third data block is stored inside the data portion of the stored data block.

[0222] also, Figure 8 (4) shows an example of a data block having a hierarchical configuration in which a plurality of data blocks are stored in parallel inside the data portion of one data block.

[0223] exist Figure 7 and 8 In the configuration, each data block has a previous reference Figure 6 That is, one data block has a data format configured by the following three data segments.

[0224] (a) Data size storage area

[0225] (b) Data type storage area

[0226] (c) Actual data storage area

[0227] [3. Data blocks that constitute skeleton information]

[0228] Next, data blocks constituting the skeleton information will be described.

[0229] Figure 9 It is a diagram explaining the data blocks constituting the skeleton information 41.

[0230] like Figure 9 As shown in , the skeleton information 41 includes the following three blocks as the first-level storage data blocks, and the skeleton information itself is a data block.

[0231] (1) Header data block

[0232] (2) Send source data block

[0233] (3) Skeleton data block

[0234] The skeleton information 41 includes these three data blocks as storage data blocks in the first hierarchy.

[0235] Furthermore, the "(1) header data block" includes two data blocks as second-level storage data blocks:

[0236] Format type block

[0237] Version block.

[0238] Furthermore, the "(2) transmission source data block" includes two data blocks as second-level storage data blocks:

[0239] IP address blocks

[0240] Receive port block.

[0241] Furthermore, the following data blocks are included as second-level storage data blocks within the "(3) skeleton data block":

[0242] A block of bone data arrays.

[0243] Furthermore, the "skeleton data array block" includes the skeleton data blocks as storage data blocks, and each "skeleton data block" includes these data blocks as storage data blocks:

[0244] Bone ID Block

[0245] Parent bone ID block

[0246] Transform the data block.

[0247] Will refer to Figure 10 and subsequent figures describe the details of the stored data of each data block constituting the skeleton information 41.

[0248] First, refer to Figure 10 The storage data of the second-level data blocks stored in the three first-level storage data blocks constituting the skeleton information 41 is described, that is, stored in these three data blocks:

[0249] (1) Header data block

[0250] (2) Send source data block

[0251] (3) Skeleton data block.

[0252] The “format type block” which is a storage data block in the second hierarchy of the “(1) header data block” is a data block storing a format type and a format name which are information on the data format of the skeleton information 41 .

[0253] Furthermore, the “version block” is a data block that stores version information of the skeleton information 41 .

[0254] The “IP address block” as the second-level storage data block of the “(2) transmission source data block” is a data block storing the IP address of the transmission source of the skeleton information 41 (ie, the data transmission device 30 ).

[0255] Furthermore, the “reception port block” is a data block that stores a reception port identifier used in a case where the data transmission device 30 receives another skeleton information 41 from another device.

[0256] It should be noted that the data receiving device (PC, etc.) 50 can confirm the transmission source of the received skeleton information by referring to the IP address stored in the “IP address block” included in the skeleton information 41 received from the data transmitting device 30 .

[0257] The “skeleton data array block” as the storage data block in the second level of the “(3) skeleton data block” stores a “skeleton data block” corresponding to each bone constituting the skeleton of the avatar 51 reflecting the movement of the user 10, and each individual “skeleton data block” also includes these data blocks as storage data blocks:

[0258] Bone ID Block

[0259] Parent bone ID block

[0260] Transform the data block.

[0261] Will refer to Figure 11 Describes a specific example of the data stored in the "Skeleton Data Array Block".

[0262] like Figure 11 As shown in , the “skeleton data array block” stores “skeleton data blocks” corresponding to the respective bones constituting the skeleton of the avatar 51 that reflects the movement of the user 10 .

[0263] like Figure 11 As shown in

[0264] Bone data-block corresponding to torso bone 1

[0265] Bone data-block corresponding to torso bone 2

[0266] …

[0267] The bone data-block corresponding to the right hand bone, etc.

[0268] Figure 12 (1) shows an example of the skeleton configuration of the avatar 51. Figure 4 As depicted, the avatar 51 has a configuration in which a plurality of bones are connected by joints.

[0269] It should be noted that for the sake of description Figure 12 The skeleton configuration of the avatar 51 shown in (1) is simplified. In practice, more bones and joints with a higher density than the example shown in the figure can be set.

[0270] Figure 12(1) shows the skeleton positions of the trunk skeleton 1, the head skeleton, the right hand skeleton, the left hand skeleton, the right foot skeleton and the left foot skeleton. All the parts connected by the joints represented by the white circles are bones, and the ID (identifier) corresponding to each individual bone is set for each of these bones. The skeleton corresponding ID is recorded as Figure 11 One of the "skeleton data blocks" shown in the "skeleton data block" stores the "skeleton ID block" of the data block.

[0271] like Figure 11 As shown in , for example, the bone ID of the torso bone 1 is B001, the bone ID of the right hand bone is B023, and so on.

[0272] In addition, the ID of each bone's parent bone is recorded as Figure 11 The "parent bone ID block" of one of the "skeleton data blocks" shown in the storage data block is stored in the "parent bone ID block".

[0273] like Figure 12 As shown in (2) Skeleton Configuration (Parent-Child Relationship) in , all bones constituting the avatar are assigned to any node with the root as a vertex node in a hierarchical configuration.

[0274] It should be noted that, as described above, the diagrams are shown in a simplified manner for the sake of explanation. Figure 12 (1) shows the skeletal configuration of the avatar 51, and the skeletal configuration does not correspond to Figure 12 (2) Skeleton configuration (master-slave relationship).

[0275] like Figure 12 As shown in (2), each node constituting the hierarchical configuration of the skeleton is set as a node having a parent-child relationship with respect to another node. Two skeletons having a parent-child relationship are skeletons connected by a joint. Therefore, the movement of the two skeletons having a parent-child relationship represents an associated movement. In other words, the movement of the child skeleton is the movement affected by the movement of the parent skeleton.

[0276] For example, the parent node of the left hand bone is the left lower arm bone.

[0277] Furthermore, the parent node of the left lower arm bone is the left upper arm bone.

[0278] The parent node of the left upper arm bone is the left shoulder bone.

[0279] The parent node of the left shoulder bone is the torso bone m.

[0280] The torso bone m also has a parent node and eventually reaches the root node which is the topmost node.

[0281] In this way, each skeleton corresponding to a node that makes up the avatar has a parent-child relationship with respect to another node and is shown in a hierarchical configuration with the root node being the uppermost node.

[0282] In the Figure 11 In the "parent bone ID block" of one of the storage data blocks of the "bone data block" shown in Figure 12 The ID of the parent node of the bone defined by the bone hierarchy configuration shown in (2).

[0283] In addition, as Figure 11 The "Transformation Data Block" in one of the storage data blocks of the "Skeleton Data Block" shown in the figure records the position (x, y, z) and angle (w, x, y, z) of each bone. It should be noted that the angle (w, x, y, z) is the four-dimensional Euler angle information.

[0284] It should be noted that the position (x, y, z) and angle (w, x, y, z) information of each bone recorded in the “transformation data block” of the skeleton information 41 is the position (x, y, z) and angle (w, x, y, z) of the bone corresponding to the predefined avatar reference pose, e.g. Figure 12 The reference posture with hands and feet separated is shown in (1).

[0285] Regarding the frame information 42, the position (x, y, z) and angle (w, x, y, z) information of each bone is also recorded as a "transformation data block" corresponding to each bone. However, as this information, the position (x, y, z) and angle (w, x, y, z) information of each bone corresponding to the movement of the user 10 is recorded.

[0286] The "conversion data block" recorded in the frame information 42 will be described later.

[0287] It should be noted that, as described above, two bones in a parent-child relationship are connected by a joint, and the position (x, y, z) and angle (w, x, y, z) of the child node's bone are based on the position (x, y, z) and angle (w, x, y, z) of the child node's parent node's bone. In other words, the data is constrained to the position (x, y, z) and angle (w, x, y, z) of the parent node's bone.

[0288] [4. Data blocks constituting frame information]

[0289] Next, data blocks constituting the frame information will be described.

[0290] Figure 13 4 is a diagram explaining data blocks constituting the frame information 42.

[0291] like Figure 13 As shown in , the frame information 42 includes the following three blocks as the first-level storage data blocks, and the frame information itself is a data block.

[0292] (1) Header data block

[0293] (2) Send source data block

[0294] (3) Frame data block

[0295] These three blocks are included as a first level of storage data blocks.

[0296] "(1) Header Data Block" and "(2) Transmission Source Data Block" are the same as those in the previous reference. Figure 9 A data block similar to the skeleton information 41 described in the subsequent figures. In other words,

[0297] The "(1) header data block" includes these two data blocks as the second-level storage data blocks:

[0298] Format type block

[0299] Version block.

[0300] "(2) Transmission source data blocks" include these two data blocks as second-level storage data blocks:

[0301] IP address blocks

[0302] Receive port block.

[0303] "(3) Frame data block" is a data block specific to the frame information 42, and includes the following data blocks as second-level storage data blocks:

[0304] Frame number block

[0305] Timestamp Block

[0306] A block of arrays of bone transform data.

[0307] Furthermore, the stored data block as the “bone transformation data array block” includes a plurality of bone transformation data blocks, and the stored data block as the “bone transformation form data block” also includes these blocks:

[0308] Bone ID Block

[0309] Transform the data block.

[0310] Each of the plurality of skeleton transformation data blocks is a data block in which data representing movement in units of a body part of the user is respectively stored.

[0311] As described above, the frame information 42 includes a plurality of data blocks each storing data representing motion in units of a body part of the user.

[0312] Will refer to Figure 14 and subsequent figures describe the details of the stored data of each data block constituting the frame information 42.

[0313] First, refer to Figure 14 The following describes the storage data of the second-level data blocks stored in the three first-level data blocks constituting the frame information 42, namely, the data stored in these three data blocks:

[0314] (1) Header data block

[0315] (2) sending the source data block; and

[0316] (3) Frame data block.

[0317] "(1) Header Data Block" and "(2) Transmission Source Data Block" are the same as those in the previous reference. Figure 9 A data block similar to the skeleton information 41 described in the subsequent figures. In other words,

[0318] The “format type block” as the second-level storage data block of the “(1) header data block” is a data block storing a format type and a format name as information on the data format of the frame information 42 .

[0319] Furthermore, the “version block” is a data block storing version information of the frame information 42 .

[0320] The “IP address block” as the second-level storage block of the “(2) transmission source data block” is a data block storing the IP address of the transmission source of the frame information 42 (ie, the data transmission device 30 ).

[0321] Furthermore, the “reception port block” is a data block storing a reception port identifier used in a case where the data transmission device 30 receives another frame information 42 from another device.

[0322] It should be noted that by referring to the IP address stored in the “IP address block” included in the frame information 42 received from the data transmitting device 30 , the data receiving device (PC, etc.) 50 can confirm the transmission source of the received frame information.

[0323] Next, we will refer to Figure 15 The following figures give a description of these data blocks as storage data blocks in the second level of "(3) frame data blocks":

[0324] Frame number block

[0325] Timestamp Block

[0326] A block of arrays of bone transform data.

[0327] like Figure 15 As shown in

[0328] The "frame number block" is a data block storing the frame number of the reproduced image frame in the data receiving apparatus 50 that implements avatar reproduction by applying frame information.

[0329] The "time stamp block" is a data block storing a time stamp indicating a reproduction time of a reproduction image frame in the data receiving device 50 that implements avatar reproduction by applying frame information.

[0330] The "bone transformation data array block" is a data block in which "bone transformation data blocks" in units of bones are arranged and stored.

[0331] Will refer to Figure 16 Describes the configuration of the "bone transformation data block" stored in the "bone transformation data array block" on a bone-by-bone basis.

[0332] like Figure 16 As shown in , the “bone transformation data array block” stores a “bone transformation data block” corresponding to each bone constituting the skeleton of the avatar 51 that reflects the movement of the user 10.

[0333] like Figure 16 As shown in

[0334] Bone transform data-block corresponding to torso bone 1

[0335] Bone transform data-block corresponding to torso bone 2

[0336] …

[0337] Bone transform data-block corresponding to the right hand bone, etc.

[0338] As described above, the “skeletal transformation data block” is a set of data blocks corresponding to each bone constituting the skeleton of the avatar 51 that reflects the movement of the user 10 .

[0339] like Figure 16 As shown in , each "bone transformation data block" includes these data blocks as storage data blocks:

[0340] Bone ID Block

[0341] Transform the data block.

[0342] The "bone ID block" is a data block storing the respective IDs (identifiers) of the individual bones constituting the skeleton corresponding to the avatar.

[0343] like Figure 16As shown in , for example, the bone ID of the torso bone 1 is B001, the bone ID of the right hand bone is B023, and so on.

[0344] In addition, as Figure 16 In the "transformation data block" of one of the storage data blocks of the "bone transformation data block" shown in the figure, the position (x, y, z) and angle (w, x, y, z) of each bone are recorded.

[0345] It should be noted that the position (x, y, z) and angle (w, x, y, z) information of each bone recorded in the "transformation data block" of the skeleton information 41 described above is the position (x, y, z) and angle (w, x, y, z) of the bone corresponding to the predefined avatar reference pose, e.g. Figure 12 The reference pose with hands and feet separated is shown in .

[0346] On the other hand, the position (x, y, z) and angle (w, x, y, z) information of each bone corresponding to the movement of the user 10 is recorded in the “transformation data block” corresponding to the bone in the frame information 42 .

[0347] That is, the position (x, y, z) and angle (w, x, y, z) of each bone corresponding to the user's movement analyzed based on the detection value of the motion sensor 20 worn by the user 10 are recorded.

[0348] In this manner, each skeletal transformation data block is a data block that stores data representing movement in units of a user's body part.

[0349] That is, the frame information 42 includes a large number of data blocks each storing data representing motion in units of a body part of the user.

[0350] As described above, the frame information 42 is information generated every (1 / 50) second, for example, and records the position (x, y, z) and angle (w, x, y, z) information of each bone based on the user's movement in each frame (that is, the user's movement every (1 / 50) second).

[0351] The data receiving device (PC, etc.) 50 controls the movement of the avatar 51 displayed on the display unit of the data receiving device (PC, etc.) 50 by referring to the position (x, y, z) and angle (w, x, y, z) of each bone recorded in the "transformation data block" in units of bones stored in the frame information 42.

[0352] It should be noted that the data sending device (smart phone, etc.) 30 performs the following processing: analyzes the user's movement in each frame (that is, the user's movement every (1 / 50) second) based on the detection value of the motion sensor 20 worn by the user 10, and generates a "transformation data block" in units of bones that stores the position (x, y, z) and angle (w, x, y, z) information of each bone corresponding to the analyzed user's movement.

[0353] Will refer to Figure 17 A detailed sequence example of a bone-unit transformation data generation process stored in a bone-unit “transformation data block” executed in the data transmission device (smartphone or the like) 30 is described.

[0354] Figure 17 The drawings shown below are diagrams explaining a detailed sequence example of the conversion data generation process based on the sensor detection value performed by the data transmission device (smartphone, etc.) 30.

[0355] It should be noted that the conversion data is configuration data of the frame information 42 and corresponds to data representing the instantaneous motion of the user. For example, the conversion data constituting one item of the frame information 42 corresponds to data representing one item of the user's motion at (1 / 50) second intervals.

[0356] As described above, the data transmitting device 30 transmits the transformation data, for example, at intervals of (1 / 50) seconds to the data receiving device 50. The data receiving device 50 sequentially analyzes the transformation data received at intervals of (1 / 50) seconds and applies the transformation data to the avatar displayed on the display unit at intervals of (1 / 50) seconds.

[0357] With this process, the avatar displayed on the display unit of the data receiving device 50 can perform similar movements to those of the user 10 .

[0358] Figure 17 The lower part of shows a detailed sequence example of the conversion data generation process based on the sensor detection value performed by the data transmission device 30 .

[0359] As shown in the figure, the data transmission device 30 sequentially performs each of the following processes in the conversion data generation process based on the sensor detection value.

[0360] (Processing 1) Sensor detection value input processing

[0361] (Processing 2) Noise Removal Processing

[0362] (Process 3) Motion Vector Group Generation Process

[0363] (Process 4) Motion estimation process (motion estimation process to which the learning model is applied)

[0364] (Processing 5) Connection Angle Setting Vector Generation Processing

[0365] Each of the processes (process 1) to (process 5) is sequentially performed to generate "transformation data" representing the three-dimensional motion of the user at a specific moment.

[0366] "(Processing 1) Sensor detection value input processing" is a processing of inputting detection values of sensors attached to various body parts of the user (head, right arm, left arm, waist, right foot, left foot, etc.).

[0367] It should be noted that, as described earlier, the transmission process of the sensor detection value from the sensor to the data transmission device (eg, smartphone) 30 is performed using, for example, proximity communication such as Bluetooth (registered trademark) communication.

[0368] "(Processing 2) Noise removal processing" is processing for removing noise components from sensor detection values input to the data transmission device 30. For example, the noise components are removed by applying a prescribed noise removal algorithm, such as processing for removing high-frequency components of a prescribed frequency or higher.

[0369] “(Processing 3) Motion vector group generation processing” is processing of generating a motion vector representing the individual motion of each body part of the user to which the sensor is attached, based on the sensor detection value from which noise is removed.

[0370] As described above, an inertial measurement unit (IMU) is used as a sensor, for example.

[0371] IMU is a motion sensor that can simultaneously measure acceleration in the xyz three-axis directions, angular velocity around the xyz three-axis, etc., and the data sending device (such as a smart phone) 30 can obtain the IMU output values of the user's head, right arm, left arm, waist, right foot and left foot at six points.

[0372] The data transmitting device 30 generates a plurality of motion vectors corresponding to the respective body parts (head, right arm, left arm, waist, right foot and left foot) of the user based on the sensor output values of the respective body parts.

[0373] “(Processing 4) Motion estimation processing (motion estimation processing to which the learning model is applied)” is processing of estimating the three-dimensional motion of the user based on the motion vectors corresponding to the respective body parts of the user generated in (Processing 3).

[0374] For example, a three-dimensional motion estimation process of the user based on the motion vector is performed using a pre-generated learning model.

[0375] The learning model is a learning model that receives a motion vector corresponding to each body part of the user and outputs the user's motion.

[0376] The learning model is a learning model generated in advance using a large amount of sample data, such as correspondence data between a large number of different motion vector groups and the user's motions.

[0377] "(Processing 5) Connection part angle setting vector generation processing" is an interpolation processing of the user's movement estimated using the learning model in (Processing 4), and is a processing for generating movement information of the user's entire body by interpolating the movement and position of insufficient body parts in the transformation data estimated using the learning model.

[0378] By executing these (Process 1) to (Process 5), skeleton-corresponding “transformation data” corresponding to the three-dimensional motion of the user 10 is generated.

[0379] As reference Figure 17 As described, the conversion data generation process based on the sensor detection value includes a process of generating a motion vector based on each sensor detection value, a process of inputting the generated motion vector into a learning model to estimate the user's motion, and the like.

[0380] [5. Transmission Process of Avatar Control Information Configured by Skeleton Information and Frame Information]

[0381] Next, a description will be given of a transmission process of avatar control information configured by skeleton information and frame information.

[0382] Figure 18 is a diagram showing an example of a transmission process of the avatar control information 40 from the data transmitting device (smartphone, etc.) 30 to the data receiving device (PC, etc.) 50 .

[0383] The avatar control information 40 includes skeleton information 41 and frame information 42 .

[0384] As described above, the skeleton information 41 and the frame information 42 are both one data block, and a plurality of data blocks are stored therein.

[0385] The data transmitting device (smartphone, etc.) 30 stores a data block corresponding to the skeleton information 41 or the frame information 42 in, for example, a UDP packet, and transmits the data block to the data receiving device (PC, etc.) 50. The UDP packet is a communication packet generated according to the User Datagram Protocol (UDP).

[0386] It should be noted that the communication packets are not limited to UDP packets, and packets according to another protocol (eg, WebSocket, TCP, WebRTC, etc.) may be used.

[0387] like Figure 18 As shown in the lower part of , the data sending device (smart phone, etc.) 30 generates a skeleton information data block storage group 43 storing skeleton information 41 and a frame information data block storage group 44 storing frame information 42, and sends the generated groups to the data receiving device (PC, etc.) 50.

[0388] like Figure 18 As shown in the lower left of FIG, the skeleton information data block storage group 43 includes various data blocks constituting the skeleton information 41, that is, referring to Figure 9 Describes various skeleton information configuration data blocks.

[0389] Similarly, as shown in the lower right corner of FIG18, the frame information data block storage group 44 includes various data blocks constituting the frame information 42, that is, referring to Figure 13 Describes various frame information configuration data blocks.

[0390] It should be noted that the data transmitting device (smartphone, etc.) 30 transmits the skeleton information data block storage packet 43 storing the skeleton information 41 to the data receiving device (PC, etc.) 50 every 3 seconds, for example.

[0391] Furthermore, the frame information data block storage packet 44 storing the frame information 42 is transmitted to the data receiving device (PC etc.) 50 every (1 / 50) second, for example.

[0392] It should be noted that, as described above, if the avatar 51 reflecting the movement of the user 10 does not change, then in principle, the skeleton information 41 only needs to be transmitted once. However, considering the occurrence of reception errors in the data receiving device (PC, etc.) 50, the avatar change processing, etc., the skeleton information is transmitted at regular intervals (for example, every three seconds).

[0393] In the case of changing the avatar, the data transmitting device (smartphone, etc.) 30 transmits skeleton information of the new avatar to the data receiving device (PC, etc.) 50 according to the avatar change process generation timing.

[0394] Upon receiving skeleton information of the new avatar from the data transmitting device (smartphone, etc.) 30, the data receiving device (PC, etc.) 50 displays the new avatar on the display unit and then causes the new avatar to perform actions (movements) corresponding to the received frame information.

[0395] For the frame information 42, the data sending device (smart phone, etc.) 30 sequentially generates frame information data blocks reflecting the user's movement, for example, every (1 / 50) second, generates a new frame information data block storage group 44 every (1 / 50) second, and sends the group to the data receiving device (PC, etc.) 50.

[0396] As described previously, the data receiving device 50 sequentially analyzes the transformation data received at intervals of (1 / 50) seconds, and applies the analyzed transformation data to the avatar displayed on the display unit at intervals of (1 / 50) seconds.

[0397] Through this process, the avatar displayed on the display unit of the data receiving device 50 can perform similar movements to those of the user 10 .

[0398] [6. Added the generation, transmission and application processing of the frame information of the avatar change instruction data block]

[0399] Next, the generation, transmission, and application processing of the frame information to which the avatar change-dressing instruction data block is added will be described.

[0400] As understood from the foregoing description, the data transmitting device (smartphone, etc.) 30 generates the skeleton information 41 and the frame information 42 to be transmitted to the data receiving device (PC, etc.) 50 using a data block having a predetermined data format.

[0401] That is to say, as previously mentioned Figure 6 As described, the skeleton information 41 and the frame information 42 are generated using a plurality of data blocks configured by the following three data areas:

[0402] (a) Data size storage area

[0403] (b) Data type storage area

[0404] (c) Actual data storage area.

[0405] Various data can be stored in the data area of the data block, and the data receiving device (PC, etc.) 50 can refer to the "(b) data type" described above to determine the type of data stored in the "(c) data" area for each data block received from the data sending device (smart phone, etc.) 30, that is, determine what data the stored data is, and use the data stored in the "(c) data" area to perform processing based on the determination result.

[0406] The generation, transmission, and application processing of the frame information to which the avatar change-dressing instruction data block is added will be described later.

[0407] Figure 19 A data block configuration example of the frame information (data block) 42 generated by the data transmission device (smartphone, etc.) 30 is shown.

[0408] Figure 19 The data block configuration of the frame information 42 shown in FIG is the following data block configuration:

[0409] Among them, the "dressing instruction data block 71" is added to the previous reference Figure 13The data block configuration of the frame information 42 is described.

[0410] The “change-dress instruction data block 71 ” is a data block for instructing to change the attire of the avatar 51 during execution of motion control reflecting the motion of the user 10 in the data receiving device (PC, etc.) 50 .

[0411] like Figure 19 As shown in , in the “cross-dressing instruction data block 71 ”, data such as a cross-dressing identifier and cross-dressing trigger information (eg, frame number, time stamp, user operation, specific posture, gesture, etc.) are stored as storage data.

[0412] The data sending device (smartphone, etc.) 30 generates a "disguise instruction data block 71" storing these data, adds the "disguise instruction data block" as a configuration block of the frame information 42, and sends the "disguise instruction data block" to the data receiving device (PC, etc.) 50.

[0413] When a "change-dressing instruction data block 71" is detected from the frame information 42 received from the data sending device (smart phone, etc.) 30, the data receiving device (PC, etc.) 50 analyzes the stored data of the "change-dressing instruction data block 71" and implements a process of changing the attire of the avatar 51 displayed on the display unit of the data receiving device (PC, etc.) 50 based on the analysis result.

[0414] It should be noted that Figure 19 The position of the "change dress instruction data block 71" in the frame information 42 shown in the figure is for example, and the "change dress instruction data block 71" may be set to a position other than the position shown in the figure, such as a storage block of the frame data block.

[0415] Next, we will refer to Figure 20 A specific example of the processing executed in the data receiving device (PC, etc.) 50 is described.

[0416] Figure 20 The example shown in is an example in which a data receiving device (PC, etc.) 50 receives frame information 42(n) from a data sending device (smartphone, etc.) 30, and performs processing for changing the attire of an avatar 51 displayed on a display unit of the data receiving device (PC, etc.) 50 based on the stored data of the "change-dressing instruction data block 71" included in the received frame information 42(n).

[0417] It should be noted that this example shows a processing example in the following case:

[0418] Outfit identifier = D012

[0419] Dress-up trigger = frame number (f00235) or timestamp (t001253)

[0420] These data are stored as storage data of the "dressing instruction data block 71".

[0421] "Outfit identifier = D012" is an identifier for specifying the outfit after the change.

[0422] It should be noted that in the storage unit of the data receiving device (PC, etc.) 50, correspondence data between costume image data applicable to the avatar 51 and its identifier is pre-stored, and costume image data corresponding to the identifier can be acquired based on the costume identifier.

[0423] "Changing disguise trigger = frame number (f00235) or time stamp (t001253)" is changing disguise trigger information indicating the timing of changing disguise, and is information instructing to perform changing disguise processing at the timing of frame number = (f00235) or time stamp = (t001253).

[0424] According to these instructions, at the timing when an image having a designated frame number or a reproduction time corresponding to a designated time stamp arrives, the data receiving device (PC, etc.) 50 changes the attire of the avatar 51 being displayed.

[0425] It should be noted that the frame number information and timestamp information are information that can be obtained from the data block (frame number block, timestamp block) in the frame information 42, and the data receiving device (PC, etc.) 50 can obtain the frame number information and timestamp information by referring to the frame number block and timestamp block in the frame information 42.

[0426] like Figure 20 As shown in the upper right corner of (avatar change-dressing processing example), at the generation timing of the change-dressing trigger, that is, at the timing of the frame number = (f00235) or the timestamp = (t001253), the data receiving device (PC, etc.) 50 performs the processing of changing the costume of the avatar 51 displayed on the display unit of the data receiving device (PC, etc.) 50 to the costume of the specified costume identifier = D012.

[0427] It should be noted that if the data stored in the "Changing Costume Instruction Data Block 71" does not contain information specifying an outfit identifier, the data receiving device (PC, etc.) 50 may be configured to sequentially change outfits according to a predefined changing costume sequence. Alternatively, a process of changing to a randomly selected outfit may be implemented.

[0428] Figure 21 Shown from Figure 20 An example of processing when frame information 42 (n+k) is transmitted and received after a certain period of time has passed since the transmission and reception of the frame information 42 (n) shown in FIG.

[0429] Figure 211 is a diagram explaining the following processing: a data receiving device (PC, etc.) 50 receives frame information 42 (n+k) from a data sending device (smartphone, etc.) 30, and changes the attire of an avatar 51 displayed on a display unit of the data receiving device (PC, etc.) 50 based on the stored data of the "change-dressing instruction data block 71" included in the received frame information 42 (n+k).

[0430] Figure 21 The example shown in shows a processing example in the following case:

[0431] Outfit identifier = D258

[0432] Dress-up trigger = frame number (f00523) or timestamp (t003538)

[0433] These data are stored as storage data of the "dressing instruction data block 71".

[0434] like Figure 21 As shown in the upper right part of (Avatar Change-up Processing Example), at the generation timing of the change-up trigger, that is, at the timing of the frame number = (f00523) or the timestamp = (t003538), the data receiving device (PC, etc.) 50 performs a process of changing the costume of the avatar 51 displayed on the display unit of the data receiving device (PC, etc.) 50 to a costume having a specified costume identifier = D258.

[0435] It should be noted that, referring to Figure 20 and 21 The described cross-dressing processing example is an example in which the cross-dressing trigger information stored in the "cross-dressing instruction data block 71" is a frame number or a time stamp.

[0436] The cross-dressing trigger information stored in the "cross-dressing instruction data block 71" is not limited to these information, and for example, a specific user operation, a specific avatar posture, a gesture, etc. may be set as the trigger information.

[0437] Next, we will refer to Figure 22 and 23 The flowchart shown in describes the sequence of the processing performed by the data transmitting apparatus 30 and the processing performed by the data receiving apparatus 50 in the case where the avatar disguise processing is implemented.

[0438] It should be noted that the processing according to the flowchart described below can be executed according to a program stored in a storage unit of the data sending device 30 or the data receiving device 50, and is executed under the control of a control unit having a program execution function such as a CPU, for example.

[0439] First, refer to Figure 22The flowchart shown in describes a processing sequence performed by the data transmission device 30 in the case where the avatar dressing process is performed or not.

[0440] The following will describe in sequence Figure 22 The processing details of each step are shown in the flowchart.

[0441] (Step S101)

[0442] First, the data transmission device 30 such as a smartphone determines whether an input of an execution request for an avatar disguise process is detected in step S101 .

[0443] The execution request of the avatar dressing process is initiated, for example, by an input operation by the user on the data transmission device 30 such as a smartphone.

[0444] In a case where an input of an execution request for the avatar changing process is not detected, the process proceeds to step S102 , and in a case where an input of an execution request for the avatar changing process is detected, the process proceeds to step S103 .

[0445] (Step S102)

[0446] In the case that no input of a request for executing the avatar dressing process is detected in step S101, the data transmitting device 30 generates frame information not including the dressing instruction data block in step S102, that is, the frame information having the aforementioned reference Figure 13 The normal frame information 42 describes the data block configuration.

[0447] (Step S103)

[0448] On the other hand, in the case where an input of a request for executing the avatar change-dressing process is detected in step S101 , the data transmission device 30 generates a change-dressing instruction data block in step S103 .

[0449] As previously referred Figure 19 As described, the data transmission device 30 generates a cross-dressing instruction data block that stores data such as a cross-dressing identifier and cross-dressing trigger information (eg, frame number, time stamp, user operation, specific posture, gesture, etc.) as stored data.

[0450] (Step S104)

[0451] In addition, in step S104, the data transmitting device 30 generates frame information including the cross-dressing instruction data block generated in step S103, that is, the cross-dressing instruction data block generated in step S103. Figure 19 The depicted frame information 42 includes a cross-dressing instruction data block.

[0452] (Step S105)

[0453] When the frame information generation process of the frame information not including the change dress instruction data block in step S102 or the frame information including the change dress instruction data block in step S104 is completed, the data transmitting device 30 transmits the frame information generated in step S105 to the data receiving device 50 .

[0454] Next, we will refer to Figure 23 The flowchart shown in describes a processing sequence performed by the data receiving apparatus 50 in the case where the avatar dressing process is performed or not.

[0455] The following will describe in sequence Figure 23 The processing details of each step are shown in the flowchart.

[0456] (Step S121)

[0457] First, in step S121 , the data receiving device 50 such as a PC receives avatar control information including skeleton information and frame information from the data transmitting device 30 .

[0458] (Step S122)

[0459] Next, in step S122 , the data receiving device 50 analyzes the data blocks constituting the frame information received from the data transmitting device 30 .

[0460] (Step S123)

[0461] Next, in step S123 , the data receiving device 50 determines whether a cross-dressing instruction data block is detected from the frame information received from the data transmitting device 30 .

[0462] In the case where a cross-dressing instruction data block is detected from the received frame information, the process proceeds to step S124, and in the case where a cross-dressing instruction data block is not detected, the process proceeds to step S125.

[0463] (Step S124)

[0464] In the case where the change of dress instruction data block is detected from the received frame information in step S123, the process proceeds to step S124, and the data receiving device 50 performs avatar change of dress processing according to the stored data of the change of dress instruction data block received from the data transmitting device 30 in step S124.

[0465] Specifically, as mentioned above Figure 20 and 21 As described above, based on the stored data of the disguise instruction data block, a process of changing to a designated costume is executed according to the generation of a designated disguise trigger. It should be noted that, if no costume is designated, a disguise process according to a predetermined disguise sequence or random disguise is executed.

[0466] (Step S125)

[0467] The process of step S125 is performed regardless of whether the disguise process is performed, and is a step of performing motion control of the avatar displayed on the display unit of the data receiving device 50 based on the frame information received from the data transmitting device 30 .

[0468] Specifically, the movement of the avatar displayed on the display unit of the data receiving device 50 is controlled according to the stored data of the skeleton transformation data block included in the frame information received from the data transmitting device 30 .

[0469] (Step S126)

[0470] In addition, in step S126, the data receiving device 50 determines whether the process of receiving the frame information from the data sending device 30 is completed. If the process is not completed, the process returns to step S121 and repeats the process in step S121 and subsequent steps for the frame information to be received next.

[0471] In the case where it is determined in step S126 that the reception processing of the frame information from the data transmission device 30 has ended, the processing ends.

[0472] As described above, the data transmitting device 30 that transmits avatar control information including skeleton information and frame information can generate frame information including a disguise instruction data block for performing disguise processing of the avatar and transmit the frame information to the data receiving device 50 .

[0473] When receiving frame information including a cross-dressing indication data block from the data sending device 30, the data receiving device 50 can change to the processing of the represented costume at the represented trigger generation timing based on the data stored in the cross-dressing indication data block (such as a cross-dressing identifier, cross-dressing trigger information, etc.).

[0474] [7. Processing Example When Multiple Data Transmitting Devices Each Transmit Avatar Control Information]

[0475] Next, a description will be given of a processing example in the case where a plurality of data transmission devices each transmits avatar control information.

[0476] Will refer to Figure 24 A processing example in which a plurality of data transmitting devices each transmit avatar control information is described.

[0477] Figure 24 Two data transmitting devices, namely, data transmitting device a ( 30 a ), data transmitting device b ( 30 b ), and one data receiving device 50 are shown.

[0478] The data transmission device a (30a) inputs the sensor detection value from the motion sensor worn by the user a (10a), generates avatar control information (avatar a control information) reflecting the motion of the user a, and transmits the avatar control information to the data transmission device b (30b).

[0479] The avatar a control information includes skeleton information of the avatar a included in the image reproduced by the data receiving device 50 and frame information representing the motion of the user generated based on a detection value of a motion sensor worn by the user a.

[0480] Skeleton information includes the previous reference Figure 9 The frame information also includes the multiple data blocks described above. Figure 13 Described multiple data blocks.

[0481] The data transmission device b (30b) inputs the sensor detection value from the motion sensor worn by the user b (10b), and generates avatar control information (avatar b control information) reflecting the motion of the user b.

[0482] The avatar b control information includes skeleton information of the avatar b included in the image reproduced by the data receiving device 50 and frame information representing the motion of the user generated based on the detection value of the motion sensor worn by the user b.

[0483] The data transmitting device b (30b) transmits the avatar a control information received from the data transmitting device a (30a) and the avatar b control information generated by the data transmitting device b (30b) (ie, two kinds of avatar control information) to the data receiving device 50.

[0484] The data receiving device 50 controls the actions of the avatar a and the avatar b using the two types of avatar control information received from the data transmitting device b (30b).

[0485] Will refer to Figure 25 The configuration of two items of avatar control information transmitted from the data transmitting device b (30b) to the data receiving device 50 will be described.

[0486] like Figure 25 As shown in , the two items of avatar control information sent from the data transmitting device b (30b) to the data receiving device 50 are these two items of avatar control information:

[0487] Avatar a control information

[0488] Avatar b controls the information.

[0489] Avatar a control information includes the following information:

[0490] Avatar a skeleton information; and

[0491] The frame information corresponding to user a is generated based on the sensor detection value of user a.

[0492] The skeleton information of avatar a includes the previous reference Figure 9 The corresponding frame information of user a includes the above reference Figure 13 Describes multiple data blocks.

[0493] On the other hand, the avatar b control information includes the following information:

[0494] Avatar b skeleton information; and

[0495] Frame information corresponding to user b is generated based on the sensor detection values of user b.

[0496] The skeleton information of avatar b includes the previous reference Figure 9 The frame information corresponding to user b includes the data blocks described above. Figure 13 Describes multiple data blocks.

[0497] The data receiving device 50 needs to distinguish the two items of avatar control information (i.e., avatar a control information and avatar b control information) received from the data sending device b (30b), and implement the processing of applying the avatar a control information to avatar a and applying the avatar b control information to avatar b.

[0498] As described above, the transmission source data blocks included in the skeleton information and the frame information constituting the two items of avatar control information are used as information for reliably distinguishing the two items of avatar control information and performing the processing.

[0499] Will refer to Figure 26 A transmission source data block used to distinguish between avatar a control information and avatar b control information is described.

[0500] like Figure 26 As shown in , each of the skeleton information and the frame information constituting the two items of avatar control information includes a transmission source data block.

[0501] Each source data block stores an IP address block as a storage block, and the IP address block stores the IP address of the source.

[0502] Specifically, the IP address of the data transmission device a (30a) as the transmission source is stored in the IP address block in the transmission source data block included in the skeleton information (data block) and frame information (data block) constituting the avatar a control information.

[0503] Furthermore, the IP address of the data transmission device b (30b) as the transmission source is stored in the IP address block in the transmission source data block included in the skeleton information (data block) and frame information (data block) constituting the avatar b control information.

[0504] The data receiving device 50 classifies (ie, distinguishes) two items of avatar control information (ie, avatar a control information and avatar b control information) with reference to the IP addresses stored in the IP address block.

[0505] By classifying the avatar control information by IP address, it is possible to implement a process of applying the avatar a control information to the avatar a and applying the avatar b control information to the avatar b.

[0506] That is to say, if Figure 27 As shown in , based on the skeleton information (data block) of the avatar a control information and the IP address of the data sending device a in the IP address block stored in the frame information (data block), the data receiving device 50 determines that the skeleton information (data block) and the frame information (data block) are data blocks storing information corresponding to the avatar a, and applies the skeleton information and frame information to the avatar a displayed on the data receiving device 50.

[0507] Similarly, based on the skeleton information (data block) of the avatar b control information and the IP address of the data sending device b in the IP address block stored in the frame information (data block), the data receiving device 50 determines that the skeleton information (data block) and the frame information (data block) are data blocks storing information corresponding to the avatar b, and applies the skeleton information and frame information to the avatar b displayed on the data receiving device 50.

[0508] As described above, by referring to the skeleton information (data block) and the IP address of the IP address block stored in the frame information (data block), the data receiving device 50 can classify the received information into information (skeleton information and frame information) in units of skeleton information data blocks or frame information data blocks and in units of transmitting devices. As a result, the data receiving device 50 can correctly apply the corresponding user's movement to the avatar corresponding to each transmitting device, that is, to each of the multiple avatars corresponding to each user.

[0509] It should be noted that, referring to Figure 24 The processing example described is a processing example in which avatar a control information reflecting the movement of user a generated by data sending device a (30a) is sent to data sending device b (30b), and data sending device b (30b) sends the avatar a control information received from data sending device a (30a) together with the avatar b control information generated by the data sending device b (30b) to the data receiving device 50.

[0510] In addition to such processing examples, such as Figure 28 As shown in , each of the data transmitting device a ( 30 a ) and the data transmitting device b ( 30 b ) can individually transmit the generated avatar control information reflecting the motion of each user to the data receiving device 50 .

[0511] Furthermore, in this case, the data receiving device 50 receives avatar control information corresponding to different avatars from a plurality of data transmitting devices in parallel.

[0512] In this case, the data receiving device 50 refers to the skeleton information received from each data transmitting device and the IP address in the IP address block stored in the frame information, thereby classifying the information received in parallel from multiple devices into information (skeleton information and frame information) per transmitting device, in units of skeleton information data blocks or frame information data blocks. As a result, the data receiving device 50 can apply the corresponding user's movement to the avatar corresponding to each transmitting device, that is, each of the multiple avatars corresponding to each user.

[0513] Next, we will refer to Figure 29 The flowchart shown in describes a processing sequence performed in a case where the data receiving apparatus 50 receives multiple items of avatar control information corresponding to avatars.

[0514] The following will describe in sequence Figure 29 The process of each step is shown in FIG.

[0515] (Step S221)

[0516] First, in step S221 , the data receiving device receives avatar control information including skeleton information and frame information.

[0517] It should be noted that the data is stored in a UDP packet, and the skeleton information data block or the frame information data block is stored in the UDP packet.

[0518] For example, in the case of receiving multiple items of avatar control information a, b generated by multiple data transmitting devices a, b, the data receiving device receives the following four data blocks.

[0519] (a1) Skeleton information data block generated by data sending device a

[0520] (a2) Frame information data block generated by data transmission device a

[0521] (b1) Skeleton information data block generated by data sending device b

[0522] (b2) Frame information data block generated by data transmission device b

[0523] (Step S222)

[0524] Next, in step S222, the data receiving device analyzes data blocks constituting the skeleton information data block or the frame information data block stored in the received UDP packet.

[0525] In the data block analysis process, the data receiving device acquires the IP address stored in the IP address block of the transmission source data block.

[0526] (Step S223)

[0527] Next, as a result of the data block analysis process in step S222, the data receiving device determines whether different transmission source data (specifically, IP addresses) are detected from the plurality of received skeleton information data blocks and frame information data blocks.

[0528] In a case where different IP addresses are not detected from the plurality of received skeleton information data blocks or frame information data blocks, the determination in step S223 is “NO”.

[0529] In this case, the data receiving device determines that only the avatar control information corresponding to one avatar is received, and performs the process of step S224.

[0530] On the other hand, in a case where different IP addresses are detected from a plurality of received skeleton information data blocks and frame information data blocks, the determination in step S223 is “Yes”.

[0531] In this case, the data receiving apparatus determines that multiple items of avatar control information corresponding to a plurality of different avatars are received, and performs the processing of steps S225 to S226.

[0532] (Step S224)

[0533] In the case of "No" determination in step S223, that is, in the case where different IP addresses are not detected from the plurality of received skeleton information data blocks and frame information data blocks, the process in step S224 is performed.

[0534] In this case, the data receiving apparatus determines that only avatar control information corresponding to one avatar is received, and in step S224, performs a process of controlling movement of the avatar displayed on the display unit according to the stored data of the conversion data block included in the received frame information.

[0535] (Step S225)

[0536] On the other hand, in a case where the determination in step S223 is “YES”, that is, in a case where different IP addresses are detected from the plurality of received skeleton information data blocks and frame information data blocks, the processing in step S225 is performed.

[0537] In this case, the data receiving device determines that multiple items of avatar control information corresponding to multiple avatars are received, and in step S225, based on the skeleton information stored in each UDP packet or the sending source data (IP address) recorded in the sending source data block in the frame information, the constituent blocks of the skeleton information and frame information in units of UDP packets are classified into blocks in units of sending sources.

[0538] For example, in a case where two IP addresses of the data sending device a and the data sending device b are detected as IP addresses, the IP addresses are classified as avatar control information of the data sending device a or avatar control information of the data sending device b in units of skeleton information data blocks or frame information data blocks.

[0539] Specifically, based on the IP address of a skeleton information data block or the IP address block stored in a frame information data block, it is determined that all data blocks in the skeleton information data block or frame information data block are data blocks from the data transmitting device having the IP address. As described above, the data receiving device performs data transmission source determination processing in units of skeleton information data blocks and frame information data blocks.

[0540] (Step S226)

[0541] Next, in step S226, the data receiving device controls the movement of the avatar corresponding to the transmission source according to the data stored in the skeleton transformation data block of the frame information in units of the transmission source.

[0542] For example, when the IP address of the data sending device a and two IP addresses of the data sending device b are detected from multiple frame information data blocks, the multiple frame information data blocks are classified into data blocks corresponding to the data sending device a or data blocks corresponding to the data sending device b in units of frame information data blocks according to the IP addresses stored in the blocks.

[0543] Furthermore, the following control is implemented using the skeleton transformation data stored in the two frame information data blocks obtained as a result of the classification process.

[0544] (a) Controlling avatar a based on the skeleton transformation data block in the frame information data block in which the IP address of data transmitting device a is stored

[0545] (b) Controlling the avatar b based on the skeleton transformation data block of the frame information data block in which the IP address of the data transmitting device b is stored

[0546] (Step S227)

[0547] After any control process of motion control of one avatar using one item of avatar control information in step S224 or motion control of multiple avatars using multiple items of avatar control information in step S226 is performed, it is determined in step S227 whether reception of avatar control information ends.

[0548] In a case where the processing has not ended, the processing returns to step S221, and the processing in and after step S221 is performed on the newly received avatar control information.

[0549] In a case where it is determined in step S227 that the reception of the avatar control information is completed, the processing ends.

[0550] [8. Example of Switching Between Multiple Avatar Controls Based on Multiple Avatar Control Information in Time Units]

[0551] Next, an example of switching multiple avatar controls based on multiple avatar control information in units of time will be described.

[0552] Similar to the example described previously, the example described below is an example in which a data receiving device receives multiple avatar control information corresponding to multiple users and performs multiple avatar controls based on the avatar control information, but the example described below is an example of implementing a process of switching avatar controls in units of time.

[0553] Will refer to Figure 30 Describe this example.

[0554] Figure 30 Two data transmitting devices, namely, data transmitting device a ( 30 a ), data transmitting device b ( 30 b ), and one data receiving device 50 are shown.

[0555] The data transmitting device a (30a) inputs the sensor detection value from the motion sensor worn by the user a (10a), generates avatar control information (avatar a control information) reflecting the motion of the user a, and transmits the avatar control information to the data receiving device 50.

[0556] The data transmitting device b (30b) inputs the sensor detection value from the motion sensor worn by the user b (10b), generates avatar control information (avatar b control information) reflecting the motion of the user b, and transmits the avatar control information to the data receiving device 50.

[0557] Here, the avatar control information of user a (10a) is applied to the display Figure 30 The avatar on the data receiving device (PC) 50 shown in FIG.

[0558] That is, the user a (10a) performs a performance action (movement) as a performer.

[0559] like Figure 30 As shown in the time axis in the upper right corner of , the avatar a performing the performance is displayed on the data receiving device (PC) 50 from time t0 to t1 and from time t2.

[0560] Between time t1 and t2 , another avatar b corresponding to user b ( 10 b ) is displayed on the data receiving device (PC) 50 .

[0561] exist Figure 31 An example of display of the data receiving device (PC) 50 from time t1 to t2 is shown in FIG.

[0562] like Figure 31 As shown in , avatar b has a role of a host, and user b (10b) performs actions (movements) as a host.

[0563] At time t2, the display of the data receiving device (PC) 50 is switched to the performance execution screen of the avatar a.

[0564] As described above, the data receiving device (PC) 50 receives the avatar a control information and the avatar b control information from the data transmitting device a (30a) and the data transmitting device b (30b), respectively, and applies the avatar control information to the avatar a and the avatar b. In this example, the data receiving device (PC) 50 further performs selection control as to which avatar control information is applied and which avatar control is to be implemented.

[0565] Such selection control can be performed based on the data stored in the “priority information data block” newly set in the frame information 42, for example.

[0566] Figure 32 An example of the frame information 42 to which the "priority information data block 72" is added is shown.

[0567] Figure 32 The data block configuration of the frame information 42 shown in FIG is that the "priority information data block 72" is added to the previously referenced Figure 13 The data block configuration of the frame information 42 is described.

[0568] The “priority information data block 72 ” is a data block storing priority information applied to switch the avatar reflecting the user's motion in the data receiving device (PC, etc.) 50 .

[0569] like Figure 32 As shown in, for example, the output priority information as storage data is recorded in the "priority information data block 72", specifically, the avatar priority in units of time, such as the setting for prioritizing the output of data from time t0 to t1 and from t2.

[0570] This example corresponds to Figure 30 The priority information of avatar a is shown in .

[0571] On the other hand, Figure 31 The frame information corresponding to the avatar b shown in is stored with priority information indicating that the time from t1 to t2 has a high priority.

[0572] Figure 30 and 31 The data sending device a (30a) and the data sending device b (30b) shown in the figure generate a "priority information data block 72" that stores priority information, add the priority information data block as a configuration block of the frame information 42, and send the data block to the data receiving device (PC, etc.) 50.

[0573] When the "priority information data block 72" is detected from the frame information 42 received from the data sending device (smart phone, etc.) 30, the data receiving device (PC, etc.) 50 analyzes the stored data of the "priority information data block 72", changes the priority of the motion application processing for the avatar displayed on the display unit of the data receiving device (PC, etc.) 50 according to the analysis result, selects the avatar with a high priority, and only performs motion control of the selected avatar.

[0574] It should be noted that Figure 32 The position of the “priority information data block 72” in the frame information 42 shown in FIG. 1 is for example, and the “priority information data block 72” may be set to a position other than that shown in the figure, such as a storage block of the frame data block.

[0575] Next, we will refer to Figure 33 The flowcharts shown in and subsequent drawings describe a sequence of processing performed by the data transmitting apparatus 30 and processing performed by the data receiving apparatus 50 in the case where an avatar selection process according to priority is implemented and motion control of only the selected avatar is performed.

[0576] It should be noted that the processing according to the flowchart described below can be executed according to a program stored in a storage unit of the data sending device 30 or the data receiving device 50, and is executed under the control of a control unit having a program execution function such as a CPU, for example.

[0577] First, refer to Figure 33The flowchart shown in describes a processing sequence performed by the data transmission device 30 in the case of executing or not executing the selection application processing according to the avatar priority level.

[0578] The following will describe in sequence Figure 33 The processing details of each step are shown in the flowchart.

[0579] (Step S301)

[0580] First, the data transmission device 30 such as a smartphone determines whether an input of an execution request of an avatar selection process according to priority is detected in step S301 .

[0581] The execution request of the avatar selection process according to the priority level is initiated, for example, by an input operation by the user on the data transmission device 30 such as a smartphone.

[0582] In a case where an input of an execution request of the avatar selection process according to the priority is not detected, the process proceeds to step S302 , and in a case where the input is detected, the process proceeds to step S303 .

[0583] (Step S302)

[0584] In the case where the input of the execution request of the avatar selection process according to the priority is not detected in step S301, the data transmitting device 30 generates frame information not including the priority information data block in step S302, that is, the frame information having the aforementioned reference Figure 13 The normal frame information 42 describes the data block configuration.

[0585] (Step S303)

[0586] On the other hand, in the case where an input of an execution request for the avatar selection process according to the priority is detected in step S301, the data transmission device 30 generates a priority information data block in step S303.

[0587] As previously referred Figure 32 As described, the data transmission device 30 generates a priority information data block as storage data in which avatar priority data in time units is stored, such as output priority information, specifically, settings such as prioritizing output from time t0 to t1 and from t2.

[0588] (Step S304)

[0589] In addition, in step S304, the data transmitting device 30 generates frame information including the priority information data block generated in step S303, that is, the frame information of the data block of the priority information generated in step S303. Figure 32 Frame information 42 is depicted including a priority information data block.

[0590] (Step S305)

[0591] When the frame information generation process of the frame information excluding the priority information data block is completed in step S302 or the frame information generation process of the frame information including the priority information data block is completed in step S304 , the data transmitting device 30 transmits the frame information generated in step S305 to the data receiving device 50 .

[0592] Next, we will refer to Figure 34 and 35 The flowchart shown in describes a processing sequence performed by the data receiving apparatus 50 in the case where a selection application process according to the avatar priority is executed or not executed.

[0593] The following will describe in sequence Figure 34 and 35 The processing details of each step of the process are shown in .

[0594] (Step S321)

[0595] First, in step S321 , the data receiving device 50 such as a PC receives avatar control information including skeleton information and frame information from the data transmitting device 30 .

[0596] It should be noted that data is stored in UDP packets, and skeleton information (data block) or frame information (data block) is stored in UDP packets.

[0597] For example, in the case of receiving multiple items of avatar control information a, b generated by multiple data transmitting devices a, b, the data receiving device receives the following four data blocks.

[0598] (a1) Skeleton information data block generated by data sending device a

[0599] (a2) Frame information data block generated by data transmission device a

[0600] (b1) Skeleton information data block generated by data sending device b

[0601] (b2) Frame information data block generated by data transmission device b

[0602] (Step S322)

[0603] Next, in step S322, the data receiving device analyzes the skeleton information data block or the data blocks constituting the frame information data block stored in the received UDP packet.

[0604] In the data block analysis process, the data receiving device acquires the IP address stored in the IP address block of the transmission source data block.

[0605] (Step S323)

[0606] Next, the data receiving device determines whether different transmission source data (specifically, IP addresses) are detected from the plurality of received skeleton information data blocks and frame information data blocks as a result of the data block analysis process in step S223.

[0607] In a case where different IP addresses are not detected from the plurality of received skeleton information data blocks or frame information data blocks, the determination in step S323 is “NO”.

[0608] In this case, the data receiving apparatus determines that only the avatar control information corresponding to one avatar is received, and performs the process of step S324.

[0609] On the other hand, in a case where different IP addresses are detected from a plurality of received skeleton information data blocks and frame information data blocks, the determination in step S323 is “Yes”.

[0610] In this case, the data receiving device determines that multiple items of avatar control information corresponding to a plurality of different avatars are received, and performs the processing in step S325 and subsequent steps.

[0611] (Step S324)

[0612] In the case of “NO” determination in step S323 , that is, in the case where different IP addresses are not detected from the plurality of received skeleton information data blocks and frame information data blocks, the process in step S324 is performed.

[0613] In this case, the data receiving apparatus determines that only avatar control information corresponding to one avatar is received, and in step S324, performs a process of controlling movement of the avatar displayed on the display unit according to the stored data of the conversion data block included in the received frame information.

[0614] (Step S325)

[0615] On the other hand, in a case where the determination in step S323 is “YES”, that is, in a case where different IP addresses are detected from the plurality of received skeleton information data blocks and frame information data blocks, the processing in step S325 is performed.

[0616] In this case, the data receiving device determines that multiple items of avatar control information corresponding to multiple avatars are received, and in step S325, based on the skeleton information stored in each UDP packet or the sending source data (IP address) recorded in the sending source data block in the frame information, the constituent blocks of the skeleton information and frame information in units of UDP packets are classified into blocks in units of sending sources.

[0617] For example, in a case where two IP addresses of the data sending device a and the data sending device b are detected as IP addresses, the IP addresses are classified as avatar control information of the data sending device a or avatar control information of the data sending device b in units of skeleton information data blocks or frame information data blocks.

[0618] Specifically, based on the IP address of a skeleton information data block or the IP address block stored in a frame information data block, it is determined that all data blocks in the skeleton information data block or frame information data block are data blocks from the data transmitting device having the IP address. As described above, the data receiving device performs data transmission source determination processing in units of skeleton information data blocks and frame information data blocks.

[0619] (Step S326)

[0620] Next, in step S326, the data receiving device determines whether a priority information data block is detected in the received frame information.

[0621] In the case where a priority information data block is detected in the frame information, the process proceeds to step S327.

[0622] On the other hand, in the case where the priority information data block is not detected in the frame information, the process proceeds to step S329.

[0623] (Step S327)

[0624] In a case where the priority information data block is detected in the received frame information in step S326, the data receiving apparatus selects an avatar having a high priority according to the priority information stored in the priority information data block detected from the frame information in step S327.

[0625] (Step S328)

[0626] Next, in step S328, the data receiving device selectively controls the avatar with high priority selected in step S327.

[0627] That is, based on the data stored in the skeleton transformation data block corresponding to the frame information of the selected avatar with high output priority, a process of controlling the movement of only the avatar with high priority is performed.

[0628] (Step S329)

[0629] On the other hand, in the case where the priority information data block is not detected in the received frame information in step S326, the data receiving device controls the plurality of avatars simultaneously in step S329.

[0630] Specifically, the movement of the avatar corresponding to the transmission source is simultaneously controlled based on the data stored in the skeleton transformation data block of the frame information of the transmission source unit. That is, for example, the following control is performed.

[0631] (a) Controlling avatar a based on the skeleton transformation data block in the frame information data block in which the IP address of data transmitting device a is stored

[0632] (b) Controlling avatar b based on the skeleton transformation data block in the frame information data block in which the IP address of the data transmitting device b is stored

[0633] In this way, control of a plurality of avatars is performed based on the plurality of received avatar control information.

[0634] (Step S330)

[0635] After either the motion control for only one avatar having a high priority in step S328 or the motion control for a plurality of avatars in step S329 is performed, it is determined in step S330 whether the reception of the avatar control information ends.

[0636] In a case where the processing has not ended, the processing returns to step S321, and the processing in and after step S321 is performed on the newly received avatar control information.

[0637] In a case where it is determined in step S330 that the reception of the avatar control information is completed, the process ends.

[0638] [9. Example of using a data block storing pose and posture information]

[0639] Next, an example of using a data block storing posture and pose information will be described.

[0640] As described above, in the system of the present disclosure, frame information reflecting the user's movement based on the detection value of the motion sensor worn by the data sending device (smart phone, etc.) 30 on the user is sequentially sent to the data receiving device (PC, etc.) 50, and the avatar displayed on the display unit of the data receiving device (PC, etc.) 50 performs movement similar to the user's movement.

[0641] However, a user may not be able to perform the ideal movement that the user wants the avatar to perform, for example.

[0642] For example, even if the user wants the avatar to perform a "roll" motion, the user may not be able to perform the roll, or even if the user wants the avatar to perform a high-speed and agile "punch" motion, the user may not be able to perform a high-speed and agile "punch" motion.

[0643] Will refer to Figure 36 A specific example is described. An image in which avatar a and avatar b are fighting is displayed on the display unit of the data receiving device (PC, etc.) 50.

[0644] Avatar a performs movements that mirror the movements of user a shown in the figure.

[0645] User a (10a) wears a motion sensor, and frame information reflecting motion based on the detection value of the motion sensor worn by user a (10a) is generated in data transmitting device a (30a), and the frame information is transmitted to data receiving device (PC, etc.) 50.

[0646] The data receiving device (PC, etc.) 50 controls the movement of the avatar based on the frame information received from the data transmitting device a (30a).

[0647] For example, when the user a (10a) performs a fist-throwing motion, the avatar a displayed on the data receiving device 50 also performs a fist-throwing motion.

[0648] However, although the user a (10a) wants the avatar to perform a punch that is high-speed and agile, the user a (10a) can only perform a slow punch.

[0649] In this case, for example, it is possible to cause the user a (10a) to perform a high-speed and agile "punch" motion that the user wants the avatar a to perform.

[0650] The example described below is an example of realizing a process of causing an avatar to perform an action that the user wants the avatar to perform, rather than the user's own movement itself.

[0651] In this example, a data block storing posture and posture information (ie, a "posture / posture information storage block") is added to frame information 42 sent from a data transmitting device (smartphone, etc.) 30 to a data receiving device (PC, etc.) 50.

[0652] Figure 37 A data block configuration example of the frame information (data block) 42 generated by the data transmission device (smartphone, etc.) 30 is shown.

[0653] Figure 37 The data block configuration of the frame information 42 shown in FIG is one in which the "posture / posture information storage block 73" is added to the previously referenced Figure 13 The data block configuration of the frame information 42 is described.

[0654] The “posture / posture information storage block 73 ” is a data block for instructing control in the data receiving device (PC, etc.) 50 to cause the avatar to perform motions corresponding to various postures and poses prepared in advance without using skeletal transformation data reflecting the motions of the user 10 .

[0655] like Figure 37 As shown in , the “gesture / posture information storage block 73” includes, for example, gestures, gesture specifying information (ie, a gesture that replaces the motion of the avatar with a specified gesture or gesture), and gesture specifying information as stored data.

[0656] For example, various posture and gesture designation information such as "roll," "quick punch," "sit upright," and "jump" is used.

[0657] The data transmitting device (smartphone, etc.) 30 generates a “posture / posture information storage block 73 ” storing posture and posture specifying information, adds it as a configuration block of the frame information 42 , and transmits it to the data receiving device (PC, etc.) 50 .

[0658] When the "posture / posture information storage block 73" is detected from the frame information 42 received from the data sending device (smart phone, etc.) 30, the data receiving device (PC, etc.) 50 analyzes the storage data of the "posture / posture information storage block 73" and performs action control of the avatar 51 displayed on the display unit of the data receiving device (PC, etc.) 50 based on the analysis result.

[0659] In the storage unit of the data receiving device (PC, etc.) 50, motion control data for performing avatar action control corresponding to various postures and gestures is pre-stored. The data receiving device (PC, etc.) 50 obtains the motion control data corresponding to the specified posture and gesture from the storage unit according to the posture and gesture designation data stored in the "posture / posture information storage block 73" received from the data sending device 30, and applies the motion control data to the avatar.

[0660] By implementing such processing, it is possible to make the avatar perform the ideal action desired by the user.

[0661] It should be noted that Figure 37 The position of the "posture / posture information storage block 73" in the frame information 42 shown in the figure is for example, and the "posture / posture information storage block 73" can be set to a position other than the position shown in the figure, such as a storage block of the frame data block.

[0662] Next, we will refer to Figure 38 and 39The flowchart shown in describes a sequence of processing performed by the data transmitting apparatus 30 and processing performed by the data receiving apparatus 50 in a case where a posture / posture information storage block is set in the frame information 42 and avatar motion control is implemented.

[0663] It should be noted that the processing according to the flowchart described below can be executed according to a program stored in a storage unit of the data sending device 30 or the data receiving device 50, and is executed under the control of a control unit having a program execution function such as a CPU, for example.

[0664] Next, we will refer to Figure 38 The flowchart shown in describes a processing sequence performed by the data transmission device 30.

[0665] The following will describe in sequence Figure 38 The processing details of each step are shown in the flowchart.

[0666] (Step S401)

[0667] First, the data transmission device 30 such as a smartphone determines whether an input of an execution request for a gesture / posture replacement process is detected in step S401 .

[0668] The execution request of the gesture / posture replacement process is implemented, for example, by an input operation by a user on the data transmission device 30 such as a smartphone.

[0669] In a case where an input of an execution request for the gesture / posture replacement process is not detected, the process proceeds to step S402 , and in a case where the input is detected, the process proceeds to step S403 .

[0670] (Step S402)

[0671] In the case where no input of an execution request for the posture / posture replacement process is detected in step S401, the data transmitting device 30 generates frame information that does not include the posture / posture information storage block, that is, the frame information having the previously referenced Figure 13 The normal frame information 42 describes the data block configuration.

[0672] (Step S403)

[0673] On the other hand, in the case where input of an execution request of the gesture / posture replacement process is detected in step S401 , the data transmission device 30 generates a gesture / posture information storage block in step S403 .

[0674] As previously referred Figure 37As described, the data sending device 30 generates a posture / posture information storage block as storage data, which stores, for example, postures, posture designation information (i.e., postures that replace the movement of the avatar with prescribed postures or postures), and posture designation information.

[0675] Specifically, for example, a posture / posture information storage block storing various postures and posture specifying information such as “roll,” “punch,” “sit up straight,” and “jump” is generated.

[0676] (Step S404)

[0677] Furthermore, in step S404, the data transmitting device 30 generates frame information including the posture / posture information storage block generated in step S403, that is, the frame information of the posture / posture information storage block generated in step S403. Figure 37 Frame information 42 is depicted including a posture / posture information storage block.

[0678] (Step S405)

[0679] When the frame information generation processing of the frame information excluding the posture / posture information storage block is completed in step S402 or the frame information generation processing of the frame information including the posture / posture information storage block is completed in step S404, the data transmitting device 30 transmits the frame information generated in step S405 to the data receiving device 50.

[0680] Next, we will refer to Figure 39 The flowchart shown in describes a processing sequence executed by the data receiving device 50 that implements avatar motion control based on the posture / posture information storage block set in the frame information 42.

[0681] The following will describe in sequence Figure 39 The processing details of each step are shown in the flowchart.

[0682] (Step S421)

[0683] First, in step S421 , the data receiving device 50 such as a PC receives avatar control information including skeleton information and frame information from the data transmitting device 30 .

[0684] (Step S422)

[0685] Next, in step S422 , the data receiving device 50 analyzes the data blocks constituting the frame information received from the data transmitting device 30 .

[0686] (Step S423)

[0687] Next, in step S423 , the data receiving device 50 determines whether a posture / posture information storage block is detected from the frame information received from the data transmitting device 30 .

[0688] In a case where a posture / posture information storage block is detected from the received frame information, the process proceeds to step S424 , and in a case where a posture / posture information storage block is not detected, the process proceeds to step S425 .

[0689] (Step S424)

[0690] In step S423, when a posture / posture information storage block is detected from the received frame information, the processing continues to step S424, and in step S424, the data receiving device 50 performs avatar motion control according to the storage data of the posture / posture information storage block received from the data sending device 30.

[0691] Specifically, according to the replacement target posture (such as a fist-shaking posture, a sitting-up posture, etc.) of the storage data or posture information designated as the posture / posture information storage block, the movement of the avatar is controlled by applying the posture or posture corresponding avatar control information pre-stored in the storage unit of the data receiving device 50.

[0692] (Step S425)

[0693] On the other hand, in a case where no posture / posture information storage block is detected from the received frame information in step S423, the processing continues to step S425, and the data receiving device 50 performs avatar motion control in step S425 based on the frame information received from the data sending device 30 that does not include the posture / posture information storage block.

[0694] Specifically, the movement of the avatar displayed on the display unit of the data receiving device 50 is controlled according to the stored data of the skeleton transformation data block reflecting the movement of the user included in the frame information received from the data transmitting device 30 .

[0695] (Step S426)

[0696] After the avatar motion control process in step S424 or step S425 , the data receiving apparatus 50 determines in step S426 whether the process of receiving frame information from the data transmitting apparatus 30 is ended.

[0697] In a case where the processing has not ended, the processing returns to step S421, and the processing in step S421 and subsequent steps is repeated for the next item of frame information to be received.

[0698] In a case where it is determined in step S426 that the process of receiving the frame information from the data transmission device 30 has ended, the process ends.

[0699] As described above, the data sending device 30 that sends avatar control information including skeleton information and frame information can generate frame information including a posture / posture information storage block, which is for replacing the avatar's movement with a movement different from the user's movement (that is, a movement according to a predefined posture or posture), and send the frame information to the data receiving device 50.

[0700] Upon receiving frame information including a posture / posture information storage block from the data transmitting device 30, the data receiving device 50 causes the avatar to perform a movement corresponding to the designated posture or posture based on the posture or posture designation information (such as a roll, a punch, or a sitting position) stored in the posture / posture information storage block. It should be noted that control information for causing the avatar to perform various postures or postures is pre-stored in the storage unit of the data transmitting device 30.

[0701] The data receiving device 50 obtains motion control information corresponding to the posture and posture designation information (such as designation information of rolling, punching and sitting up) stored in the posture / posture information storage block received from the data sending device 30 from the storage unit, and causes the avatar to execute the motion control information.

[0702] By implementing such processing, it is possible to make the avatar perform actions that the user cannot actually perform.

[0703] [10. Other examples]

[0704] Other examples will be described next.

[0705] Each example will be described below in sequence.

[0706] (1) Example of using an image data storage block that stores image data captured by a camera (such as a user's facial expression)

[0707] (2) Example of using a voice data storage block that stores voice data (such as a user's voice) acquired through a microphone

[0708] (3) The following example: a data transmitting device acquires motion data of multiple users and transmits the motion data to a data receiving device, and the data receiving device assigns the motion of each user to an avatar enabled by the user and implements avatar motion control

[0709] (4) Example of generating and using a data block storing detection values of a sensor other than a motion sensor (such as a touch sensor)

[0710] (5) Example of generating and using a data block storing a lighting program

[0711] (10-(1) Example of using an image data storage block storing image data (such as a user's facial expression) captured by a camera)

[0712] First, an example using an image data storage block storing image data captured by a camera, such as a user's facial expression, will be described.

[0713] Will refer to Figure 40 A specific example of a usage example of an image data storage block that stores image data captured by a camera, such as a user's facial expression, is described.

[0714] Figure 40 A user 10 wearing a motion sensor, a data transmitting device 30 and a data receiving device 50 are shown.

[0715] The detection values of the motion sensor worn by the user 10 are transmitted to the data transmitting device 30 , which generates frame information storing skeletal transformation data reflecting the user's motion based on the sensor detection values and transmits the frame information to the data receiving device 50 .

[0716] In this example, the data transmitting device 30 further captures an image of the user 10 using a camera 31 mounted on the data transmitting device 30 , generates a data block storing the captured image of the user 10 , and adds the data block to the frame information.

[0717] For example, an image data storage block storing a facial expression image obtained by capturing a facial image of the user 10 is generated, and frame information including the generated image data storage block is generated and transmitted to the data receiving device 50 .

[0718] When an image data storage block is detected from the frame information received from the data transmitting device 30, the data receiving device 50 performs avatar display control to reflect the image data stored in the detected image data storage block in the avatar 51. For example, when image data representing the expression of the user 10 is detected in the image data storage block, expression control processing to change the expression of the avatar 51 according to the expression of the user 10 is performed.

[0719] Figure 41 A data block configuration example of the frame information (data block) 42 generated by the data transmission device (smartphone, etc.) 30 is shown.

[0720] Figure 41 The data block configuration of the frame information 42 shown in FIG is that the "image data storage block 74" is added to the previously referenced Figure 13 The data block configuration of the frame information 42 is described.

[0721] like Figure 41As shown in FIG. 7 , the “image data storage block 74 ” is a data block that stores, for example, a user's facial image representing the user's facial expression.

[0722] It should be noted that Figure 41 The position of the “image data storage block 74 ” in the frame information 42 shown in FIG. 1 is for example, and may be set to a position other than that shown in the figure.

[0723] The data sending device (smart phone, etc.) 30 generates an "image data storage block 74" that stores a facial image of a user, etc., adds the "image data storage block" as a constituent block of the frame information 42, and sends the "image data storage block" to the data receiving device (PC, etc.) 50.

[0724] When the “image data storage block 74” is detected from the frame information 42 received from the data sending device (smart phone, etc.) 30, the data receiving device (PC, etc.) 50 analyzes the storage data of the “image data storage block 74” and performs display control of the avatar 51 displayed on the display unit of the data receiving device (PC, etc.) 50 based on the analysis result.

[0725] As described above, for example, when image data representing the expression of the user 10 is detected in the “image data storage block 74 ”, expression control processing of changing the expression of the avatar 51 according to the expression of the user 10 is executed.

[0726] It should be noted that the "image data storage block 74" can store not only image data representing the expressions of the user 10, but also image data representing fine movements of the user 10 that are difficult to obtain using only the detection values of the motion sensor. For example, image data representing fine movements of the user 10's hands, specifically, the rock-paper-scissors gesture, etc., can be stored.

[0727] In the case where the data sending device (smartphone, etc.) 30 obtains image data representing the action of rock-paper-scissors from the "image data storage block 74" in the frame information 42 received from the data sending device 30, it is possible to implement processing for controlling the action of the hand of the avatar 51 according to the user's action.

[0728] (10-(2) Example of using an audio data storage block storing audio data (such as user's voice) acquired through a microphone)

[0729] Next, an example of using a voice data storage block that stores voice data acquired through a microphone (such as a user's voice) will be described.

[0730] Will refer to Figure 42A specific example of a usage example of an audio data storage block that stores audio data acquired through a microphone, such as a user's voice, is described.

[0731] Figure 42 A user 10 wearing a motion sensor, a data transmitting device 30 and a data receiving device 50 are shown.

[0732] The detection values of the motion sensor worn by the user 10 are transmitted to the data transmitting device 30 , which generates frame information storing skeletal transformation data reflecting the user's motion based on the sensor detection values and transmits the frame information to the data receiving device 50 .

[0733] In this example, the data transmitting device 30 further acquires voice data such as the voice of the user 10 through the microphone 32 attached to the data transmitting device 30, generates a data block storing the acquired voice data, and adds the data block to the frame information.

[0734] In the case where an audio data storage block is detected from the frame information received from the data transmitting device 30, the data receiving device 50 performs avatar control to reflect the audio data stored in the detected audio data storage block in the avatar 51. For example, in the case where voice data representing the voice of the user 10 is detected in the voice data storage block, voice output control is performed to output the voice of the user 10 or a processed sound generated based on the voice of the user 10 as the voice of the avatar 51.

[0735] Figure 43 A data block configuration example of the frame information (data block) 42 generated by the data transmission device (smartphone, etc.) 30 is shown.

[0736] Figure 43 The data block configuration of the frame information 42 shown in FIG is that the "audio data storage block 75" is added to the previously referenced Figure 13 The data block configuration of the frame information 42 is described.

[0737] like Figure 43 As shown in FIG. 1 , the “audio data storage block 75 ” stores audio data such as the user's voice, for example.

[0738] It should be noted that Figure 43 The position of the "audio data storage block 75" in the frame information 42 shown in the figure is for example, and may be set to a position other than that shown in the figure.

[0739] The data transmitting device (smartphone, etc.) 30 generates an “audio data storage block 75 ” storing the user’s voice, etc., adds the audio data storage block as a configuration block of the frame information 42 , and transmits the audio data storage block to the data receiving device (PC, etc.) 50 .

[0740] When the “audio data storage block 75” is detected from the frame information 42 received from the data sending device (smart phone, etc.) 30, the data receiving device (PC, etc.) 50 analyzes the storage data of the “audio data storage block 75” and performs audio output control of the avatar 51 displayed on the display unit of the data receiving device (PC, etc.) 50 based on the analysis result.

[0741] It should be noted that the “audio data storage block 75 ” can store not only audio data representing the voice of the user 10 , but also audio data representing voice generated by the movement of the user 10 , such as the clapping of the user 10 or ambient sounds.

[0742] In the case where the data sending device (smartphone, etc.) 30 obtains audio data representing the hand beats of the user 10 or the ambient sounds of the user 10 from the "audio data storage block 75" in the frame information 42 received from the data sending device 30, it is possible to implement control to output these audios in addition to the display processing of the avatar 51 on the display unit.

[0743] (10-(3) Example: A data transmitting device acquires motion data of a plurality of users and transmits the motion data to a data receiving device, and the data receiving device assigns the motion of each user to an avatar corresponding to the user and performs avatar motion control)

[0744] Next, an example will be described in which a data transmitting device acquires motion data of a plurality of users and transmits the motion data to a data receiving device, and the data receiving device assigns each user's motion to a user-based avatar and performs avatar motion control.

[0745] Will refer to Figure 44 A specific processing example of this example is described.

[0746] Figure 44 A plurality of users a ( 10 a ) to e ( 10 e ) wearing motion sensors, a data transmitting device a ( 30 a ) corresponding to the user a ( 10 a ), and a data receiving device 50 are shown.

[0747] It should be noted that user b (10b) to user e (10e) also have respective data transmission devices.

[0748] User a (10a) to user e (10e) each wear a motion sensor, the sensor detection value is sent to each user's data transmission device, and avatar control information (skeleton information and frame information) reflecting each user's movement is generated in each data transmission device.

[0749] The avatar control information generated by the data transmission devices of the four users b (10b) to e (10e) other than the user a (10a) is transmitted to the data transmission device 30a of the user a (10a).

[0750] The data transmitting device 30a of user a (10a) transmits the avatar control information of user a (10a) and the respective avatar control information of users b (10b) to e (10e) received from the data transmitting devices of users b (10b) to e (10e) to the data receiving device 50.

[0751] The data receiving device 50 receives avatar control information of each of the five users a (10a) to e (10e) from the data transmitting device a (30a), and reflects the movement of each user on each avatar a to e displayed on the display unit of the data receiving device 50.

[0752] That is, the following five processes are executed in parallel.

[0753] Motion control reflecting the movement of user a (10a) is performed on avatar a, motion control reflecting the movement of user b (10b) is performed on avatar b, motion control reflecting the movement of user c (10c) is performed on avatar c, motion control reflecting the movement of user d (10d) is performed on avatar d, and motion control reflecting the movement of user e (10e) is performed on avatar e.

[0754] In order to apply the motion of each user to each avatar in this way, the data transmission device a (30a) generates, for example, Figure 45 , and transmits the frame information to the data receiving device 50 .

[0755] Figure 45 The data block configuration of the frame information 42 shown in FIG is the following frame configuration: the skeletal transformation data array block corresponding to each user (ie, each of users a to e) is set to the previously referenced Figure 13 The frame data block of the frame information 42 is described as a storage data block.

[0756] In the skeleton transformation data array block corresponding to each user, a user ID (UID) corresponding to each user as an identifier of each user is stored, and further a skeleton transformation data block corresponding to each user is stored as a lower block.

[0757] In reference Figure 44 In the example described, the users are five users a to e, and the configuration example of the frame data block in this case has the following configuration example: Figure 46 The configuration shown in .

[0758] like Figure 46 As shown in , the lower data block stored in the frame data block includes the following five separate transformed data blocks corresponding to the user.

[0759] Transformation data block of user a (UID=U001)

[0760] Transformation data block of user b (UID=U002)

[0761] Transformation data block of user c (UID=U003)

[0762] Transformation data block of user d (UID=U004)

[0763] Transformation data block of user e (UID=U005)

[0764] It should be noted that the skeleton ID block and the transformation data block of each user are stored in the transformation data block corresponding to each user.

[0765] The data receiving device 50 receives data stored therein from the data sending device a (30a). Figure 46 Frame information 42 of the transformed data blocks (UID=U001 to U005) of the five users a (10a) to e (10e) shown in FIG.

[0766] The data receiving device 50 obtains the transformation data block (UID=U001 to U005) of each user a (10a) to e (10e) of the individual included in the frame information 42 received from the data sending device a (30a), analyzes the movement in units of users based on the stored data, and performs avatar control reflecting the movement of each user in the avatars a to e associated with each user.

[0767] Through this processing, it is possible to perform motion control reflecting the motions of multiple users in the avatar associated with each user.

[0768] (10-(4) Example of Generating and Using a Data Block Storing Detection Values of a Sensor Different from a Motion Sensor (For Example, a Touch Sensor))

[0769] Next, an example of generating and using a data block storing a detection value of a sensor other than a motion sensor, such as a touch sensor, will be described.

[0770] Will refer to Figure 47 A specific example of a usage example of a touch sensor analysis data storage block that stores detection data and analysis data of a touch sensor worn by a user is described.

[0771] Figure 47 A user 10 wearing a motion sensor, a data transmitting device 30 and a data receiving device 50 are shown.

[0772] The detection values of the motion sensor worn by the user 10 are transmitted to the data transmitting device 30 , which generates frame information storing skeletal transformation data reflecting the user's motion based on the sensor detection values and transmits the frame information to the data receiving device 50 .

[0773] In this example, the user 10 also wears a touch sensor 21 on the toe. The detection value of the touch sensor 21 is also transmitted to the data transmission device 30.

[0774] The data sending device 30 analyzes the timing, kicking strength, etc. of the user 10 kicking the ball 12 based on the detection value of the touch sensor 21 attached to the user 10, generates a data block (touch sensor analysis data storage block) to store the analysis data, and adds the data block to the frame information.

[0775] In the case where the data receiving device 50 detects a touch sensor analysis data storage block from the frame information received from the data sending device 30, the data receiving device 50 performs display control for controlling the movement of the avatar 51 or the ball displayed on the display unit of the data receiving device 50 based on the data stored in the detected touch sensor analysis data storage block (that is, the timing of the user 10 kicking the ball 12, the kicking strength, etc.).

[0776] Figure 48 A data block configuration example of the frame information (data block) 42 generated by the data transmission device (smartphone, etc.) 30 is shown.

[0777] Figure 48 The data block configuration of the frame information 42 shown in FIG. 4 is that the “touch sensor analysis data storage block 76” is added to the previously referenced Figure 13 The data block configuration of the frame information 42 is described.

[0778] like Figure 48 As shown in FIG. 7 , the “touch sensor analysis data storage block 76 ” is a data block that stores analysis data such as the user's contact timing and contact pressure with respect to a contact object (eg, a ball).

[0779] It should be noted that Figure 48 The position of the “touch sensor analysis data storage block 76 ” in the frame information 42 shown in FIG. 1 is for example only and may be set to a position other than that shown in the figure.

[0780] The data sending device (smart phone, etc.) 30 generates such a "touch sensor analysis data storage block 76", adds the "touch sensor analysis data storage block" as a configuration block of the frame information 42, and sends the "touch sensor analysis data storage block" to the data receiving device (PC, etc.) 50.

[0781] When the “touch sensor analysis data storage block 76” is detected from the frame information 42 received from the data sending device (smart phone, etc.) 30, the data receiving device (PC, etc.) 50 analyzes the data stored in the “touch sensor analysis data storage block 76” and performs display control of the user's contact target object, such as the avatar 51 or the ball displayed on the display unit of the data receiving device (PC, etc.) 50, based on the analysis result.

[0782] In addition, it should be noted that Figure 49 As shown in , the motion sensor can be attached to the user's contact object (such as a ball), and the data sending device 30 can generate a data block of analysis data storing motion information of an object such as a ball based on the sensor detection value, add the data block to the frame information, store the data block, and send the frame information to the data sending device 30.

[0783] With such a configuration, the data transmission device 30 can implement display control of moving an image of an object such as a ball according to the motion of an actual object operated by the user.

[0784] (10-(5) Example of generating and using a data block storing a lighting program)

[0785] Next, an example of generating and using a data block storing a lighting program will be described.

[0786] Will refer to Figure 50 A specific example of a usage example of a lighting control information storage block that stores lighting control information (lighting control program) in addition to the user's motion information is described.

[0787] Figure 50 A user 10 wearing a motion sensor, a data transmitting device 30 and a data receiving device 50 are shown.

[0788] It should be noted that the user 10 wearing the motion sensor is performing a live performance at a live venue.

[0789] The detection values of the motion sensor worn by the user 10 are transmitted to the data transmitting device 30 , which generates frame information storing skeletal transformation data reflecting the user's motion based on the sensor detection values and transmits the frame information to the data receiving device 50 .

[0790] In this example, the data sending device 30 also inputs lighting control information (lighting control program) from a lighting controller that controls the lighting equipment at the location where the user 10 is located (for example, a stage serving as a live performance venue), generates a data block that stores the input lighting control information (lighting control program), and adds the data block to the frame information.

[0791] For example, lighting control information (lighting control program) including the on / off timing, output light color, intensity, direction, etc. of the lighting equipment of the stage where the user 10 is located is input from the lighting controller, and frame information of a lighting control information storage block that stores the input lighting control information (lighting control program) is generated and sent to the data receiving device 50.

[0792] In the case where a lighting control information storage block is detected from the frame information received from the data sending device 30, the data receiving device 50 uses the lighting control information (lighting control program) stored in the detected lighting control information storage block to perform expression control processing for controlling the on / off timing and the color, intensity and direction of the output light relative to the lighting of the avatar 51 displayed on the display unit.

[0793] Figure 51 A data block configuration example of the frame information (data block) 42 generated by the data transmission device (smartphone, etc.) 30 is shown.

[0794] Figure 51 The data block configuration of the frame information 42 shown in FIG is that the "illumination control information storage block 77" is added to the previously referenced Figure 13 The data block configuration of the frame information 42 is described.

[0795] like Figure 52 As shown in FIG. 7 , the “illumination control information storage block 77 ” is a data block that stores illumination control information such as illumination on / off timing, color, intensity, and direction of output light.

[0796] It should be noted that Figure 52 The position of the “illumination control information storage block 77 ” in the frame information 42 shown in FIG. 1 is by way of example, and may be set to a position other than that shown in the figure.

[0797] The data transmitting device (smartphone, etc.) 30 generates such a “lighting control information storage block 77 ”, adds it as a configuration block of the frame information 42 , and transmits it to the data receiving device (PC, etc.) 50 .

[0798] When the “lighting control information storage block 77” is detected from the frame information 42 received from the data sending device (smart phone, etc.) 30, the data receiving device (PC, etc.) 50 analyzes the stored data of the “lighting control information storage block 77” and uses the lighting control information such as the on / off timing of the lighting and the color, intensity and direction of the output light obtained as a result of the analysis to perform display control of the lighting on the avatar 51 displayed on the display unit of the data receiving device (PC, etc.) 50.

[0799] [11. Application Examples of the Configuration and Processing of the Present Disclosure]

[0800] Next, application examples of the configuration and processing of the present disclosure will be described.

[0801] As application examples using the configuration and processing of the present disclosure described above, there are the following application examples, for example.

[0802] (a) Application example using an image analysis data block that stores analysis data of images captured by a camera

[0803] For example, security cameras are used to detect suspicious individuals breaking in.

[0804] The data sending device 30 analyzes the image captured by the security camera, generates an image analysis data block storing information that the posture of the person in the image is greatly shifted based on the analysis result, includes the generated image analysis data block in the frame information 42 constituting the avatar control information, and sends the frame information to the data receiving device 50.

[0805] It should be noted that in the present application example, skeleton transformation data based on the detection value of the motion sensor may not be included in the frame information 42 .

[0806] The data receiving device 50 analyzes the image information data blocks in the frame information received from the data sending device 30, determines the timing when the characters in the image make large movements, the timing when the characters make specific movements, etc., and can efficiently estimate the timing of occurrence of events such as theft or accidents and the details of the events at the timing based on the determination results.

[0807] In addition, the data sending device 30 analyzes images captured by a camera that captures images of sports such as football, for example, generates an image analysis data block that stores information such as the gross movements and characteristic actions of athletes in the image (such as the timing of executing a shooting action), includes the image analysis data block in the frame information 42, and sends the image analysis data block to the data receiving device 50.

[0808] The data receiving device 50 analyzes the stored data of the image analysis data block in the frame information received from the data transmitting device 30 and can efficiently select important actions of the players in the image, such as an image of a shot.

[0809] In addition, the data sending device 30 analyzes images captured by a camera provided at a place such as a nursing place, generates an image analysis data block that stores information such as the timing of a patient performing a characteristic action (e.g., a falling action) in the image, includes the image analysis data block in the frame information 42, and sends the image analysis data block to the data receiving device 50.

[0810] The data receiving device 50 analyzes the stored data of the image analysis data block in the frame information received from the data transmitting device 30, and can efficiently select the patient's action in the image, such as an image of a fall, and can quickly discover a dangerous state.

[0811] (b) Application Example Using an Object Motion Analysis Data Block That Stores Analysis Data Based on Detection Values of Motion Sensors Attached to Objects Other Than People

[0812] For example, in the case of implementing a battle performance, etc., a motion sensor is attached to a sword, the data sending device 30 generates an object motion analysis data block that stores analysis data of the sword's movement, includes the object motion analysis data block in the frame information 42, and sends the object motion analysis data block to the data receiving device 50.

[0813] The data receiving device 50 analyzes the stored data of the image analysis data block in the frame information received from the data transmitting device 30, and can efficiently select the sword action in the image, such as the image of the sword being swung downward, and can efficiently detect the image of interest.

[0814] (c) Application example using an image analysis data block storing analysis data of captured images of a camera provided in a virtual studio

[0815] For example, the data sending device 30 generates an image analysis data block storing information such as spatial information of a camera provided in a virtual studio or field of view (FoV) representing an image capture area, and includes the image analysis data block in the frame information 42 to send the frame information to the data receiving device 50.

[0816] The data receiving device 50 analyzes information received from the data transmitting device 30, such as the spatial information of the virtual studio and the camera in the field of view (FoV) representing the image capture area, and reflects the analysis results in the real-space image displayed on the display unit. By implementing such processing, it is possible to display an image in which the camera performances in the real space and the virtual space are interlocked.

[0817] (d) Application Examples Using the Acquired Avatar Control Information (Motion Data)

[0818] The avatar control information (motion data) configured by the aforementioned skeleton information and frame information may be recorded and held as multiple items of avatar control information (motion data) in units of users.

[0819] Therefore, for example, it is possible to implement processing in which Mr. B performs an action while viewing the movement of avatar a (=Mr. A) while performing reproduction of avatar a based on Mr. A's avatar control information (motion data).

[0820] For example, the avatar control information (motion data) of each actor in the play can be individually and sequentially acquired and recorded.

[0821] In addition, it is possible to implement the synthesis processing of multiple avatar control information (motion data), and it is also possible to implement the processing of editing avatar control information (motion data), such as additionally obtaining and then connecting additional avatar control information (motion data) to continue to specific recorded avatar control information (motion data).

[0822] As a specific example of use, the present invention can also be used for dance editing. In addition, the present disclosure can also be used for simulation processing of "what will happen if this action is performed at this timing" during a sports event.

[0823] In addition, in sports broadcasts, etc., it is also possible to implement processing of capturing images from various viewpoints and synthesizing and displaying avatar images reproduced using avatar control information (motion data) and image data, wherein the image data is distributed by an image distribution system using a camera system capable of reproducing the individual movements of each athlete.

[0824] [12. Configuration Example of Information Processing Device]

[0825] Next, a configuration example of an information processing device of the data transmission device 30 such as a smartphone or the data reception device 50 such as a PC used for the processing of the present disclosure will be described.

[0826] Figure 52 A configuration example of a data transmitting device 100 corresponding to the data transmitting device 30 such as a smartphone used in the process of the present disclosure and a configuration example of a data receiving device 200 corresponding to the configuration of a data receiving device 50 such as a PC are shown.

[0827] The data transmission device 100 shown on the left side includes, for example, a smartphone, and the data reception device 50 shown on the right side includes, for example, a PC. However, the data transmission device 100 is not limited to a smartphone and may be a tablet terminal, a PC, or VR glasses having functions such as data processing and communication.

[0828] Furthermore, the data receiving device 50 is not limited to a PC, and may be a tablet terminal, a smartphone, or VR glasses or the like having functions such as data processing and communication.

[0829] First, we will describe Figure 52 The left side shows the configuration of the data transmitting device 100.

[0830] The data transmission device 100 includes a sensor IF 101 , a motion analysis unit 102 , a camera 103 , a microphone 104 , another sensor IF 105 , a transmission information generation unit (skeleton information and frame information generation unit) 106 , and a communication unit 107 .

[0831] The sensor IF 101 is an IF corresponding to the sensor (motion sensor) 151, such as an IF attached to each body part of the user (for example, Figure 3 The IMU of the right hand, left hand, right arm, left arm, right foot, left foot, head, waist, etc. described above is used, and data transmission and reception are performed with these sensors 151. For example, the sensor IF is an IF that implements proximity communication such as Bluetooth (registered trademark) communication.

[0832] The motion analysis unit 102 performs processing for estimating the user's posture by inputting detection values of sensors attached to various body parts (head, right arm, left arm, waist, right foot, left foot, etc.) of the user input via the sensor IF 101 .

[0833] For example, according to the previous reference Figure 17 The described process is used to perform the motion analysis process of the user performed in the motion analysis unit 102. That is, the following processes are sequentially performed to analyze the motion of the user and generate conversion data as the motion information of the user.

[0834] (Processing 1) Sensor detection value input processing

[0835] (Processing 2) Noise Removal Processing

[0836] (Process 3) Motion Vector Group Generation Process

[0837] (Process 4) Motion Estimation Process

[0838] The motion analysis unit 102 sequentially performs each of these (processing 1) to (processing 4), for example, to generate “motion information (conversion data)” of the user.

[0839] The camera 103 captures, for example, images such as facial images that enable analysis of the user's expression.

[0840] The microphone 104 acquires, for example, the user's speech sound, the user's movement sound (clapping, etc.), or environmental sound.

[0841] Furthermore, the sensor IF 105 acquires various sensor detection values such as a touch sensor and a pressure sensor.

[0842] The motion analysis information analyzed by the motion analysis unit 102 and the information acquired by the camera 103 , the microphone 104 , and another sensor IF 105 are input to the transmission information generation unit 106 .

[0843] Based on the sensor detection values and information acquired by the camera 103, microphone 104 and another sensor IF 105, the transmission information generation unit (skeleton information and frame information generation unit) 106 generates avatar control information configured by skeleton information and frame information using the user's motion information generated by the motion analysis unit 102.

[0844] It should be noted that both the skeleton information and the frame information have a prescribed data format.

[0845] That is to say, as previously mentioned Figure 6 As described above, a data block includes the following three data areas.

[0846] (a) Data size storage area

[0847] (b) Data type storage area

[0848] (c) Actual data storage area

[0849] Furthermore, the skeleton information configured as a data block stores a plurality of data blocks therein.

[0850] That is, for example, there are a lot of previous references Figure 9 Data blocks described by etc.

[0851] Similarly, frame information configured as data blocks also stores a plurality of data blocks therein.

[0852] That is, for example, there are a lot of previous references Figure 13 Data blocks described by etc.

[0853] The frame information includes a plurality of data blocks in which data representing motion in units of body parts of the user are respectively stored.

[0854] The sending information generation unit (skeleton information and frame information generation unit) 106 generates a plurality of data blocks storing various different data respectively by using the motion analysis unit 102 to generate the user's motion information based on the sensor detection value and the information acquired by the camera 103, the microphone 104 and another sensor IF 105, and generates a skeleton information data block storing the generated plurality of data blocks and a frame information data block storing the generated plurality of data blocks.

[0855] The transmission information generating unit (skeleton information and frame information generating unit) 106 generates avatar control information configured by the generated skeleton information and frame information, and transmits the generated avatar control information to the data receiving apparatus 200 via the communication unit 107 .

[0856] The communication unit 104 receives avatar control information including skeleton information and frame information from the transmission information generating unit (skeleton information and frame information generating unit) 106 , and transmits these input data to the data receiving device 200 .

[0857] As previously referred Figure 18 As described, the communication unit 107 stores a data block corresponding to the skeleton information or the frame information in, for example, a UDP packet, and transmits the data block to the data receiving device 200. The UDP packet is a communication packet generated according to the User Datagram Protocol (UDP).

[0858] It should be noted that the communication unit 107 transmits the skeleton information data block storage packet storing the skeleton information to the data receiving device 200 every 3 seconds, for example.

[0859] Furthermore, a frame information data block storage packet storing frame information is transmitted to the data receiving device 200 every (1 / 50) seconds, for example.

[0860] Next, the configuration of the data receiving device 200 will be described.

[0861] As shown in the figure, the data receiving device 200 includes a communication unit 201 , a reception data analysis unit (a motion information analysis and the like execution unit) 202 , a display control unit 203 , and a display unit 204 .

[0862] The data receiving device 200 receives avatar control information including skeleton information and frame information from the user terminal 100 such as a smartphone via the communication unit 201 .

[0863] For example, a skeleton information data block storage group storing skeleton information is received every 3 seconds.

[0864] Furthermore, the frame information data block storage packet storing the frame information is received every (1 / 50) second, for example.

[0865] The reception data analysis unit (motion information analysis and the like execution unit) 202 performs analysis processing on the avatar control information including skeleton information and frame information received from the user terminal 100 such as a smartphone.

[0866] Specifically, for example, referring to Figure 9 The skeleton information including a large number of data blocks described in the above reference Figure 13The method obtains data blocks from the frame information including a large number of data blocks described in the embodiment of the present invention, and obtains and analyzes the stored data of each data block.

[0867] For example, the configuration of the avatar is analyzed through skeleton information analysis processing, and the avatar movement in units of frames is analyzed through analysis of frame information.

[0868] The analysis process of the frame information executed by the reception data analysis unit (motion information analysis and the like execution unit) 202 is, for example, the following process.

[0869] The received data analysis unit (motion information analysis etc. execution unit) 202 acquires a plurality of data blocks in which motion data in units of user's body parts on the data transmission device 100 side are respectively stored, and extracts motion data in units of user's body parts from the plurality of acquired data blocks.

[0870] These analysis results are input to the display control unit 203 .

[0871] The display control unit 203 performs motion control of the avatar displayed on the display unit 204 based on the data analyzed by the receiving data analysis unit (motion information analysis and other execution unit) 202, that is, the avatar configuration information obtained as a result of the analysis of the skeleton information and the avatar motion information in frames obtained through the analysis of the frame information.

[0872] That is, the display control unit 203 performs avatar motion control, wherein the motion data in units of the user's body parts on the data sending device 100 side extracted from the multiple data blocks using the receiving data analysis unit (motion information analysis, etc. execution unit) 202 is reflected as motion in units of the user's body parts as motion in units of the avatar's parts.

[0873] Under the control of the display control unit 203 , the avatar performing movements reflecting the movements of the user on the data transmission device 100 side is displayed on the display unit 204 .

[0874] [13. Hardware configuration example of information processing equipment]

[0875] Next, we will refer to Figure 39 A hardware configuration example of an information processing device constituting a data transmitting device and a data receiving device that implement the processing according to the aforementioned example is described.

[0876] Figure 53 The hardware shown in is an example of the hardware configuration of the information processing device of the present disclosure, for example, an information processing device constituting a data transmitting device and a data receiving device such as a PC and a smartphone.

[0877] Will describe Figure 53The hardware configuration shown in .

[0878] The central processing unit (CPU) 301 functions as a data processing unit that executes various processes according to a program stored in a read-only memory (ROM) 302 or a storage unit 308. For example, the CPU 301 executes processes according to the sequence described in the aforementioned example. The random access memory (RAM) 303 stores programs, data, and the like to be executed by the CPU 301. The CPU 301, ROM 302, and RAM 303 are connected to each other via a bus 304.

[0879] The CPU 301 is connected to an input / output interface 305 via a bus 304 , and an input unit 306 including various sensors, cameras, switches, microphones, etc. and an output unit 307 including a display, a speaker, etc. are connected to the input / output interface 305 .

[0880] The storage unit 308 connected to the input / output interface 305 includes, for example, a hard disk etc. that stores programs executed by the CPU 301 and various data. The communication unit 309 acts as a transmitter and receiver of data communication via a network such as the Internet or a local area network, and communicates with external devices.

[0881] The drive 310 connected to the input / output interface 305 drives a removable medium 311 such as a magnetic disk, an optical magneto-optical disk, or the like, or a semiconductor memory such as a memory card, and records or reads data.

[0882] [14. Summary of the configuration of the present disclosure]

[0883] The examples of the present disclosure have been described in detail above with reference to specific examples. However, it is apparent that those skilled in the art may modify or substitute the examples without departing from the spirit of the present disclosure. In other words, the present invention is disclosed by way of illustration and should not be construed in a limiting sense. To determine the spirit of the present disclosure, the claims should be considered.

[0884] It should be noted that the technology disclosed herein can have the following configurations:

[0885] (1) A data transmission device comprising:

[0886] a motion analysis unit that analyzes the user's motion based on sensor detection values of a motion sensor worn on the user's body;

[0887] a transmission information generating unit that receives the analysis result of the motion analyzing unit and generates transmission information storing data indicating the motion of the user; and

[0888] The communication unit sends the transmission information generated by the transmission information generation unit to the data receiving device, wherein

[0889] The transmission information generation unit generates, as the transmission information, frame information in which a plurality of data blocks each storing data representing a motion in units of a body part of a user are combined.

[0890] (2) The data transmission device according to (1), wherein each data block constituting the frame information includes the following data format, the data format including the following three data storage areas:

[0891] (a) Data size storage area

[0892] (b) Data type storage area

[0893] (c) Actual data storage area.

[0894] (3) The data transmission device according to (1) or (2), wherein the transmission information generation unit generates the frame information having a hierarchical configuration of the plurality of data blocks.

[0895] (4) The data transmission device according to any one of (1) to (3), wherein the transmission information generation unit generates the frame information as one data block having the same data format as a data block stored inside the frame information.

[0896] (5) The data transmitting device according to any one of (1) to (4), wherein the motion analysis unit analyzes the motion of the user per predetermined time based on a sensor detection value per predetermined time of a motion sensor worn on the body of the user, and

[0897] The transmission information generation unit sequentially generates, as the transmission information, frame information storing data indicating the user's motion at each predetermined time, based on the user's motion at each predetermined time input from the motion analysis unit.

[0898] (6) The data transmitting device according to (5), wherein the data receiving device includes a device for performing avatar motion control for causing an avatar displayed on a display unit to reflect a motion of the user using stored data of a data block constituting frame information received from the data transmitting device, and

[0899] The transmission information generating unit sequentially generates frame information at intervals equal to image frame intervals of the avatar display image in the data receiving device.

[0900] (7) The data transmitting device according to any one of (1) to (6), wherein the transmission information generation unit generates a transmission source data block storing transmission source data representing the data transmitting device as a data block constituting the frame information.

[0901] (8) The data transmitting device according to (7), wherein the transmission source data block has a configuration in which an IP address block is stored, the IP address block storing the IP address of the data transmitting device.

[0902] (9) The data transmitting device according to any one of (1) to (8), wherein the data receiving device includes a device for performing avatar motion control for causing an avatar displayed on a display unit to reflect a motion of the user using stored data of a data block constituting frame information received from the data transmitting device, and

[0903] The transmission information generating unit generates skeleton information representing the three-dimensional structure of the avatar as the transmission information, and

[0904] The skeleton information has a configuration in which a plurality of data blocks having the same data format as that of the data blocks constituting the frame information are combined.

[0905] (10) The data transmission device according to (9), wherein the transmission information generation unit generates the skeleton information as one data block having the same data format as a data block stored inside the skeleton information.

[0906] (11) The data transmitting device according to any one of (1) to (10), wherein the data receiving device includes a device for performing avatar motion control for causing an avatar displayed on a display unit to reflect a motion of the user using stored data of a data block constituting frame information received from the data transmitting device, and

[0907] The transmission information generation unit generates a change of dress instruction data block that instructs change of dress of the avatar as a data block constituting the frame information, and transmits the change of dress instruction data block to the data receiving device via the communication unit.

[0908] (12) The data transmitting device according to any one of (1) to (11), wherein the data receiving device includes a device for performing avatar motion control for causing an avatar displayed on a display unit to reflect a motion of the user using stored data of a data block constituting frame information received from the data transmitting device, and

[0909] The transmission information generating unit generates a priority information data block storing output priority information of a plurality of avatars displayed in the data receiving device as a data block constituting the frame information, and transmits the priority information data block to the data receiving device via the communication unit.

[0910] (13) The data transmitting device according to any one of (1) to (12), wherein the data receiving device includes a device for performing avatar motion control for causing an avatar displayed on a display unit to reflect a motion of the user using stored data of a data block constituting frame information received from the data transmitting device, and

[0911] The sending information generation unit generates a posture / posture information storage block that stores instruction information for performing a prescribed posture or posture in an avatar displayed in a data receiving device instead of data representing the user's movement stored in the frame information as a data block constituting the frame information, and sends the posture / posture information storage block to the data receiving device via the communication unit.

[0912] (14) The data transmitting device according to any one of (1) to (13), wherein the data receiving device includes a device that performs avatar motion control that causes a plurality of avatars displayed on a display unit to reflect motions of a plurality of users, and

[0913] The transmission information generation unit generates frame information including data blocks in units of users storing motion information of individual users, and transmits the frame information to the data receiving device via the communication unit.

[0914] (15) The data transmitting device according to any one of (1) to (14), wherein the transmission information generating unit generates a data block storing detection information of a sensor other than the motion sensor as a data block constituting the frame information, and transmits the data block to the data receiving device via the communication unit, and

[0915] The sensor other than the motion sensor is at least one of a camera, a microphone, or a touch sensor.

[0916] (16) A data receiving device comprising:

[0917] a received data analyzing unit for analyzing received data from the data sending device; and

[0918] a display control unit that receives the analysis result of the received data analysis unit and performs motion control of the avatar displayed on the display unit, wherein

[0919] The reception data analyzing unit acquires a plurality of data blocks in which motion data in units of body parts of the user on the data transmitting device side are respectively stored from frame information included in the reception data, and extracts the motion data in units of body parts of the user from the plurality of acquired data blocks, and

[0920] The display control unit performs avatar motion control by reflecting motion in units of user's body parts as motion in units of avatar parts using motion data in units of user's body parts extracted from the plurality of data blocks by the received data analysis unit.

[0921] (17) A data processing method executed in a data transmitting device, the data processing method comprising:

[0922] a motion analysis step, wherein the motion analysis unit analyzes the user's motion based on sensor detection values of a motion sensor worn on the user's body;

[0923] a transmission information generating step, wherein a transmission information generating unit receives the analysis result of the motion analyzing step and generates transmission information storing data indicating the motion of the user; and

[0924] The communication step, wherein the communication unit transmits the transmission information generated in the transmission information generating step to the data receiving device, wherein

[0925] The steps for generating the sending information include:

[0926] Frame information in which a plurality of data blocks each storing data representing motion in units of a body part of the user are combined is generated as the transmission information.

[0927] (18) A data processing method executed in a data receiving device, the data processing method comprising:

[0928] a received data analyzing step, wherein the received data analyzing unit analyzes the received data from the data transmitting device; and

[0929] A display control step, wherein a display control unit receives the analysis result in the received data analysis step and performs motion control of an avatar displayed on a display unit, wherein

[0930] The steps for receiving data analysis include:

[0931] a step of acquiring a plurality of data blocks in which motion data in units of body parts of the user on the data transmitting device side are respectively stored from frame information included in the received data; and

[0932] extracting motion data in units of user's body parts from the plurality of acquired data blocks, and

[0933] The display control steps include:

[0934] Avatar motion control is performed, wherein motion in units of user's body parts is reflected as motion in units of avatar parts using the motion data in units of user's body parts extracted from the plurality of data blocks in the received data analysis step.

[0935] (19) A program for causing a data transmission device to execute data processing, the data processing comprising:

[0936] a motion analysis step, causing the motion analysis unit to analyze the user's motion based on sensor detection values of a motion sensor worn on the user's body;

[0937] a transmission information generating step of causing the transmission information generating unit to receive the analysis result of the motion analyzing step and generate transmission information storing data indicating the motion of the user; and

[0938] The communication step causes the communication unit to transmit the transmission information generated in the transmission information generation step to the data receiving device, wherein

[0939] In the Send Information Generation step, perform the following steps:

[0940] Frame information in which a plurality of data blocks each storing data representing motion in units of a body part of the user are combined is generated as the transmission information.

[0941] (20) A program for causing a data receiving device to execute data processing, the data processing comprising:

[0942] a received data analyzing step, causing the received data analyzing unit to analyze the received data from the data transmitting device; and

[0943] a display control step of causing a display control unit to receive the analysis result in the received data analysis step and perform motion control of an avatar displayed on a display unit, wherein

[0944] In the Receive Data Analysis step, perform the following steps:

[0945] acquiring a plurality of data blocks in which motion data in units of body parts of the user on the data transmitting device side are respectively stored from frame information included in the received data, and

[0946] Extracting motion data in units of user's body parts from the plurality of acquired data blocks, and

[0947] In the Display Control step:

[0948] Avatar motion control is performed, wherein motion in units of user's body parts is reflected as motion in units of avatar parts using the motion data in units of user's body parts extracted from the plurality of data blocks in the received data analysis step.

[0949] It should be noted that the series of processing described in this manual can be performed by hardware, software or a combination of the two. In the case of performing the processing by software, the program in which the processing sequence is recorded can be installed and implemented in a memory in a computer incorporated in special hardware, or the program can be installed and implemented in a general-purpose computer capable of performing various processing. For example, the program can be recorded in a recording medium in advance. In addition to being installed in a computer from a recording medium, the program can be received through a network such as a local area network (LAN) or the Internet and installed to a recording medium such as an internal hard disk.

[0950] Furthermore, the various processes described herein can be implemented not only in a time-series manner as described, but can also be implemented in parallel or individually, depending on the processing capabilities of the devices implementing the processes or as needed. Furthermore, the system described herein is a logical assembly of multiple devices and is not limited to a system in which the various configured devices are housed in the same housing.

[0951] Industrial Applicability

[0952] As described above, according to the configuration of the examples of the present disclosure, an apparatus and method for causing an avatar displayed on a display unit to reflect the user's motion substantially in real time are realized.

[0953] Specifically, for example, the system includes a motion analysis unit that analyzes the user's motion based on detection values from sensors worn by the user, and a transmission information generation unit that receives the motion analysis results and generates transmission information storing data representing the user's motion. The transmission information generation unit generates frame information that combines multiple data blocks that each store data representing the motion of each body part of the user. The data receiving device obtains the data representing the motion of each body part from the data blocks in the received frame information and performs avatar motion control to cause an avatar displayed on a display unit to reflect the user's motion.

[0954] According to the present configuration, an apparatus and method for a process of causing an avatar displayed on a display unit to reflect the user's motion substantially in real time are realized.

[0955] Reference Signs List

[0956] 10 - User

[0957] 20 - Motion Sensor

[0958] 21 - Touch sensor

[0959] 30——Data sending device

[0960] 31 - Camera

[0961] 40——Avatar control information

[0962] 41——Skeleton information

[0963] 42——Frame information

[0964] 50——Data receiving device

[0965] 51 - Incarnation

[0966] 71——Crossdressing instruction block

[0967] 72 - Priority information data block

[0968] 73——Posture / posture information storage block

[0969] 74——Image data storage block

[0970] 75 - Audio data storage block

[0971] 76——Touch sensor analysis data storage block

[0972] 77——Lighting control information storage block

[0973] 100——Data sending device

[0974] 101——Sensor IF

[0975] 102——Motion Analysis Unit

[0976] 103 - Camera

[0977] 104——Microphone

[0978] 105——Another sensor IF

[0979] 106——Sending information generation unit (skeleton information and frame information generation unit)

[0980] 107——Communication Unit

[0981] 200——Data receiving device

[0982] 201——Communication unit

[0983] 202——Received data analysis unit (executing unit for motion information analysis, etc.)

[0984] 203——Display control unit

[0985] 204——Display unit

[0986] 301——CPU

[0987] 302——ROM

[0988] 303——RAM

[0989] 304——Bus

[0990] 305——Input / Output Interface

[0991] 306——Input Unit

[0992] 307——Output unit

[0993] 308 - Storage Unit

[0994] 309 - Communication Unit

[0995] 310 - Driver

[0996] 311 - Removable Media

Claims

1. A data transmitting device, comprising: a motion analysis unit that analyzes the user's motion based on sensor detection values of a motion sensor worn on the user's body; a transmission information generating unit that receives the analysis result of the motion analyzing unit and generates transmission information storing data representing the motion of the user; as well as The communication unit sends the transmission information generated by the transmission information generation unit to the data receiving device, wherein The transmission information generation unit generates, as the transmission information, frame information in which a plurality of data blocks each storing data representing a motion in units of a body part of a user are combined.

2. The data transmitting device according to claim 1 , wherein each data block constituting the frame information comprises the following data format, the data format comprising the following three data storage areas: (a) Data size storage area (b) Data type storage area (c) Actual data storage area. 3 . The data transmission device according to claim 1 , wherein the transmission information generation unit generates the frame information having a hierarchical configuration of the plurality of data blocks.

4. The data transmission device according to claim 1, wherein the transmission information generation unit generates the frame information as one data block having the same data format as a data block stored inside the frame information.

5. The data transmitting device according to claim 1 , wherein the motion analyzing unit analyzes the motion of the user per predetermined time based on a sensor detection value per predetermined time of a motion sensor worn on the user's body, and The transmission information generation unit sequentially generates, as the transmission information, frame information storing data indicating the user's motion at each predetermined time, based on the user's motion at each predetermined time input from the motion analysis unit.

6. The data transmitting device according to claim 5 , wherein the data receiving device includes a device for performing avatar motion control for causing an avatar displayed on a display unit to reflect a motion of the user using stored data of a data block constituting the frame information received from the data transmitting device, and The transmission information generating unit sequentially generates frame information at intervals equal to image frame intervals of the avatar display image in the data receiving device.

7. The data transmitting device according to claim 1, wherein the transmission information generating unit generates a transmission source data block storing transmission source data indicating the data transmitting device as a data block constituting the frame information.

8. The data transmitting apparatus according to claim 7, wherein the transmission source data block has a configuration in which an IP address block is stored, the IP address block storing an IP address of the data transmitting apparatus.

9. The data transmitting device according to claim 1 , wherein the data receiving device includes a device for performing avatar motion control for causing an avatar displayed on a display unit to reflect a motion of the user using stored data of a data block constituting the frame information received from the data transmitting device, and The transmission information generating unit generates skeleton information representing the three-dimensional structure of the avatar as the transmission information, and The skeleton information has a configuration in which a plurality of data blocks having the same data format as that of the data blocks constituting the frame information are combined.

10. The data transmission device according to claim 9, wherein the transmission information generation unit generates the skeleton information as one data block having the same data format as a data block stored inside the skeleton information.

11. The data transmitting device according to claim 1 , wherein the data receiving device includes a device for performing avatar motion control for causing an avatar displayed on a display unit to reflect a motion of the user using stored data of a data block constituting the frame information received from the data transmitting device, and The transmission information generation unit generates a change of dress instruction data block that instructs change of dress of the avatar as a data block constituting the frame information, and transmits the change of dress instruction data block to the data receiving device via the communication unit.

12. The data transmitting device according to claim 1 , wherein the data receiving device includes a device for performing avatar motion control for causing an avatar displayed on a display unit to reflect a motion of the user using stored data of a data block constituting the frame information received from the data transmitting device, and The transmission information generating unit generates a priority information data block storing output priority information of a plurality of avatars displayed in the data receiving device as a data block constituting the frame information, and transmits the priority information data block to the data receiving device via the communication unit.

13. The data transmitting device according to claim 1 , wherein the data receiving device includes a device for performing avatar motion control for causing an avatar displayed on a display unit to reflect a motion of the user using stored data of a data block constituting the frame information received from the data transmitting device, and The sending information generation unit generates a posture / posture information storage block that stores instruction information for performing a prescribed posture or posture in an avatar displayed in a data receiving device instead of data representing the user's movement stored in the frame information as a data block constituting the frame information, and sends the posture / posture information storage block to the data receiving device via the communication unit.

14. The data transmitting device according to claim 1 , wherein the data receiving device comprises a device that performs avatar motion control that causes a plurality of avatars displayed on a display unit to reflect motions of a plurality of users, and The transmission information generation unit generates frame information including data blocks in units of users storing motion information of individual users, and transmits the frame information to the data receiving device via the communication unit.

15. The data transmitting device according to claim 1 , wherein the transmission information generating unit generates a data block storing detection information of a sensor other than the motion sensor as a data block constituting the frame information, and transmits the data block to the data receiving device via the communication unit, and The sensor other than the motion sensor is at least one of a camera, a microphone, or a touch sensor.

16. A data receiving device, comprising: a received data analysis unit for analyzing received data from the data sending device; as well as a display control unit that receives the analysis result of the received data analysis unit and performs motion control of the avatar displayed on the display unit, wherein The reception data analyzing unit acquires a plurality of data blocks in which motion data in units of body parts of the user on the data transmitting device side are respectively stored from frame information included in the reception data, and extracts the motion data in units of body parts of the user from the plurality of acquired data blocks, and The display control unit performs avatar motion control by reflecting motion in units of user's body parts as motion in units of avatar parts using motion data in units of user's body parts extracted from the plurality of data blocks by the received data analysis unit.

17. A data processing method executed in a data sending device, the data processing method comprising: a motion analysis step, wherein the motion analysis unit analyzes the user's motion based on sensor detection values of a motion sensor worn on the user's body; a transmission information generating step, wherein a transmission information generating unit receives the analysis result of the motion analyzing step and generates transmission information storing data indicating the motion of the user; as well as The communication step, wherein the communication unit transmits the transmission information generated in the transmission information generating step to the data receiving device, wherein The steps for generating the sending information include: Frame information in which a plurality of data blocks each storing data representing motion in units of a body part of the user are combined is generated as the transmission information.

18. A data processing method executed in a data receiving device, the data processing method comprising: a received data analyzing step, wherein the received data analyzing unit analyzes the received data from the data sending device; as well as A display control step, wherein a display control unit receives the analysis result in the received data analysis step and performs motion control of an avatar displayed on a display unit, wherein The steps for receiving data analysis include: a step of acquiring a plurality of data blocks in which motion data in units of body parts of the user on the data transmitting device side are respectively stored from frame information included in the received data; and extracting motion data in units of user's body parts from the plurality of acquired data blocks, and The display control steps include: Avatar motion control is performed, wherein motion in units of user's body parts is reflected as motion in units of avatar parts using the motion data in units of user's body parts extracted from the plurality of data blocks in the received data analysis step.

19. A program for causing a data transmitting device to perform data processing, the data processing comprising: a motion analysis step, causing the motion analysis unit to analyze the user's motion based on sensor detection values of a motion sensor worn on the user's body; a transmission information generating step of causing the transmission information generating unit to receive the analysis result of the motion analyzing step and generate transmission information storing data indicating the motion of the user; as well as The communication step causes the communication unit to transmit the transmission information generated in the transmission information generation step to the data receiving device, wherein In the Send Information Generation step, perform the following steps: Frame information in which a plurality of data blocks each storing data representing motion in units of a body part of the user are combined is generated as the transmission information.

20. A program for causing a data receiving device to perform data processing, the data processing comprising: a received data analyzing step, causing the received data analyzing unit to analyze the received data from the data sending device; as well as a display control step of causing a display control unit to receive the analysis result in the received data analysis step and perform motion control of an avatar displayed on a display unit, wherein In the Receive Data Analysis step, perform the following steps: acquiring a plurality of data blocks in which motion data in units of body parts of the user on the data transmitting device side are respectively stored from frame information included in the received data, and Extracting motion data in units of user's body parts from the plurality of acquired data blocks, and In the Display Control step: Avatar motion control is performed, wherein motion in units of user's body parts is reflected as motion in units of avatar parts using the motion data in units of user's body parts extracted from the plurality of data blocks in the received data analysis step.

Images