Information processing device, information processing method, and program

Abstract

An information processing apparatus includes circuitry configured to obtain interpolated motion data for interpolating between temporally and spatially independent first motion data and second motion data, and to generate a motion signal based on the first motion data, the second motion data, and the obtained interpolated motion data. An indicator related to the speed of motion of each motion data is calculated.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of Japanese Priority Patent Application JP 2022-162381 filed on October 7, 2022, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to an information processing device, an information processing method, and a program. Background Art

[0004] In recent years, animation production and distribution using motion capture for obtaining motion information indicating user motion have been actively performed. For example, motion data imitating user motion is generated using the motion information obtained through motion capture, and an avatar video based on the motion data is transmitted.

[0005] Against this backdrop, the amount of motion data is increasing year by year, and technologies for reusing previously generated motion data have been developed. For example, Patent Document 1 discloses a technology for synthesizing the movements of multiple pieces of motion data using an avatar or the like in a virtual space and reproducing the resulting movement in real time.

[0006] [Citation List]

[0007] [Patent Document]

[0008] [PTL 1]

[0009] WO 2019 / 203190 A Summary of the Invention

[0010] [Technical Issues]

[0011] When such a plurality of motion data are connected, interpolation editing for connecting the plurality of motion data more naturally may be required. In such interpolation editing, parameters such as boundary conditions can be specifically specified, but it is difficult for the user to set all parameters.

[0012] Therefore, the present disclosure proposes a new and improved information processing device, information processing method, and program that are capable of more naturally connecting multiple pieces of motion data.

[0013] [Solution to the problem]

[0014] According to the present disclosure, an information processing device is provided, which includes a circuit configured to obtain interpolated motion data for interpolating between first motion data and second motion data that are independent in time and space, and calculate an indicator related to the motion speed of each motion data based on the first motion data, the second motion data and the obtained interpolated motion data.

[0015] In addition, according to the present disclosure, there is provided an information processing method executed by a computer, the method comprising: obtaining interpolated motion data for interpolating between first motion data and second motion data that are independent in time and space, and calculating an indicator related to the motion speed of each motion data based on the first motion data, the second motion data and the obtained interpolated motion data.

[0016] In addition, according to the present disclosure, a non-transitory computer-readable storage medium has a program embodied thereon, which, when executed by a computer, causes the computer to perform a method comprising: obtaining interpolated motion data for interpolating between first motion data and second motion data that are independent in time and space; and calculating an indicator related to the motion speed of each motion data based on the first motion data, the second motion data and the obtained interpolated motion data. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is an explanatory diagram for describing an information processing system according to an embodiment of the present disclosure.

[0018] Figure 2 1 is an explanatory diagram for describing an example of a functional configuration of the server 10 according to an embodiment of the present disclosure.

[0019] Figure 3 2 is an explanatory diagram for describing an example of a functional configuration of the PC 20 according to an embodiment of the present disclosure.

[0020] Figure 4A is an explanatory diagram for describing an example of motion data connection processing.

[0021] Figure 4B is an explanatory diagram for describing an example of motion data connection processing.

[0022] Figure 5 It is an explanatory diagram for describing an overview of interpolation editing for interpolating between a plurality of pieces of motion data.

[0023] Figure 6A is an explanatory diagram for describing motion blending as an example of interpolation editing.

[0024] Figure 6B is an explanatory diagram for describing generation of interpolation motion data C as another example of interpolation editing.

[0025] Figure 7 is an explanatory diagram for describing a specific example of a process in which the generation unit 241 generates interpolated motion data according to an embodiment of the present disclosure.

[0026] Figure 8 is an explanatory diagram for describing a specific example of interpolation parameters.

[0027] Figure 9 is an explanatory diagram for describing an example of a GUI according to an embodiment of the present disclosure.

[0028] Figure 10 is a flowchart for describing an example of an operation process of the PC 20 according to an embodiment of the present disclosure.

[0029] Figure 11 2 is a block diagram showing the hardware configuration of the PC 20 according to the embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] Those skilled in the art will understand that various variations, combinations, sub-combinations, and modifications may occur depending on design requirements and other factors, as long as they are within the scope of the appended claims or their equivalents. Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Note that in this specification and the drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant explanations are omitted.

[0031] Furthermore, “Description of the embodiment” will be described according to the following order of items.

[0032] 1. Overview of Information Processing Systems

[0033] 2. Functional Configuration Example

[0034] 2.1. Functional Configuration Example of Server 10

[0035] 2.2. Example of PC 20 functional configuration

[0036] 3. Details

[0037] 3.1. Connection of motion data

[0038] 3.2. Interpolation Editing of Motion Data

[0039] 3.3. Generation of interpolated motion data

[0040] 3.4. Rating

[0041] 3.5. Graphical User Interface (GUI)

[0042] 4. Operation Processing Example

[0043] 5. Examples of Actions and Effects

[0044] 6. Deformation Examples

[0045] 7. Hardware Configuration

[0046] 8. Supplement

[0047] <<1. Overview of Information Processing System>>

[0048] As motion data, in order to visualize the motion information of a moving body such as a person or an animal, for example, skeleton data represented by a skeleton structure indicating the body structure is used. The skeleton data includes information such as the position and posture of the parts. More specifically, the skeleton data includes various types of information, such as the global position of the root joint and the relative posture of each joint. Note that, for example, the parts in the skeleton structure correspond to the ends of the body, joint parts, etc. In addition, the skeleton data may also include bones that are line segments connecting the parts. For example, the bones in the skeleton structure may correspond to human bones, but the positions and number of the bones do not necessarily need to match the actual human bones. In addition, the motion data may also include various types of information, such as skeletal information indicating the length of each bone and the connection relationship with other bones, and time series data of the ground contact information of the two feet.

[0049] The position and posture of each component in the skeleton data can be acquired using various motion capture technologies. For example, there are camera-based technologies that attach markers to each body part and acquire the marker's position using an external camera, etc., and sensor-based technologies that attach motion sensors to body parts and acquire positional information based on time-series data acquired by the motion sensors.

[0050] Skeleton data also has various uses. For example, motion data, which is time-series data of skeleton data, is used to improve dance or sports performance, or is used in applications such as virtual reality (VR) and augmented reality (AR). Furthermore, motion data is used to generate avatar videos that mimic the user's movements and to distribute these avatar videos.

[0051] The information processing system according to the embodiment of the present disclosure enables multiple pieces of motion data to be connected more naturally. Figure 1 Describe the overview of the information processing system.

[0052] Figure 1 : is an explanatory diagram for describing an information processing system according to an embodiment of the present disclosure. Figure 1 As shown in , the information processing system according to the embodiment of the present disclosure includes a network 1 , a server 10 , and a personal computer (PC) 20 .

[0053] (Network 1)

[0054] The network 1 according to an embodiment of the present disclosure is a wired or wireless transmission path for information transmitted from a device connected to the network 1. For example, the network 1 may include a public network such as the Internet, a telephone network, or a satellite communication network, various local area networks (LANs) including Ethernet (registered trademark), a wide area network (WAN), and the like. In addition, the network 1 may include a dedicated line network such as an Internet Protocol Virtual Private Network (IP-VPN). The server 10 and the PC 20 are connected via the network 1.

[0055] (Server 10)

[0056] The server 10 according to an embodiment of the present disclosure is a device for storing multiple pieces of motion data. In addition, the server 10 learns the relationship between the two pieces of motion data and motion-related parameters (hereinafter, it can be expressed as interpolation parameters) and the interpolated motion data, and obtains a generation model for generating the interpolated motion data.

[0057] (PC 20)

[0058] The PC 20 according to the embodiment of the present disclosure is an example of an information processing device that acquires interpolated motion data for interpolation between two pieces of motion data and calculates an index related to the motion speed of each piece of motion data based on the two pieces of motion data and the interpolated motion data.

[0059] The above has described the overview of the information processing system according to the embodiment of the present disclosure. Figure 2 and Figure 3 , details of the functional configurations of the server 10 and the PC 20 are described in sequence.

[0060] <<2. Functional Configuration Example>>

[0061] <2.1. Functional Configuration Example of Server 10>>

[0062] Figure 2 1 is an explanatory diagram for describing an example of the functional configuration of the server 10 according to an embodiment of the present disclosure. Figure 2 As shown, the server 10 according to an embodiment of the present disclosure includes a storage unit 110 , a learning unit 120 , and a communication unit 130 .

[0063] (Storage Unit 110)

[0064] The storage unit 110 according to an embodiment of the present disclosure stores software and various data. For example, the storage unit 110 stores a plurality of pieces of exercise data. In addition, the storage unit 110 may store a generation model obtained by the learning unit 120.

[0065] (Study Unit 120)

[0066] The learning unit 120 according to the embodiment of the present disclosure learns the relationship between two pieces of temporally and spatially independent motion data and the interpolation parameter and interpolated motion data, and generates a generation model. Details of the learning will be described later.

[0067] (Communication Unit 130)

[0068] The communication unit 130 according to an embodiment of the present disclosure transmits various types of information to the PC 20 via the network 1 and receives various types of information from the PC 20. For example, the communication unit 130 receives request information for requesting motion data from the PC 20. In addition, the communication unit 130 transmits motion data corresponding to the request information received from the PC 20 to the PC 20.

[0069] Furthermore, the communication unit 130 may transmit the generated model stored in the storage unit 110 to the PC 20 .

[0070] An example of the functional configuration of the server 10 according to the embodiment of the present disclosure has been described above. Figure 3 An example of the functional configuration of the PC 20 according to an embodiment of the present disclosure is described.

[0071] <2.2. Example of PC 20 Functional Configuration>>

[0072] Figure 3 1 is an explanatory diagram for explaining an example of the functional configuration of the PC 20 according to an embodiment of the present disclosure. Figure 3 As shown, the PC 20 according to the embodiment of the present disclosure includes a communication unit 210 , an operation display unit 220 , a storage unit 230 , and a control unit 240 .

[0073] (Communication Unit 210)

[0074] The communication unit 210 according to an embodiment of the present disclosure transmits various types of information to the server 10 via the network 1 and receives various types of information from the server 10. For example, the communication unit 210 transmits request information for requesting motion data to the server 10. Then, the communication unit 210 receives the motion data transmitted according to the request information from the server 10.

[0075] In addition, the communication unit 210 may receive a generation model for generating interpolated motion data from the server 10 .

[0076] (Operation display unit 220)

[0077] The operation display unit 220 according to an embodiment of the present disclosure has a function as a display unit that displays the motion data received from the server 10 and various display information (e.g., connected motion data, etc.) generated by the generation unit 241 described later. In addition, the operation display unit 220 has a function as an input unit for the user to select motion data. Note that when specific motion data is selected by the user, the communication unit 210 sends a request information for requesting the motion data to the server 10.

[0078] For example, the function as the display unit is performed by a cathode ray tube (CRT) display device, a liquid crystal display (LCD) device, or an organic light emitting diode (OLED) device.

[0079] Furthermore, a function as an input unit is performed by, for example, a touch panel, a keyboard, or a mouse.

[0080] It should be noted that Figure 1 In the embodiment, the PC 20 has a configuration in which the functions of the display unit and the operation unit are integrated, but may have a configuration in which the functions of the display unit and the input unit are separated.

[0081] (Storage Unit 230)

[0082] The storage unit 230 according to an embodiment of the present disclosure stores software and various data. The storage unit 230 stores the generation model received from the server 10. In addition, the storage unit 230 may store interpolated motion data generated by the generation unit 241 described later. In addition, the storage unit 230 may store connected motion data obtained by connecting the interpolated motion data generated by the generation unit 241, the first motion data, and the second motion data.

[0083] (Control Unit 240)

[0084] The control unit 240 according to an embodiment of the present disclosure controls the overall operation of the PC 20. Figure 3 As shown, the PC 20 includes a generating unit 241 and a calculating unit 243 .

[0085] The generation unit 241 according to an embodiment of the present disclosure is an example of an acquisition unit, and generates interpolation motion data for interpolation between first motion data and second motion data that are independent in time and space. Details on the generation of interpolation motion data will be described later.

[0086] For example, the generation unit 241 may input the first motion data, the second motion data, and at least one or more interpolation parameters about the motion into the generation model to generate interpolated motion data. In addition, the generation unit 241 may generate interpolated motion data multiple times by correcting the interpolation parameters input to the generation model.

[0087] Also, the generation unit 241 may generate connected motion data obtained by connecting the first motion data, the interpolated motion data, and the second motion data.

[0088] The calculation unit 243 according to an embodiment of the present disclosure calculates an index related to the motion speed of each motion data based on the first and second motion data received by the communication unit 210 and the interpolated motion data generated by the generation unit 241. Details on the calculation of the index related to the motion speed will be described later.

[0089] The details of the PC 20 according to the embodiment of the present disclosure have been described above. Figure 9 Details of the information processing system according to the embodiment of the present disclosure are described in sequence.

[0090] 3. Details

[0091] Recently, an application has been developed in which motion data obtained through motion capture or manually is registered in a database, and the motion data can be searched or edited within the database. For example, when editing motion data, a concatenation editing method for connecting multiple pieces of motion data may be required. First, a specific example of concatenating multiple pieces of motion data will be described with reference to FIG4 .

[0092] <3.1. Connection of Motion Data>

[0093] 4 is an explanatory diagram for explaining an example of motion data connection processing. Some segments included in specific motion data (hereinafter, represented as first motion data A) can be replaced with other motion data (hereinafter, represented as second motion data B) and corrected.

[0094] For example, a user using an application searches for specific first motion data A, and selects segment A2 from among a plurality of segments A1 to A3 included in the first motion data A as a correction segment.

[0095] Then, the user selects the second motion data B in segment A2 as a correction target. Therefore, segment A2 of the first motion data A can be corrected by being replaced with the second motion data B to be corrected. Note that the second motion data B here can be any motion data selected by the user on the application, or can be motion data automatically selected by the application.

[0096] For example, as shown in FIG4 , in a case where there are two pieces of second motion data B as candidates to be corrected for segment A2 of first motion data A, the user selects one of the two pieces of second motion data B. For example, in a case where the user selects the left image of the second motion data B shown in FIG4 , segment A2 of the first motion data A is replaced with the left image of the motion data B shown in FIG4 and is corrected.

[0097] However, if segment A2 of the first motion data A is simply replaced with the second motion data B, motion is abruptly switched at a boundary portion between the first motion data A and the second motion data B. Therefore, motion included in the motion data may be unnatural.

[0098] Note that the boundary portion here includes a portion switching from the end position of segment A1 of the first motion data A to the start position of the second motion data B. In addition, the boundary portion includes a portion switching from the end position of the second motion data B to the start position of segment A3 of the first motion data A.

[0099] Therefore, in such connection editing of a plurality of pieces of motion data, it is desirable to perform interpolation editing for interpolating between the first motion data A and the second motion data B (ie, a boundary portion).

[0100] A specific example related to the connection editing of a plurality of motion data has been described above. Next, a specific example of the interpolation editing for interpolating between a plurality of motion data will be described.

[0101] <3.2. Interpolation Editing of Motion Data>

[0102] Figure 5 1 is an explanatory diagram for describing an overview of interpolation editing for interpolating between a plurality of pieces of motion data. For example, it is assumed that first motion data A includes a walking motion and second motion data B includes a kicking motion.

[0103] Here, as Figure 5 As shown, when the motion data is suddenly switched from the first motion data A to the second motion data B, the motion is instantly switched from walking motion to kicking motion, so that the motion of the motion data may be unnatural. For example, in one frame, because the joint position of each part contained in the first motion data A is converted to the joint position of each part contained in the second motion data B, the position of a certain part may be instantly moved (distorted). Therefore, interpolation editing exists as a method to suppress unnatural motion that may occur in the connection of multiple motion data. Here, reference will be made to Figure 6A and Figure 6B Describes a specific example of interpolation editing.

[0104] Figure 6A 1 is an explanatory diagram for explaining motion blending as an example of interpolation editing. For example, motion blending is interpolation editing that linearly and geometrically combines first motion data A and second motion data B.

[0105] In interpolation editing by motion blending, interpolation may be performed such that, in the interpolation section BQ, the motion included in the first motion data A gradually switches to the motion included in the second motion data B. Note that the motion data in which the motion is gradually switched in the interpolation section BQ is an example of interpolated motion data C.

[0106] Motion blending can be used for interpolation editing, which is suitable for situations where real-time performance is required, even if some unnatural movement is still present. On the other hand, interpolation editing through motion blending may not be suitable for situations such as animation editing, where more natural movement may be required and there are many parameters related to interpolation.

[0107] Figure 6B This is an explanatory diagram for explaining the generation of interpolated motion data C, another example of interpolation editing. For example, the generation of interpolated motion data C includes generating newly generated motion data using machine learning techniques such as deep learning. Here, the newly generated motion data is an example of interpolated motion data C.

[0108] In the generation of the interpolated motion data C, although the number of parameters related to interpolation is greater than the number of parameters of motion mixing, the interpolated motion data C capable of interpolating between the first motion data A and the second motion data B with more natural motion can be generated.

[0109] However, because there are many parameters related to interpolation, it is difficult for the user to manually set all of them. Furthermore, the interpolated motion data C, which is naturally interpolated from the motion data, is not necessarily generated according to the parameters set by the user. Furthermore, when the parameters are automatically set, the desired movement for the user depends on the user's preferences, making it difficult to uniquely determine the movement when setting the parameters.

[0110] Next, details of a process in which the generation unit 241 generates the interpolated motion data C according to an embodiment of the present disclosure will be described.

[0111] <3.3. Generation of interpolated motion data>

[0112] Figure 71 is an explanatory diagram for describing a specific example of a process in which the generation unit 241 generates interpolated motion data according to an embodiment of the present disclosure. First, as a preliminary preparation, the learning unit 120 included in the server 10 generates a generation model for generating the interpolated motion data C. For example, the learning unit 120 generates the generation model through machine learning using two pieces of motion data that are temporally and spatially independent of each other, parameters related to interpolation (hereinafter referred to as interpolation parameters), and a set of interpolated motion data as teacher data.

[0113] More specifically, if Figure 7 As shown, the learning unit 120 divides the motion data into three pieces with arbitrary lengths, and sets the motion data as first motion data A, second motion data B, and third motion data L. The lengths of the first motion data A, the second motion data B, and the third motion data L can be changed according to their respective usage scenarios, and can also be changed during the learning of the generation model.

[0114] Next, the learning unit 120 inputs the first motion data A, the second motion data B, and the interpolation parameters into the generative model. At this time, the third motion data L is considered to be missing (i.e., the third motion data L is not input into the generative model). As a result, the learning unit 120 generates a temporary generative model.

[0115] Next, the learning unit 120 calculates a loss between the interpolated motion data C generated from the first motion data A and the second motion data B and the third motion data L (ie, the corrected motion) using the generated generation model.

[0116] Then, the learning unit 120 can correct the parameters of the interpolated motion data (e.g., joint positions and joint postures, etc.) to reduce loss, repeatedly learn and update the generation model. Then, the communication unit 130 included in the server 10 can send the generation model generated by the learning unit 120 to the PC 20.

[0117] Note that the learning unit 120 may generate a generation model based on arbitrary motion data (including multiple pieces of motion data for various motions), or may generate a generation model for each specific motion data such as walking or running.

[0118] Then, the generation unit 241 included in the PC 20 according to the embodiment of the present disclosure may generate the interpolated motion data C using the generation model G obtained from the server 10. More specifically, the generation unit 241 may generate the interpolated motion data x using the following formula (1): C .

[0119] [Mathematical formula 1]

[0120] x c =G(xA ,x B ; c0, c1, c2…c n ) (1)

[0122] It should be noted that in formula (1), x C is the interpolated motion data, x A is the first motion data, x B is the second motion data, c0 to c n is the interpolation parameter.

[0123] The interpolation parameters are various parameters related to the motion of the motion data, such as boundary conditions or constraint conditions, and may be partially set by a user or may be automatically set based on the first motion data A and the second motion data B.

[0124] Here, we will refer to Figure 8 Describes specific examples of interpolation parameters.

[0125] (Interpolation parameters)

[0126] Figure 8 is an explanatory diagram for describing a specific example of interpolation parameters. Figure 8 The figure shows a case where the first motion data A and the second motion data B are independent in time and space. Here, spatial independence indicates a state where the first motion data A and the second motion data B are separated in space, and temporal independence indicates a state where the time from the start time to the end time of the first motion data A does not overlap with the time from the start time to the end time of the second motion data B.

[0127] For example, the position of the motion data in the three-dimensional space is represented by a position vector P. For example, the first motion data A at time t A The position of P A =(x A ,y A , z A ) is represented. In addition, the second motion data B at time t B The position of P B =(x B ,y B , z B )express.

[0128] Note that, for example, the position of the motion data in three-dimensional space may be the position of the character's root joint (waist joint), the position of other joints (e.g., toe joints), or a position calculated based on the positions of multiple joints.

[0129] In addition, the posture of the motion data in the three-dimensional space is represented by the total joint posture S corresponding to the relative postures of all joints. For example, the first motion data A at time tA The position of S A (t A ) is represented. In addition, the second motion data B at time t B The posture of S B (t B )express.

[0130] There may be various interpolation parameters. In this specification, eight parameters (A) to (H) are introduced, but other parameters may be included. In addition, some of the parameters (A) to (H) may be manually specified by the user. In addition, all of the parameters (A) to (H) may not be used to generate interpolated motion data. That is, some of the parameters (A) to (H) may be input into the generation model.

[0131] (A) Spatial location

[0132] The interpolation parameter may include a parameter related to the spatial position of the motion data. For example, the parameter related to the spatial position is the distance between the first motion data A and the second motion data B in the three-dimensional space. In other words, the parameter related to the spatial position is the difference between the spatial positions of the first motion data A and the second motion data B, and is represented by ||p A -p B || indicates.

[0133] Here, the position p of the first motion data A Or the position p of the second motion data B It can be manually set by the user, or can be automatically set based on the first motion data A and the second motion data B.

[0134] For example, by estimating the trajectory of the part or joint of the interpolated motion data according to the velocity of each root joint of the first motion data A and the second motion data B, the difference between the spatial positions can be automatically set based on the estimation result.

[0135] (B) Interpolation time

[0136] The interpolation parameter may include a parameter related to the interpolation time of the interpolated motion data. The parameter related to the interpolation time is the time difference between the first motion data A and the second motion data B on the time line, and is represented by t B -t A express.

[0137] In addition, the end time of the first motion data A or the start time of the second motion data B may be manually set by the user, or may be automatically set.

[0138] For example, the interpolation time may be automatically set based on the difference between the velocity or spatial position of the root joint of the first motion data A and the second motion data B.

[0139] (C) Posture of the first motion data A (interpolation start posture, boundary conditions)

[0140] The interpolation parameters may include parameters related to the posture of the first motion data A. The parameters related to the posture of the first motion data A are used to calculate the interpolation parameters at the interpolation start time t A After that, the motion data A is trimmed and the interpolation start time t is adjusted. A The parameters of the posture, and by S A (t A By adjusting the parameters related to the posture of the first motion data A, the posture of the interpolated motion data C at the start of interpolation can be made more natural.

[0141] In addition, the parameters regarding the posture of the first motion data A can be automatically set as the interpolation start posture based on the most stable posture estimated based on the velocity of the root joint of the first motion data A.

[0142] (D) Posture of the second motion data B (interpolation end posture, boundary conditions)

[0143] The interpolation parameters may include parameters related to the posture of the second motion data B. The parameters related to the posture of the second motion data B are used to calculate the interpolation parameters at the interpolation end time t B Before trimming the motion data B, adjust the interpolation end time t B The parameters of the posture, and by S B (t B By adjusting the parameters related to the posture of the second motion data B, the posture of the interpolated motion data C at the end of interpolation can be made more natural.

[0144] Furthermore, the parameters regarding the posture of the second motion data B may be automatically set as the interpolation end posture based on the most stable posture estimated based on the velocity of the root joint of the second motion data B.

[0145] (E) Contact conditions

[0146] The interpolation parameters may include parameters related to contact conditions. The parameters related to contact conditions may be parameters related to grounding, such as the time t from the start of interpolation. A To the end time of interpolation t B How many steps are needed?

[0147] For example, parameters related to contact conditions can be specified as the interpolation start time t A To the end time of interpolation t B For example, when parameters related to the contact condition of "five steps" are set throughout the interpolation time, a contact condition for generating a contact condition from the interpolation start time t is generated. ATo the end time of interpolation t B Interpolated motion data C of five steps.

[0148] Furthermore, in the case where the state in which the foot is in contact with the ground is "1" and the state in which the foot is not in contact with the ground (ie, the foot is away from the ground) is "0", it is possible to A With t B Parameters related to the contact condition, such as c(t) = {0, 1}, are sequentially specified for each time t between the two. For example, when parameters related to the contact condition "c(t1) = {0, 1}" are set, interpolated motion data C is generated at time t1 in which the toes of the left foot are not in contact with the ground and the toes of the right foot are in contact with the ground.

[0149] In addition, a more appropriate number of steps can be estimated according to the difference in spatial position or interpolation time between the first motion data A and the second motion data B, and parameters regarding the contact condition can be automatically set based on the estimation result.

[0150] (F) Position constraints

[0151] The interpolation parameters may include parameters related to position constraints. The parameters related to position constraints are parameters related to the fixation of the intermediate postures of the interpolated motion data C. The parameters related to position constraints are parameters used to A With t B The parameters of the entire joint posture of the interpolated motion data C are fixed at the specific time t between them, and are represented by S C (t) indicates.

[0152] For example, when the user expects A With t B In the case of interpolated motion data for a specific time t between when the knee is placed on the ground, the user can set the parameters of all joint postures for placing the knee on the ground at time t as parameters related to the position constraint.

[0153] (G) Semantic conditions

[0154] The interpolation parameters may include parameters related to semantic conditions. The parameters related to semantic conditions include parameters related to the pattern of the interpolated motion data C.

[0155] The style here is defined by features of motion data, labels expressed as styles, and the like, and the learning unit 120 can provide parameters regarding semantic conditions by assigning labels to motion data and causing a generation model to learn.

[0156] Note that specific examples of the style include "dynamic," "dance-like," "jumping," and "male (female)," etc. In addition, parameters related to the semantic conditions may be given to each application, and the generation unit 241 may generate the interpolated motion data C without providing such parameters related to the semantic conditions.

[0157] (H) Random numbers or noise

[0158] Interpolation parameters may include parameters related to random numbers or noise. For example, a generative model for generating interpolated motion data C may input random numbers as a probabilistic generation model. By using parameters related to random numbers or noise, different interpolated motion data C can be generated even if the other interpolation parameters described above are fixed.

[0159] The generation unit 241 may generate various variations of interpolated motion data C by inputting a random number as a seed, or may generate the interpolated motion data C without giving a random number (ie, the random number is set to a fixed value).

[0160] Specific examples of the interpolation parameters have been described above. As described above, the generation unit 241 generates interpolated motion data C based on the first motion data A, the second motion data B, and the interpolation parameters.

[0161] In addition, the generation unit 241 may generate connected motion data obtained by connecting the first motion data A, the interpolated motion data C, and the second motion data. However, depending on the setting value of the interpolation parameter, there is a possibility that the pieces of motion data included in the generated connected motion data are not necessarily naturally (smoothly) connected.

[0162] Therefore, the information processing system according to the embodiment of the present disclosure has a mechanism capable of more naturally connecting the first motion data A, the interpolated motion data C, and the second motion data B. More specifically, the calculation unit 243 calculates an index related to the motion speed of each motion data based on the first motion data A, the second motion data B, and the interpolated motion data C generated by the generation unit 241.

[0163] Next, a specific example of the index calculated by the calculation unit 243 and a specific example related to a process of presenting candidates for interpolated motion data to the user based on the index will be described.

[0164] <3.4. Rating>

[0165] First, the generation unit 241 generates the first motion data x by using the above formula (1). A , second motion data x B and interpolation parameters c0 to c n , generate interpolated motion data x CThen, the calculation unit 243 may calculate a total index s including at least one index S by using the following formula (2).

[0166] [Mathematical formula 2]

[0167] s=S(x A ,x B , x c ) (2)

[0168] For example, the calculation unit 243 may calculate the total index s including the five indexes S by using the following formula (3).

[0169] [Mathematical formula 3]

[0170] s=w p S 位置 +w v S 速度 +w j S 急动度 +w f S 脚 +w m S 元 (3)

[0171] In formula (3), w p , w v , w j , w f and w m is a weighting factor used for each metric and can be set automatically or specified for each use case.

[0172] In addition, the total index s in formula (3) is defined by a combination of physical indicators and heuristic indicators. However, the indicators S included in the total index s are not limited to the physical indicators and heuristic indicators described below. In addition, the indicators S in the total index s do not necessarily include all the indicators S described below. For example, expression (3) may not include w m S 元 First, the four physical indices in formula (3) will be described.

[0173] (Physical indicators)

[0174] For example, the physical indicators include indicators related to the motion speed of the first motion data A, the interpolated motion data C, and the second motion data B (hereinafter, these motion data may be represented as each motion data). For example, the motion speed here includes the speed of the root joint, the speed of other joints (such as the speed of the foot), or the speed obtained by combining the speeds of multiple joints. In addition, the indication related to the motion speed may also include indicators related to the speed, acceleration, or position of the motion.

[0175] For example, the physical indicator may include a continuous position indicator S indicating the degree of continuity of each piece of motion data. 位置 Continuous position indicator S 位置 It may be an indicator based on a difference between a maximum value of the velocity of the interpolated motion data C and an average velocity of the first motion data A and the second motion data B in a plurality of frames before and after the interpolated motion data C.

[0176] In the segment for converting from the first motion data A to the interpolated motion data C and the segment for converting from the interpolated motion data C to the second motion data B, in particular, if the joint position of each motion data moves instantaneously, unnatural behavior will occur. As an index for reducing such unnatural behavior, the continuous position index S is defined 指标 , and as the value obtained by the following formula (4) becomes smaller, the continuous position index S 指标 The larger it is calculated.

[0177] [Formula 4]

[0178] Max(||v c ||)-||v AB ||→0 (4)

[0179] According to formula (4), as the maximum velocity Max(||v C ||) and the average speed of the first motion data A and the second motion data B in multiple frames before and after the interpolated motion data C || v AB The smaller the difference between ||, the greater the continuous position index S 指标 That is, the continuous position index S 位置 The calculated large value indicates a state in which the joint positions of the respective pieces of motion data are continuously moving.

[0180] In addition, the physical index may include a speed index S indicating the smoothness of the speed of each motion data. 速度 Speed ​​indicator S 速度 It may be an indicator based on the difference between the average speed of the interpolated motion data C and the average speeds of the first motion data A and the second motion data B.

[0181] If the average speed of the first motion data A and the second motion data B is greatly different from the average speed of the interpolated motion data C, the interpolated motion data C may move suddenly faster or slower than the first motion data A or the second motion data B, resulting in unnatural behavior. Speed ​​index S 速度 is defined as an index for reducing such unnatural behavior, and as the value obtained by the following formula (5) is smaller, the speed index S 速度 The larger it is calculated.

[0182] [Formula 5]

[0183] ||Average(v C )||-||v AB ||→0(5)

[0184] According to formula (5), when the average speed of the interpolated motion data C||average value (v C )|| and the average speed of the first motion data A and the second motion data B||v AB When the difference between || decreases, the speed index S 速度 In other words, the speed index S 速度 The fact that it is calculated to be large indicates a state in which each piece of motion data moves more smoothly.

[0185] Furthermore, the physical index may include an acceleration index S indicating the extent to which the jerkiness (jerkiness: rate of change of acceleration) of the interpolated motion data C is small. 急动度 . Acceleration index S 急动度 It can be an indicator based on the differential component of the acceleration of the interpolated motion data C.

[0186] In the case where the interpolated motion data C generated by the generation unit 241 is suddenly accelerated or decelerated, unnatural behavior may occur. 急动度 is defined as an index for reducing such unnatural behavior, and as the value obtained by the following formula (6) becomes smaller, the acceleration index S 急动度 The larger it is calculated.

[0187] [Formula 6]

[0188]

[0189] According to formula (6), when the jerk as the differential value of the interpolated motion data C decreases, the acceleration index S 急动度 In other words, the acceleration index S 急动度 Calculated as large indicates a state in which the jerkiness of each motion data is small (ie, a state in which the vehicle is not suddenly accelerated or decelerated).

[0190] In addition, the physical index may include a foot index S indicating whether the foot of the interpolated motion data C does not slide when grounded. 脚 Foot index S 脚 It is an index related to the speed of the toe of the foot when the toe of the interpolated motion data C is in contact with the ground.

[0191] When the foot of the interpolated motion data C is in contact with the ground, if the magnitude of the foot velocity is 0 or greater, the foot of the interpolated motion data C slides relative to the ground. In many cases, the sliding of the foot may be an unnatural behavior. As an index for reducing such unnatural behavior, a foot index S is defined. 脚 , and the smaller the value obtained by the following formula (7), the smaller the foot index S 脚 The larger it is calculated.

[0192] [Formula 7]

[0193] ∫ c v c,脚 dt→0 (7)

[0194] According to formula (7), when the magnitude of the foot speed of the interpolated motion data C decreases, the foot index S 脚 In other words, the foot index S 脚 The fact that is calculated to be large indicates a state in which the foot of the interpolated motion data C does not slide relative to the ground.

[0195] A specific example of the physical index has been described above. Next, an example of the heuristic index included in Formula (3) will be described.

[0196] (Heuristic Indicator)

[0197] Heuristic indicators can include different meta-indicators S for each use case 元 Meta-Indicator S 元 It may be a usage-dependent indicator based on the difference between the properties of the interpolated motion data and the properties required for the interpolated motion data.

[0198] When the properties of the interpolated motion data C generated by the generation unit 241 are significantly different from the user's sensitivity or usage situation, the user may feel unnatural. For example, in the case where the first motion data A and the second motion data B are motions performed by a woman, and the interpolated motion data C generated by the generation unit 241 is motions performed by a larger man, the connected motion data obtained by connecting the motion data may have unnatural behavior. As an indicator for reducing such unnatural behavior according to the usage situation, a meta-indicator S is defined. 元 , and when the function for determining the property of the interpolated motion data C is F and the desired property is m, as the value obtained by the following formula (8) is smaller, the meta-index S 元 The larger it is calculated.

[0199] [Formula 8]

[0200] F(M C )-s→0 (8)

[0202] According to formula (8), as the attribute m obtained according to the usage situation is compared with the attribute F (M) of the interpolated motion data C generated by the generation unit 241, C ) is smaller, the meta-index S 元 In other words, the meta-index S 元 The fact that it is calculated to be large indicates that the interpolated motion data C generated by the generation unit 241 corresponds to a state of the user's sensitivity or usage.

[0203] After the calculation unit 243 calculates the total index s as described above, the generation unit 241 changes the value of the interpolation parameter and generates the interpolated motion data x again. C , and the calculation unit 243 again calculates the interpolated motion data x according to the generated C Calculate the total index s.

[0204] Such a series of processing from correction of the interpolation parameters to calculation of the overall index s may be repeatedly performed a plurality of times.

[0205] That is, the generation unit 241 corrects the interpolation parameters and generates the interpolation motion data x again. C , and the calculation unit 243 calculates the interpolated motion data x again generated by the generation unit 241 C A series of processes for calculating the total index s may be repeatedly performed multiple times.

[0206] Note that the above method of calculating the physical index is not limited to each of the above formulas, and a more rigorous calculation method can be used. In addition, as a method of calculating the heuristic index, an index suitable for other use cases can be used.

[0207] In addition, in the calculation formulas of the above-mentioned physical indicators and heuristic indicators, an example has been described in which the indicator S increases as the value obtained by each calculation formula decreases, but formula transformation can be performed so that the indicator S increases as the value obtained by each calculation formula increases.

[0208] The above has described a specific example of the index calculated by the calculation unit 243. According to the above series of processes, a plurality of combinations of pieces of interpolated motion data C and scores (total index or index) calculated from the pieces of motion data C are prepared.

[0209] Here, the generation unit 241 may add the interpolated motion data C whose score (total index or index) is calculated by the calculation unit 243 to the candidate list.

[0210] Then, the generation unit 241 may sort the candidate list of the interpolation motion data C in order of scores (more specifically, in descending order of scores), and the operation display unit 220 may display the sorted candidates of the interpolation motion data.

[0211] Therefore, the user can confirm the candidates of the interpolated motion data C sorted in descending order of the scores, and the convenience of the user in viewing the interpolated motion data C can be improved.

[0212] Next, we will refer to Figure 9 An example of a graphical user interface (GUI) according to an embodiment of the present disclosure is described.

[0213] <3.5. Graphical User Interface (GUI)>

[0214] Figure 9 1 is an explanatory diagram for explaining an example of a GUI according to an embodiment of the present disclosure. First, the user selects first motion data A using the operation display unit 220 and inserts the first motion data A into an arbitrary segment M1 on the timeline.

[0215] Subsequently, the user selects the second motion data B using the operation display unit 220 and inserts the second motion data B into another arbitrary segment M2 on the timeline. Note that designation of arbitrary segments M1 and M2 can be performed by operations such as dragging and dropping, or by inputting time.

[0216] Here, the generation unit 241 sets a segment from the end position of the first motion data A (ie, the end position of the arbitrary segment M1 ) to the start position of the second motion data B (ie, the start position of another arbitrary segment M2 ) as the interpolation segment M3 .

[0217] Furthermore, the user can manually set some of the above-mentioned interpolation parameters using the operation display unit 220 .

[0218] Then, when the user performs an operation related to the generation of interpolated motion data (for example, pressing an execution button (not shown) etc.), the generation unit 241 generates interpolated motion data, and the calculation unit 243 calculates the score of the interpolated motion data generated by the generation unit 241.

[0219] And then, the generation unit 241 sorts the interpolation motion data C in order of scores, and the operation display unit 220 displays the candidate list of the interpolation motion data sorted in order of scores, as shown in FIG. Figure 9 shown.

[0220] exist Figure 9 The candidate list shown in FIG is for the case where the first interpolation motion data C1, the second interpolation motion data C2, the third interpolation motion data C3, and the fourth interpolation motion data C4 are calculated to have higher scores in this order. In addition, in the case where the upper limit of the interpolation motion data to be included in the candidate list is set to 4, as shown in FIG. Figure 9The four pieces of interpolated motion data C1 to C4 shown are displayed on the operation display unit 220. However, for example, in a case where the upper limit of the interpolated motion data included in the candidate list is set to 5 or more, the user can confirm the interpolated motion data C5 whose score is calculated to be smaller by scrolling down the candidate list.

[0221] Then, when the user selects a piece of interpolated motion data C from the candidate list, the generation unit 241 generates connected motion data obtained by connecting the first motion data A, the interpolated motion data C selected by the user, and the second motion data B.

[0222] Then, the operation display unit 220 displays the connected motion data generated by the generation unit 241 .

[0223] Although examples of GUIs according to embodiments of the present disclosure have been described above, the GUIs are not limited to the above examples. For example, the operation display unit 220 may display connected motion data obtained by simply connecting the first motion data A and the second motion data B, connected motion data interpolated by motion blending, and the like.

[0224] The details of the information processing system according to the embodiment of the present disclosure have been described above. Figure 10 , describing an example of an operation process of the PC 20 according to an embodiment of the present disclosure.

[0225] <<4. Operation Processing Example>>

[0226] Figure 10 101 is a flowchart for explaining an example of an operation process of the PC 20 according to an embodiment of the present disclosure. First, the user selects first motion data A using the operation display unit 220 (S101).

[0227] Subsequently, the user selects second motion data B using the operation display unit 220 ( S105 ).

[0228] Next, the generation unit 241 automatically sets at least one or more interpolation parameters ( S109 ). Here, the user may manually set some of the above interpolation parameters using the operation display unit 220 .

[0229] Then, the generation unit 241 generates interpolated motion data C based on the first motion data A and the second motion data B selected by the user and the interpolation parameters set by the generation unit 241 (or the user) ( S113 ).

[0230] Next, the calculation unit 243 calculates a score (total index or index) based on the interpolated motion data C generated by the generation unit 241 and the first motion data A and the second motion data B selected by the user ( S117 ).

[0231] Subsequently, the generation unit 241 adds the interpolated motion data C whose score is calculated by the calculation unit 243 to the candidate list (S121). Note that the generation unit 241 may add the interpolated motion data C whose score is calculated by the calculation unit 243 to a predetermined value or more to the candidate list, and may not add the interpolated motion data C whose score is less than the predetermined value to the candidate list.

[0232] Next, the generation unit 241 sorts the candidate list of the interpolation motion data C in order of score (more specifically, in descending order of score) ( S125 ).

[0233] Then, the generation unit 241 determines whether the number of interpolated motion data C added to the candidate list has reached the upper limit of the candidate list (S129). In the case that the upper limit of the candidate list has been reached (S129: Yes), the process proceeds to S133. In the case that the upper limit of the candidate list has not been reached (S129: No), the process returns to S109 again to reset (change) the interpolation parameters, and the processes of S109 to S129 are repeatedly performed until the upper limit of the candidate list is reached in S129. Note that the upper limit of the candidate list here corresponds to a predetermined number of times set in advance and can be specified by the user or can be set automatically.

[0234] In a case where the number of pieces of interpolated motion data C reaches the upper limit of the candidate list ( S129 : Yes), the operation display unit 220 displays the candidate list of interpolated motion data C sorted in order of score ( S133 ).

[0235] Subsequently, the user selects a piece of interpolation motion data C included in the candidate list on the operation display unit 220 ( S137 ).

[0236] Then, the generation unit 241 generates connected motion data obtained by interpolating the interpolation motion data C selected by the user between the first motion data A and the second motion data B ( S141 ), and the PC 20 according to the embodiment of the present disclosure ends the processing.

[0237] <<5. Examples of Actions and Effects>>

[0238] According to the above-mentioned present disclosure, various effects and effects can be obtained. For example, the generation unit 241 according to the embodiment of the present disclosure generates interpolated motion data C for interpolating between first motion data A and second motion data B that are independent in time and space, and the calculation unit 243 calculates an indicator related to the motion speed of each motion data based on the first motion data A, the second motion data B and the interpolated motion data C generated by the generation unit 241. This allows the user to determine the degree of more natural behavior of the assumed connected motion data when the first motion data A, the second motion data B and the interpolated motion data C are connected. Therefore, the generation unit 241 is able to generate motion data in which multiple motion data are more naturally connected.

[0239] Furthermore, the generation unit 241 changes the interpolation parameters to generate interpolated motion data C multiple times, and the calculation unit 243 calculates a score using each piece of interpolated motion data C generated by the generation unit 241. Furthermore, the operation display unit 220 displays a candidate list of interpolated motion data C sorted in order of score. Therefore, the user can easily select the interpolated motion data C that the user prefers. Thus, the user can confirm the motion data to be connected according to the user's wishes.

[0240] 6. Transformation

[0241] (Motion capture interpolation technology)

[0242] The interpolation process for generating interpolated motion data according to an embodiment of the present disclosure can also be applied to interpolation technology in motion capture. For example, when motion is recorded by a motion capture system, some motion may be lost due to occlusion or the like.

[0243] In this case, the generation unit 241 can set the motion before and after the lost segment as the first motion data A and the second motion data B respectively, generate interpolated motion data C that smoothly interpolates the first motion data A and the second motion data B, and interpolate the motion in the lost segment.

[0244] In this case, when setting the interpolation parameters, among the multiple interpolation parameters, the interpolation parameters regarding the spatial position, interpolation time, posture of the first motion data A, and posture of the second motion data B are fixed. PC 20 corrects other interpolation parameters (i.e., contact conditions, position constraint conditions, semantic conditions, random numbers, etc.) to repeatedly generate interpolated motion data and perform score calculation. Then, after displaying the candidates for interpolated motion data, the user specifies a piece of interpolated motion data so that interpolation of the lost segment can be performed. Note that the user does not necessarily need to specify a piece of interpolated motion data, and the generation unit 241 can automatically specify the calculated interpolated motion data C with the highest score or the interpolated motion data with a score exceeding the threshold, and interpolate the lost segment.

[0245] It is also assumed that the motion of a part of the body (eg, a hand, etc.) is lost. In this case, the generation unit 241 may set a lost segment for the part of the body (referred to as a target region).

[0246] Then, the generation unit 241 can set the motions before and after the target part is lost as the first motion data A and the second motion data B, respectively, and generate interpolated motion data C that smoothly interpolates the target part. Therefore, the motion of the body other than the target part obtained by motion capture is adopted as it is, and interpolation of the target part whose motion is lost becomes possible.

[0247] In addition, when performing interpolation processing of the target part, the target part to be interpolated on the time line of the application can be branched (for example, only the right hand, etc.) In addition, there may be multiple target parts for interpolating motion (for example, right hand + left hand, etc.).

[0248] In addition, the generation unit 241 can set the position constraint conditions for each part other than the target part to be interpolated as interpolation parameters. Therefore, interpolation motion data can be generated in which only the target part (for example, only the right hand) is interpolated and the entire body except the target part moves according to the motion data.

[0249] Furthermore, the calculation unit 243 can calculate the aforementioned score based on the target area generated by the generation unit 241. If there are multiple target areas, the calculation unit 243 can change the value of the interpolation parameter for each target area, weight each target area, and calculate the index. The operation display unit 220 can then display the interpolated motion data of the target area and the whole-body motion excluding the target area by connecting them.

[0250] In addition, in VR games or applications in the metaverse, motion capture systems are used to reflect the user's movements in real time in the avatar. In this case, there are cases where pre-registered animations (such as gestures or emotional movements) are injected into the real-time motion capture.

[0251] Here, the generation unit 421 can generate interpolated motion data C that smoothly interpolates the first motion data A and the second motion data B using real-time motion as the first motion data A and emotional motion as the second motion data B. This can allow for a natural transition from real-time motion to emotional motion. In addition, new representations (e.g., VR live, etc.) that combine real-time motion and pre-recorded motion become feasible.

[0252] (Composition using motion blending)

[0253] In addition, in the interpolation processing according to the embodiment of the present disclosure, interpolation processing by generating interpolated motion data and interpolation processing by motion blending can be used in combination. For example, the generation unit 241 can perform interpolation processing by performing motion blending on the upper body motion data and perform interpolation processing by generating interpolated motion data for the lower body.

[0254] In motion blending, for example, the generation unit 421 may interpolate the connection position of the motion data by linearly interpolating from the interpolation start posture of the first motion data A to the interpolation end posture of the second motion data B.

[0255] <<7. Hardware Configuration Example>>

[0256] The embodiments of the present disclosure have been described above. Information processing, such as the generation of interpolated motion data and the calculation of scores (total index and index) described above, is implemented through the collaboration of the software and hardware of the PC 20 described below. Note that the hardware configuration described below can also be applied to the server 10.

[0257] Figure 11 2001, a read-only memory (ROM) 2002, a random access memory (RAM) 2003, and a host bus 2004. Furthermore, the PC 20 includes a bridge 2005, an external bus 2006, an interface 2007, an input device 2008, an output device 2010, a storage device (HDD) 2011, a drive 2012, and a communication device 2015.

[0258] The CPU 2001 functions as an arithmetic processing device and a control device, and controls the overall operation of the PC 20 according to various programs. Alternatively, the CPU 2001 may be a microprocessor. The ROM 2002 stores programs, calculation parameters, and the like used by the CPU 2001. The RAM 2003 temporarily stores programs used in the execution of the CPU 2001, parameters that change appropriately during execution, and the like. These are interconnected via a host bus 2004 including a CPU bus and the like. Figure 3 The functions of the described generation unit 241 and calculation unit 243 can be realized by cooperation of the CPU 2001 , ROM 2002 , RAM 2003 , and software.

[0259] The host bus 2004 is connected to an external bus 2006 such as a peripheral component interconnect / interface (PCI) bus via a bridge 2005. Note that the host bus 2004, the bridge 2005, and the external bus 2006 do not necessarily need to be configured separately, and these functions may be installed on one bus.

[0260] The input device 2008 includes an input device for a user to input information (such as a mouse, keyboard, touch panel, buttons, microphone, switch, and joystick), an input control circuit that generates an input signal based on the user's input and outputs the input signal to the CPU 2001, and the like. By operating the input device 2008, the user of the PC 20 can input various data to the PC 20 and instruct the PC 20 to perform processing operations.

[0261] For example, the output device 2010 includes a display device such as a liquid crystal display device, an OLED device, and a lamp. Furthermore, the output device 2010 includes an audio output device such as a speaker and headphones. For example, the output device 2010 outputs reproduced content. Specifically, the display device displays various information such as reproduced video data as text or images. On the other hand, the audio output device converts audio data and the like into audio and outputs the audio.

[0262] The storage device 2011 is a device for storing data. The storage device 2011 may include a storage medium, a recording device for recording data in the storage medium, a reading device for reading data from the storage medium, a deletion device for deleting data recorded in the storage medium, and the like. For example, the storage device 2011 includes a hard disk drive (HDD). The storage device 2011 drives the hard disk and stores programs executed by the CPU 2001 and various data.

[0263] The drive 2012 is a reader / writer for storage media and is internally or externally attached to the PC 20. The drive 2012 reads information recorded in an attached removable storage medium 30 (such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory) and outputs the information to the RAM 2003. In addition, the drive 2012 can also write information to the removable storage medium 30.

[0264] For example, the communication device 2015 is a communication interface including a communication device for connecting to the network 1. Furthermore, the communication device 2015 may be a wireless LAN compatible communication device, a Long Term Evolution (LTE) compatible communication device, or a wired communication device that performs wired communication.

[0265] <<8.Supplement>>

[0266] The preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, but the present disclosure is not limited to such examples. Obviously, a person skilled in the art in the art to which the present disclosure belongs will be able to conceive of various changes or modifications within the scope of the technical concept described in the claims, and it is naturally understood that such changes or modifications also fall within the technical scope of the present disclosure.

[0267] For example, the PC 20 may also include all or some of the functional configurations of the server 10 according to an embodiment of the present disclosure. In the case where the PC 20 includes all the functional configurations of the server 10 according to an embodiment of the present disclosure, the PC 20 can perform a series of processes related to the generation of interpolated motion data and the calculation of indices without communicating via the network 1.

[0268] In addition, the server 10 may include a generation unit 241 and a calculation unit 243. In this case, example processing related to the generation of interpolated motion data and the calculation of the index may be performed on the server, and the result of the processing (e.g., a candidate list sorted in order of score) may be sent to the PC 20.

[0269] Furthermore, in the case where the server 10 includes a generation unit, the PC 20 can acquire the interpolated motion data generated by the generation unit of the server 10 from the server 10. In this case, the communication unit 210 included in the PC 20 corresponds to the acquisition unit.

[0270] Furthermore, the software can be used as a standalone application for searching and editing motion data, or the motion interpolation editing functionality can be used as a plug-in to existing digital content creation (DCC) tools.

[0271] In addition, the generation of interpolated motion data and motion blending have been exemplified as specific examples related to interpolation editing, but interpolation editing according to the present disclosure is not limited to these examples. For example, the generation unit 241 can interpolate motion data by combining interpolation editing through the generation of interpolated motion data and interpolation editing through text search of motion data. For example, a user can search for text of motion data of any part and select a piece of motion data from multiple pieces of motion data displayed as search results. In this case, the generation unit 241 can interpolate the motion of the part using the motion data specified by the user, and can interpolate the motion of another part by generating interpolated motion data.

[0272] In addition, depending on the combination of the first motion data A and the second motion data B, there is a possibility that the score calculated by the calculation unit 243 will not be high even if the interpolation parameter is changed. In this case, the operation display unit 220 can present recommendation information such as whether to change the second motion data B to another second motion data B'.

[0273] In addition, each step in the processing of the PC 20 in this specification does not necessarily need to be processed in time series in the order described in the flowchart. For example, each step in the processing of the PC 20 may be processed in an order different from the order described in the flowchart.

[0274] Furthermore, it is also possible to create a computer program for causing hardware such as CPU, ROM, and RAM built into the server 10 and PC 20 to exhibit the same functions as the respective configurations of the above-described server 10 and PC 20. Furthermore, a storage medium storing the computer program is also provided.

[0275] In addition, the effects described in this specification are merely illustrative or exemplary, and not restrictive. That is, according to the description of this specification, the technology according to the present disclosure can exhibit other effects that are obvious to those skilled in the art together with or instead of the above effects.

[0276] Note that the following configurations also belong to the technical scope of the present disclosure. (1)

[0278] An information processing device, comprising:

[0279] circuit, is configured as

[0280] obtaining interpolated motion data for interpolating between first and second motion data that are independent in time and space, and

[0281] Based on the first motion data, the second motion data, and the obtained interpolated motion data, an index related to the motion speed of each motion data is calculated. (2)

[0283] The information processing device according to (1), wherein

[0284] The index includes an index based on a difference between an average speed of the interpolated motion data and an average speed of the first motion data and the second motion data. (3)

[0286] The information processing device according to (1) or (2), wherein

[0287] The index includes a first index based on a difference between a maximum value of the velocity of the interpolated motion data and an average velocity of a plurality of frames including first motion data preceding the interpolated motion data, and

[0288] The index further includes a second index based on a difference between a maximum value of the velocity of the interpolated motion data and an average velocity of a plurality of frames including second motion data subsequent to the interpolated motion data. (4)

[0290] The information processing device according to any one of (1) to (3), wherein

[0291] The indices include indices based on differential components of acceleration of the interpolated motion data. (5)

[0293] The information processing device according to any one of (1) to (4), wherein

[0294] The metrics include metrics related to the velocity of the toe when the toe of the interpolated motion data contacts the ground. (6)

[0296] The information processing device according to any one of (1) to (5), wherein

[0297] The circuitry is further configured to calculate an indicator based on the usage scenario based on a difference between a property of the interpolated motion data and a property required for the interpolated motion data based on at least one condition of the usage scenario. (7)

[0299] The information processing device according to (6), wherein

[0300] The circuit is further configured to calculate a total index by calculating a plurality of indices related to a speed of each motion data and a plurality of indices related to a usage of each motion data, multiplying each of the plurality of indices by a weighting factor, and adding the plurality of indices. (8)

[0302] The information processing device according to (7), wherein

[0303] The circuit obtains interpolated motion data based on the first motion data and the second motion data. (9)

[0305] The information processing device according to (8), wherein

[0306] The circuit obtains interpolated motion data based on the first motion data, the second motion data, and at least one interpolation parameter related to the motion. (10)

[0308] The information processing device according to (9), wherein

[0309] The circuit obtains interpolated motion data by using a generative model, where the generative model is obtained by learning a relationship between two temporally and spatially independent motion data, at least one interpolation parameter, and the interpolated motion data. (11)

[0311] The information processing device according to (10), wherein

[0312] The circuit is further configured to modify at least one interpolation parameter to obtain interpolated motion data a plurality of times, and

[0313] The circuit calculates an overall metric based on each of the plurality of obtained interpolated motion data. (12)

[0315] The information processing device according to any one of (9) to (11), wherein

[0316] The at least one interpolation parameter includes a plurality of interpolation parameters, and

[0317] One or more interpolation parameters included in the plurality of interpolation parameters are automatically set based on a difference between a velocity of a root joint of the first motion data and a velocity of a root joint of the second motion data, or a difference between a spatial position of a root joint of the first motion data and a spatial position of a root joint of the second motion data. (13)

[0319] The information processing device according to (11), wherein

[0320] The circuit is further configured to obtain a candidate list of interpolated motion data according to a magnitude of the total indicator. (14)

[0322] The information processing device according to (13), wherein

[0323] The circuit obtains a candidate list, and in the candidate list, sorts the interpolated motion data in order according to the size of the total index. (15)

[0325] The information processing device according to (14), wherein

[0326] The circuit modifies at least one interpolation parameter within a preset predetermined number of times to obtain a predetermined number of interpolated motion data, and obtains an interpolated motion data candidate list by sorting the predetermined number of interpolated motion data in descending order of the total index. (16)

[0328] The information processing device according to any one of (13) to (15), wherein

[0329] The circuit is further configured to obtain sequentially connected motion data obtained by sequentially connecting the first motion data, the second motion data, and the interpolated motion data. (17)

[0331] The information processing device according to (16), wherein

[0332] The circuit obtains sequentially connected motion data obtained by connecting the first motion data, the second motion data, and at least one piece of interpolated motion data selected by a user from the candidate list. (18)

[0334] An information processing method executed by a computer, the method comprising:

[0335] obtaining interpolated motion data for interpolating between first and second motion data that are independent in time and space; and

[0336] Based on the first motion data, the second motion data, and the obtained interpolated motion data, an index related to the motion speed of each motion data is calculated. (19)

[0338] A non-transitory computer-readable storage medium having a program embodied thereon, the program, when executed by a computer, causing the computer to perform a method comprising:

[0339] acquiring interpolated motion data for interpolating between first motion data and second motion data that are independent in time and space; and

[0340] Based on the first motion data, the second motion data, and the obtained interpolated motion data, an index related to the motion speed of each motion data is calculated.

[0341] It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors insofar as they are within the scope of the appended claims or the equivalents thereof.

[0342] [List of Reference Numbers]

[0343] 1 Network

[0344] 10 Servers

[0345] 110 storage units

[0346] 120 learning units

[0347] 130 Communication Unit

[0348] 20PC

[0349] 210 Communication Unit

[0350] 220 Operation and display unit

[0351] 230 storage units

[0352] 240 control unit

[0353] 241 Generation Unit

[0354] 243 computing units.

Claims

1. An information processing device, comprising: circuitry configured to obtain interpolated motion data for interpolating between first and second motion data that are independent in time and space, and Based on the first motion data, the second motion data, and the obtained interpolated motion data, an index related to the motion speed of each motion data is calculated.

2. The information processing device according to claim 1, wherein: The index includes an index based on a difference between an average speed of the interpolated motion data and an average speed of the first motion data and the second motion data.

3. The information processing device according to claim 1, in, The index includes a first index based on a difference between a maximum value of the velocity of the interpolated motion data and an average velocity of a plurality of frames of the first motion data included before the interpolated motion data, and The index further includes a second index based on a difference between a maximum value of the speed of the interpolated motion data and an average speed of a plurality of frames of the second motion data subsequent to the interpolated motion data.

4. The information processing device according to claim 1, wherein: The index includes an index based on a differential component of acceleration of the interpolated motion data.

5. The information processing device according to claim 1, wherein: The index includes an index related to a velocity of a toe of the interpolated motion data when the toe contacts a surface.

6. The information processing device according to claim 1, wherein: The circuit is further configured to calculate the indicator according to the usage scenario based on a difference between a property of the interpolated motion data and a property required for the interpolated motion data based on at least one condition of the usage scenario.

7. The information processing device according to claim 6, wherein: The circuit is also configured to calculate a total index by: calculating multiple indexes related to the speed of each motion data and multiple indexes related to the usage of each motion data, multiplying each of the multiple indexes by a weighting factor, and adding the multiple indexes.

8. The information processing device according to claim 7, wherein: The circuit obtains the interpolated motion data based on the first motion data and the second motion data.

9. The information processing device according to claim 8, wherein: The circuit obtains the interpolated motion data based on the first motion data, the second motion data, and at least one interpolation parameter related to motion.

10. The information processing device according to claim 9, wherein: The circuit obtains the interpolated motion data by using a generation model, the generation model being obtained by learning a relationship between two motion data that are independent in time and space, the at least one interpolation parameter, and the interpolated motion data.

11. The information processing device according to claim 10, wherein: The circuit is further configured to modify the at least one interpolation parameter to obtain the interpolated motion data multiple times, and The circuit calculates the overall index based on each of the plurality of obtained interpolated motion data.

12. The information processing device according to claim 9, wherein: The at least one interpolation parameter comprises a plurality of interpolation parameters, and One or more interpolation parameters included in the plurality of interpolation parameters are automatically set based on a difference between a root joint velocity of the first motion data and a root joint velocity of the second motion data, or a difference between a spatial position of a root joint of the first motion data and a spatial position of a root joint of the second motion data.

13. The information processing device according to claim 11, wherein: The circuit is further configured to obtain a candidate list of interpolated motion data according to the size of the total indicator.

14. The information processing device according to claim 13, wherein: The circuit obtains a candidate list, in which the interpolated motion data are sorted in order according to the size of the total index.

15. The information processing device according to claim 14, wherein: The circuit modifies the at least one interpolation parameter within a preset predetermined number of times to obtain a predetermined number of interpolated motion data, and A candidate list of interpolation motion data is obtained by sorting the predetermined number of interpolation motion data in descending order of the total index.

16. The information processing device according to claim 13, wherein: The circuit is also configured to obtain sequentially connected motion data obtained by sequentially connecting the first motion data, the second motion data, and the interpolated motion data.

17. The information processing device according to claim 16, wherein: The circuit obtains sequentially connected motion data obtained by connecting the first motion data, the second motion data, and at least one piece of interpolated motion data selected by a user from the candidate list.

18. An information processing method executed by a computer, the method comprising: Obtain interpolated motion data for interpolating between first motion data and second motion data that are independent in time and space; and Based on the first motion data, the second motion data, and the obtained interpolated motion data, an index related to the motion speed of each motion data is calculated.

19. A non-transitory computer-readable storage medium having a program embodied thereon, wherein when the program is executed by a computer, the computer is caused to perform a method, the method comprising: Acquire interpolation motion data for interpolating between first motion data and second motion data that are independent in time and space; and Based on the first motion data, the second motion data, and the obtained interpolated motion data, an index related to the motion speed of each motion data is calculated.

Images