Action redirection method and device, cluster, storage medium and program product

Abstract

The embodiment of the invention discloses an action redirection method and device, a cluster, a storage medium and a program product, relates to the technical field of artificial intelligence, and improves the action redirection effect. The method comprises the steps of obtaining action data generated by action execution of a source object; acquiring a skeleton model of a to-be-driven role; according to the skeleton model of the to-be-driven role, skeleton tree information and skeleton length information of the to-be-driven role are determined, and the skeleton length information of the to-be-driven role comprises the skeleton length of the to-be-driven role; action data of a source object, skeleton tree information of a to-be-driven role and skeleton length information of the to-be-driven role are input into a neural network model to obtain action data of action execution of the to-be-driven role, and the neural network model is used for executing action execution of the to-be-driven role based on the skeleton length information of the to-be-driven role. And adjusting the positions of partial or all skeleton joint points in the action data of the source object to the positions matched with the skeleton model of the to-be-driven role.

Description

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on September 11, 2023, with application number 202311163734.7 and application name “A method, device and computing device cluster for action redirection”, all contents of which are incorporated by reference in this application. Technical Field

[0002] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular to an action redirection method, device, cluster, storage medium, and program product. Background Art

[0003] As motion capture technology becomes increasingly mature, it is possible to drive virtual objects by collecting motion data of real human body movements through motion capture equipment. Since the body shape or body proportion of real people is often different from that of virtual objects, when using real people's motion data to drive virtual objects, motion redirection is required, that is, remapping the real people's motion data into motion data that matches the virtual object's skeletal model, so that the driven virtual object can perform the same body movements as real people.

[0004] However, when the skeletal models of real people and virtual objects are significantly different, the effect of motion redirection in the prior art is poor, resulting in a low degree of restoration of the virtual objects to the real people's motions. Summary of the invention

[0005] The embodiments of the present application provide an action redirection method, device, cluster, storage medium and program product, which ensure the effect of action redirection.

[0006] In a first aspect, the present application provides an action redirection method, the method comprising: obtaining action data generated by a source object performing an action, wherein the type of the source object includes a real person and / or a digital person, and the action is a physical action of the source object; obtaining a skeletal model of a character to be driven; determining the skeleton tree information and bone length information of the character to be driven based on the skeletal model of the character to be driven, wherein the skeleton tree information of the character to be driven is used to describe the skeleton tree structure of the character to be driven, and the bone length information of the character to be driven includes the length of the skeleton of the character to be driven; obtaining the action data of the character to be driven performing the action by inputting the action data of the source object, the skeleton tree information of the character to be driven, and the bone length information of the character to be driven into a neural network model, wherein the neural network model is used to adjust the positions of some or all of the bone joint points in the action data of the source object to positions matching the skeleton model of the character to be driven based on the bone length information of the character to be driven.

[0007] It is understandable that by acquiring the skeleton model of the character to be driven, the skeleton tree information and the skeleton length information of the character to be driven can be determined. By inputting the skeleton length information of the character to be driven into the neural network model as an additional supervisory signal together with the action data of the source object and the skeleton tree information of the character to be driven, the neural network model can then adjust the positions of some or all of the skeleton joints in the action data of the source object to positions that match the skeleton model of the character to be driven based on the skeleton length information of the character to be driven, so that the neural network model outputs action data with a high degree of matching with the skeleton model of the character to be driven for driving the character to be driven to perform actions, thereby making the actions performed by the character to be driven have a high degree of restoration to the actions of the source object.

[0008] In a possible implementation, the neural network model includes a first encoding-decoding network structure and a second encoding-decoding network structure, and the method also includes: obtaining motion data generated by a human body performing a specified action, wherein the motion data of the human body includes position information of skeletal joints of the human body; determining the skeleton tree information of the human body according to the motion data of the human body, and the skeleton tree information of the human body is used to describe the skeleton tree structure of the human body; calculating the bone length information of the human body according to the position information of the skeletal joints of the human body, and the bone length information includes the length of the human body; using the encoder of the first encoding-decoding network structure, encoding the skeleton tree information and the bone length information of the human body into a length latent variable; using the encoder of the second encoding-decoding network structure, encoding the motion data of the human body into a motion latent variable; inputting the length latent variable and the motion latent variable into a decoder of the second encoding-decoding network structure to obtain decoded motion data; and training the neural network model with the goal of the error between the decoded motion data and the motion data of the human body being less than a preset threshold.

[0009] It can be understood that before using the neural network model for model inference to complete action redirection, the neural network model can be pre-trained. When performing model training, the human body's skeleton tree information and the human body's bone length information are jointly encoded into a length latent variable through the encoder of the first codec network. The length latent variable and the action latent variable generated by encoding the encoder of the second codec network are jointly input into the decoder of the second codec network to obtain decoded action data. The parameters of the two codec networks are updated with the goal of making the error between the decoded action data and the human body action data less than a preset threshold, thereby completing the model training. It is ensured that the trained neural network model can adjust the positions of some or all of the bone joints in the action data of the source object to a position that matches the bone model of the character to be driven based on the bone length information of the character to be driven, thereby improving the effect of subsequent use of the neural network model for action redirection.

[0010] In a possible implementation, the motion data of the human body is fused motion data, and the motion data generated by the human body performing a prescribed motion is obtained, including: obtaining multiple motion data of the human body performing the prescribed motion collected by multiple motion capture methods; fusing the multiple motion data to obtain the fused motion data.

[0011] It is understandable that the motion data of the human body used as a training sample can be fused motion data. Since the motion data collected by a single motion capture method may have errors compared to the actual data, multiple motion capture methods can be combined to obtain multiple motion data for specified actions and then fused, thereby improving the data accuracy of the training samples, thereby improving the training effect of the neural network model, and further improving the effect of subsequent motion redirection using the neural network model.

[0012] In a possible implementation, the multiple motion capture modes include at least two of inertial motion capture, visual motion capture, and optical motion capture.

[0013] In one possible implementation, before obtaining the motion data generated by a human body performing a prescribed action, the method also includes: obtaining original motion data of the human body, the original motion data including position information of skeletal joints when the human body last performed the prescribed action; determining the error between the original motion data of the human body and the standard motion data of the prescribed action, the error being used to indicate the standard degree of the human body's execution of the prescribed action; and when the error is greater than a specified threshold, outputting indication information, the indication information being used to guide the human body to adjust its limb posture when performing the prescribed action.

[0014] It is understandable that before obtaining the motion data generated by the human body performing the prescribed action as a training sample, in order to avoid the effect of using the motion data as a sample to train the neural network model in the future due to the large deviation between the prescribed action performed by the human body and the standard prescribed action, the obtained original motion data can be compared with the standard motion data. If the error is greater than the specified threshold, an instruction message can be output to remind the human body to adjust the limb posture when performing the prescribed action, thereby ensuring that the motion data used to train the model will not deviate significantly from the standard action, thereby improving the training effect of the neural network model, and further improving the implementation effect of the subsequent use of the neural network model for motion redirection.

[0015] In a second aspect, an embodiment of the present application provides an action redirection device, which is used to execute any one of the action redirection methods provided in the first aspect above.

[0016] In a possible implementation, the embodiment of the present application may divide the action redirection device into functional modules according to the method provided in the first aspect above. For example, each functional module may be divided according to each function, or two or more functions may be integrated into one processing module. Exemplarily, the embodiment of the present application may divide the action redirection device into an acquisition module, a determination module, and a processing module, etc. according to the function. The description of the possible technical solutions and beneficial effects executed by each of the functional modules divided above can refer to the technical solutions provided in the first aspect above or its corresponding possible implementation, and will not be repeated here.

[0017] In a third aspect, an embodiment of the present application provides a computing device, which includes a processor and a memory, wherein the processor is coupled to the memory; the memory is used to store computer instructions, which are loaded and executed by the processor to enable the computing device to implement the action redirection method described in the above aspects.

[0018] In a fourth aspect, an embodiment of the present application provides a computing device cluster, which includes at least one computing device, each computing device including: a processor and a memory, the processor of at least one computing device is used to execute instructions stored in the memory of at least one computing device, so that the computing device cluster executes the action redirection method provided in various optional implementations of the above-mentioned first aspect.

[0019] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, in which at least one computer program instruction is stored, and the computer program instruction is loaded and executed by a processor to implement the method for generating actions of a virtual object as described in the above aspects.

[0020] In a sixth aspect, an embodiment of the present application provides a computer program product, the computer program product including computer instructions, the computer instructions stored in a computer-readable storage medium. A processor of a computing device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computing device cluster executes the action redirection method provided in various optional implementations of the first aspect above.

[0021] For the specific description of the second to sixth aspects and their various implementations in the present application, reference may be made to the detailed description in the first aspect and its various implementations; and for the beneficial effects of the second to sixth aspects and their various implementations, reference may be made to the beneficial effects analysis in the first aspect and its various implementations, which will not be repeated here.

[0022] These and other aspects of the present application will become more apparent from the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is a schematic diagram of an action modeling according to an exemplary embodiment;

[0024] Figure 2 is a schematic diagram of a scenario of action redirection according to an exemplary embodiment;

[0025] Figure 3 yes Figure 2 A schematic diagram of a motion capture device involved in the illustrated embodiment collecting motion data of a real person;

[0026] Figure 4 is a schematic diagram of a hardware structure of a computing device according to an exemplary embodiment;

[0027] Figure 5 is a flowchart of an action redirection method according to an exemplary embodiment;

[0028] Figure 6 yes Figure 5 A schematic diagram of a first encoding-decoding network training update involved in the illustrated embodiment;

[0029] Figure 7 yes Figure 5 A schematic diagram of a second encoding-decoding network training update involved in the illustrated embodiment;

[0030] Figure 8 yes Figure 5 A schematic diagram of a process flow of action redirection involved in the illustrated embodiment;

[0031] Fig. 9 is an architecture diagram of an action redirection system according to an exemplary embodiment;

[0032] Fig.10 is a structural schematic diagram of an action redirection device according to an exemplary embodiment;

[0033] Fig.11 is a schematic diagram of a computing device according to an exemplary embodiment;

[0034] Fig.12 is a schematic diagram of a computing device cluster according to an exemplary embodiment;

[0035] Fig.13 The diagram is a schematic diagram showing a connection mode between computing device clusters according to an exemplary embodiment. DETAILED DESCRIPTION

[0036] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0037] The term "multiple" as used herein refers to two or more than two. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the related objects are in an "or" relationship.

[0038] Furthermore, in the description of this application, unless otherwise specified, "plurality" means two or more than two. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0039] In addition, in order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, the words "first", "second" and the like are used to distinguish the same items or similar items with substantially the same functions and effects. Those skilled in the art will understand that the words "first", "second" and the like do not limit the quantity and execution order, and the words "first", "second" and the like do not necessarily limit the differences. At the same time, in the embodiments of the present application, the words "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or design solutions. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete manner for ease of understanding.

[0040] First, the application scenarios of the embodiments of the present application are exemplarily introduced.

[0041] Through virtual digital space products such as animation, film, video, games, and the metaverse, the process of objects performing actions in the real space can be mapped in the virtual space for animation video display. This process is also called digital human driving. Digital human driving technology can be widely used in film and television production, animation, human-computer interaction, and motion analysis.

[0042] The motion capture technology can be used to collect the motion of a motion capture object in real space. Among them, the motion capture technology can include a variety of motion capture methods, and the various motion capture methods can include optical motion capture, inertial motion capture, and visual motion capture. The motion data of a certain motion capture object in real space is collected by a certain motion capture method, and the motion of the motion capture object in real space can be mapped in the virtual space for display through the motion capture technology. The motion data of the virtual object can be generated by the collected motion data of the captured object, so as to drive the skeleton model of the virtual object to perform actions according to the motion data of the virtual object. In this application, the motion capture object refers to a real person used to capture limb movements, and the virtual object refers to a digital person to be driven. The skeleton model can be a structure composed of virtual skeletal joints, skeleton position information and connection relationships that is not displayed inside the model of the virtual object.

[0043] (1) Optical motion capture technology, that is, motion capture technology that uses infrared sensors to collect motion data of the motion capture object. The implementation process of optical motion capture technology is as follows. Reflective markers are attached to key parts of the motion capture object, such as the joints. Cameras at multiple angles are set up to emit infrared light to the motion capture object. The infrared light is reflected by each reflective marker on the motion capture object and received by the infrared sensors on each camera. Image data of the motion capture object during the execution of the action is obtained, and the position information of each reflective marker in three-dimensional space is detected by image recognition and analysis technology as motion data.

[0044] The process of capturing motion data through optical motion capture technology relies on infrared sensors to collect infrared light reflected by reflective markers to achieve the purpose of obtaining motion data of each reflective marker. Therefore, when the reflective markers are blocked, the motion data of the key parts where the blocked reflective markers are located cannot be accurately collected, resulting in poor effect of collecting motion data in the presence of occlusion. Therefore, the requirements for the collection environment of motion data are relatively stringent.

[0045] (2) Inertial motion capture technology, that is, motion capture technology that collects motion data of the motion capture object through inertial sensors. The implementation process of inertial motion capture technology is as follows. Inertial sensors are worn on key parts of the motion capture object, such as the position of the joints. The inertial sensors may include inertial sensors that integrate electronic components such as accelerometers, gyroscopes, and magnetometers. These inertial sensors can capture the motion posture information of the motion capture object in three-dimensional space in real time, and then the spatial position of the joints can be calculated through algorithms. Specifically, by integrating inertial sensors such as accelerometers, gyroscopes, and magnetometers, the angular velocity, acceleration, etc. of each joint of the motion capture object can be recorded, and the position information of the skeletal joints can be determined as motion data. By giving an initial state, the motion data is integrated and calculated in time series to estimate the current motion posture.

[0046] The process of capturing motion data through inertial motion capture technology relies on inertial sensors to collect angular velocity and acceleration, and estimates the position and motion posture of joints by integrating the angular velocity and acceleration. Since the motion data of each joint needs to be accumulated in time series, this will lead to accumulated errors during long-term operation, making the accuracy of the motion data collected by motion capture increasingly poor.

[0047] (3) Visual motion capture technology, that is, motion capture technology that uses optical sensors to collect motion data of the motion capture object. The implementation process of optical motion capture technology is as follows. Images of the motion capture object performing actions are collected through one or more ordinary cameras, and the motion data of the key parts of the motion capture object are estimated through a deep neural network. Similarly, since it is necessary to rely on optical sensors to collect images to achieve the purpose of obtaining the motion data of each key part, when the key part is blocked, the motion data of the blocked key part cannot be accurately collected, resulting in poor effect of collecting motion data in the presence of occlusion. Therefore, the collection environment requirements for collecting motion data are relatively stringent.

[0048] Since the body shape or body proportions of the motion capture object are often different from those of the virtual object, when using the motion data of the motion capture object collected by motion capture technology to drive the virtual object, motion redirection is required, that is, remapping the motion data of the motion capture object into motion data that matches the virtual object's skeletal model, so that the driven virtual object can perform the same limb movements as a real person.

[0049] It should be noted that the motion capture object is also referred to as the source object in the embodiment of the present application, and the virtual object is also referred to as the character to be driven in the embodiment of the present application. In addition, in actual applications, there will be a situation where the motion data of a virtual object A is used to drive another virtual object B. In this case, the virtual object A is also referred to as the source object. Therefore, in the embodiment of the present application, the types of source objects include real people and digital people.

[0050] However, when the skeletal models of the source object and the character to be driven are quite different, the action redirection effect of the prior art is poor, resulting in a low degree of restoration of the actions performed by the driven virtual object to the actions of the real person. Figure 1 Schematic diagram of an action modeling provided in an embodiment of the present application. Figure 1 As shown, there is a large difference between the proportions of the forearm and upper arm of the source object 31 performing the waving action and the proportions of the forearm and upper arm of the character to be driven 32 performing the waving action. When the character to be driven 32 performs the waving action according to the collected action data of the source object 31, the forearm or upper arm bones of the character to be driven 32 conflict with the positions of the bones in other parts, resulting in the phenomenon of model penetration.

[0051] In view of this, in order to avoid the problem that when the skeletons of the source object and the character to be driven are greatly different, when the action data of the source object is used to drive the character to be driven, the action performed by the character to be driven has a poor restoration degree to the action of the source object, the embodiment of the present application proposes a new action redirection method. The method inputs the bone length information of the character to be driven into the neural network model together with the action data of the source object and the skeleton tree information of the character to be driven as an additional supervisory signal, and then the neural network model can adjust the positions of some or all of the bone joints in the action data of the source object to positions that match the skeleton model of the character to be driven based on the bone length information of the character to be driven, so that the neural network model outputs action data with a high degree of matching with the skeleton model of the character to be driven for driving the character to be driven to perform the action, so that the action performed by the character to be driven has a high degree of restoration to the action of the source object.

[0052] in, Figure 2 A schematic diagram of a scenario of action redirection provided by an embodiment of the present application is shown. Figure 2 As shown, a neural network model is stored in the computing device 10, and the neural network model can be used to adjust the positions of some or all of the skeletal joints in the action data of the source object to positions that match the skeletal model of the character to be driven based on the skeletal length information of the character to be driven. The neural network model includes a first encoding-decoding network structure and a second encoding-decoding network structure.

[0053] The computing device 10 can be used to train a neural network model, and can also be used to apply a neural network model.

[0054] During the model training stage, a real person can wear a motion capture device 11, which collects the real person's motion data. The motion capture device 11 sends the collected motion data to the computing device 10, which processes the collected motion data to determine the bone length and skeleton tree information of the bones related to the real person's execution of the action, and uses the encoder of the first encoding-decoding network structure to encode the human body's skeleton tree information and bone length information into length latent variables. Use the encoder of the second encoding-decoding network structure to encode the human body's motion data into motion latent variables, and then input the length latent variables and the motion latent variables into the decoder of the second encoding-decoding network structure to obtain decoded motion data, and train the neural network model with the goal of the error between the decoded motion data and the human body's motion data being less than a preset threshold.

[0055] The computing device 10 may be a server, a computer device or a terminal, and the computing device 10 may be a device with video processing. The motion capture device 11 may include motion capture devices corresponding to a variety of motion capture technologies, for example, the target object may wear inertial sensors and reflective marking points at the joints, and a camera with an infrared sensor at multiple angles may be set up to shoot the target object, and a camera with an optical sensor may also be set up to shoot the target object.

[0056] The computing device 10 is communicatively connected to the motion capture device 11 , and the motion capture device 11 can transmit data to the computing device 10 .

[0057] In the model application stage, the computing device 10 can obtain the action data generated by the execution action of the source object, and can also obtain the skeleton model of the character to be driven. According to the skeleton model, the skeleton tree information and the skeleton length information of the character to be driven can be determined. The computing device 10 can input the action data of the source object, the skeleton tree information of the character to be driven, and the skeleton length of the character to be driven into the trained neural network model, thereby obtaining the action data of the action performed by the character to be driven.

[0058] The source object may be a real person or a digital person, that is, a virtual object. The character to be driven may be a virtual object in a virtual space that is driven to redirect the action of the source object. The character to be driven may be a three-dimensional model created based on animation skeleton technology. The character to be driven may have its own shape, volume and orientation in the three-dimensional virtual space and occupy a part of the space in the three-dimensional virtual space.

[0059] In addition, the computing device 10 as the device where the model is applied can have a video display function. According to the action data of the source object, the bone length information of the character to be driven and the skeleton tree information, action redirection can be realized. The action performed by the source object is redirected so that the character to be driven can perform it, and the video of the redirected action performed by the character to be driven is displayed on the image display interface 12 for presentation.

[0060] For example, Figure 3 Schematic diagram of a motion capture device collecting real-person motion data according to an embodiment of the present application. Figure 3 As shown, when the real person 21 performs a prescribed action, the motion capture device collects the motion data of the real person 21. The real person 21 can divide the body skeleton according to each joint point according to the animation skeleton technology.

[0061] The motion data may include the position information of each skeletal joint point, etc. Figure 3 As shown, the position coordinates (x, y, z) of the elbow joint point, or it can be a relative angle.

[0062] In one possible implementation, the computing device 10 may obtain motion data of multiple real people performing prescribed actions through a variety of motion capture methods as sample data in the training process of the neural network model. The various motion capture methods may include visual motion capture, inertial motion capture, and optical motion capture.

[0063] That is, in one case, the computing device 10 can obtain the motion data of the real person collected by the motion capture device 11 through the motion capture device 11; in another case, the computing device 10 can obtain the video containing the real person and analyze the prescribed actions performed by the real person in the video to obtain the motion data of the real person, so as to ensure that the motion data of the real person can be used to train the neural network model later.

[0064] Figure 4 It is a hardware structure diagram of a computing device 10 provided in an embodiment of the present application. The computing device 10 may be a server. The server may be an X86 architecture server, specifically a blade server, a high-density server, a rack server or a high-performance server, etc. The server includes a processor 201, a memory 202 and a bus 203. The server may also include an external display device 204.

[0065] The processor 201 may include one or more processing units, for example, the processor 201 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0066] The memory 202 is used to store data, wherein the memory 202 includes but is not limited to a random access memory (RAM), a read-only memory (ROM), a flash memory, an optical memory, a database, etc.

[0067] The memory 202 is used to interact with the processor 201. The memory 202 may store standard action templates corresponding to various types of actions. Each type of action may correspond to one or more standard action templates, which are used to indicate the execution standards of various types of actions. That is to say, after the computing device 10 receives the action data of a real person, it may compare it with the corresponding standard action template to determine whether the action currently executed by the real person meets the standard. If it does not meet the standard, the real person may be prompted to adjust the executed action through the external display device 204 or the terminal interacting with the computing device 10.

[0068] The processor 201 , the memory 202 and the external display device 204 are usually connected to each other via the bus 203 , or are connected to each other in other ways.

[0069] The external display device 204 can also be used to display a display interface to show the user the actions performed by the source object in the virtual space by the character to be driven during the application stage of the neural network model.

[0070] Processor 201 can be used to calculate the bone length information of each bone involved in the action based on the motion data collected by various motion capture methods. Processor 201 can also be used to implement the process of adaptive motion redirection based on the motion data and combined with the bone length.

[0071] It should be noted that the application scenarios and system architectures described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Ordinary technicians in this field can know that with the evolution of the system architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0072] For ease of understanding, the following is an exemplary introduction to the action redirection method provided by the present application in conjunction with the accompanying drawings. The action redirection method is applicable to Figure 2 , Figure 4 Computing device 10 is shown.

[0073] Figure 5 A flow chart of an action redirection method provided by an exemplary embodiment of the present application is shown. The action redirection method can be executed by a computing device, and the action redirection method includes the following steps:

[0074] S101, obtaining action data generated by a source object executing an action.

[0075] In the embodiment of the present application, in the process of implementing action redirection, the computing device may obtain action data generated by the source object performing an action. The type of the source object may include a real person and / or a digital person, the action performed by the source object may be a body movement of the source object, the action data may be the position information of each skeletal joint point when the source object performs the action, and the action redirection may be redirecting the action performed by the source object to the character to be driven, so that the character to be driven performs the corresponding action.

[0076] For example, when the source object is a real person, the computing device can collect the motion data generated when the real person performs the action by one or more motion capture methods. When the source object is a digital person, the computing device can obtain the motion data generated when the digital person performs the action by analyzing the video of the digital person performing the action or directly obtaining the code.

[0077] S102, obtaining a skeleton model of the character to be driven.

[0078] In an embodiment of the present application, a computing device may obtain a skeleton model of the character to be driven, where the skeleton model is a model structure obtained by connecting the skeleton joint points of the character to be driven according to a connection relationship.

[0079] The execution order of S101 and S102 is not limited here, and they can be executed successively or simultaneously.

[0080] In a possible implementation, the skeleton model of the character to be driven may be input by the user, and the computing device may receive the skeleton model of the character to be driven input by the user. Alternatively, the computing device may store multiple virtual characters, that is, skeleton models of digital human characters, created in advance. Before driving is required, the user may select a digital human character from multiple digital human characters as the character to be driven, and the computing device may obtain the skeleton model corresponding to the character to be driven from the model library storing the skeleton models of multiple digital human characters according to the received selection operation.

[0081] S103, determining the skeleton tree information and the bone length information of the character to be driven according to the bone model of the character to be driven.

[0082] In an embodiment of the present application, the computing device can determine the skeleton tree information and bone length information of the character to be driven based on the skeleton model of the character to be driven. The skeleton tree information of the character to be driven is used to describe the skeleton tree structure of the character to be driven, and the bone length information of the character to be driven includes the length of the bones of the character to be driven.

[0083] The skeleton tree structure includes the hierarchical relationship of the skeleton tree and the connection relationship of each layer of bones in the skeleton tree.

[0084] In one possible implementation, the acquired skeletal model of the character to be driven may include the relative position information of each skeletal joint point and the connection relationship between the skeletal joint points. The bone length information of the bone connecting two skeletal joint points may be calculated through the relative position information of the two skeletal joint points.

[0085] S104, obtaining the action data of the to-be-driven character to perform the action by inputting the action data of the source object, the skeleton tree information of the to-be-driven character, and the bone length information of the to-be-driven character into the neural network model.

[0086] In an embodiment of the present application, the computing device can input the action data of the source object, the skeleton tree information of the character to be driven, and the bone length information of the character to be driven into the trained neural network model to obtain the action data of the action performed by the character to be driven output by the neural network model.

[0087] Among them, the neural network model is used to adjust the positions of some or all of the bone joint points in the action data of the source object to positions that match the bone model of the character to be driven based on the bone length information of the character to be driven.

[0088] In one possible implementation, before applying the neural network model to obtain action data for the character to be driven to perform an action, the computing device may perform model training on the neural network model.

[0089] The process of training the neural network model includes the following steps:

[0090] S11, the computing device obtains the motion data generated by the human body performing the prescribed motion.

[0091] In order to train the neural network model, it is first necessary to collect sample data sets. The computing device can obtain multi-point motion data of different human bodies performing prescribed actions. The prescribed actions can be actions of multiple types.

[0092] The prescribed action may be an action belonging to a certain action type performed by a certain body part of a human body during exercise. The computing device may store a template of a standard action corresponding to the prescribed action, and the template of the standard action may include standard action data.

[0093] For example, the human body parts may include limbs, torso, shoulders, neck and head. Each body part includes multiple bone joints. Different body parts of the human body can perform different types of actions. The types of actions that can be performed by the upper limbs include waving, swinging, punching, elbowing, etc.; the types of actions that can be performed by the lower limbs include walking, running, jumping, squatting, sitting, dancing, etc.; the types of actions that can be performed by the shoulders, neck and head include shaking heads, nodding, etc.; the types of actions that can be performed by the torso include bowing, twisting the waist, etc.

[0094] In one possible implementation, before obtaining the motion data generated by the human body performing the prescribed action, the computing device may also obtain the original motion data of the human body, and then determine the error between the original motion data of the human body and the standard motion data of the prescribed action. The error can be used to indicate the standard degree of the human body's execution of the prescribed action. When the error is greater than a specified threshold, an indication information is output.

[0095] The original motion data may include the position information of the skeletal joints when the human body last performed the prescribed motion, and the instruction information may be used to guide the human body to adjust the limb posture when performing the prescribed motion.

[0096] That is to say, after the human body performs a prescribed action, the computing device collects the original action data generated by the human body performing the prescribed action. Since the computing device stores the standard action data of the prescribed action, in order to avoid a large deviation between the prescribed action performed by the human body and the standard action, and the situation where the sample standard cannot be met when used as training data, the original action data can be compared with the standard action data to obtain the error between the two. When the error is greater than a specified threshold, an instruction message is generated and output to prompt the human body to adjust its posture and re-perform the prescribed action.

[0097] In a possible implementation, after the computing device obtains the original action data, it obtains a standard action template corresponding to the action type from a template library according to the action type of the prescribed action, and the standard action template contains the standard action data.

[0098] In one possible case, if each action type corresponds to at least two standard action templates, at least two standard action templates corresponding to the action type of the specified action are determined, and the computing device can determine the standard action data from the at least two standard action templates to be compared with the collected original motion data based on the user's selection operation.

[0099] The indication information may be a prompt message displayed to the user in a page pop-up window, or a prompt message displayed to the user through voice or a prompt light, etc. The display method of the indication information is not limited.

[0100] For example, if the action type of the prescribed action is waving, the standard action templates with the action type of waving may include standard action template 1 and standard action template 2, wherein the waving amplitude of standard action template 1 is larger than that of standard action template 2. The user may select one from standard action template 1 and standard action template 2 as required as a standard action template corresponding to the standard action data for comparison.

[0101] In one possible implementation, if the motion data of the human body is fused motion data, the computing device may obtain multiple motion data of the human body performing prescribed motions collected through multiple motion capture methods, fuse the multiple motion data, and obtain fused motion data.

[0102] In one possible implementation, the computing device may obtain the weights of at least two motion capture methods, and fuse multiple motion data of a human body performing a specified motion according to the weights of at least two motion capture methods to obtain fused motion data.

[0103] Optionally, the weights of at least two motion capture methods may be preset.

[0104] For example, if the computing device obtains motion data generated by a human body performing a prescribed action collected by optical motion capture, inertial motion capture, and visual motion capture, respectively, the computing device pre-stores the weight of the motion data collected by the visual motion capture method when performing data fusion, which is 0.3, the weight of the motion data collected by the inertial motion capture method when performing data fusion, which is 0.4, and the weight of the motion data collected by the optical motion capture method when performing data fusion, which is 0.3. When the computing device obtains motion data generated by a human body performing a prescribed action collected by different motion capture methods, for example, the position information of the skeletal joint point 1 indicated by the motion data generated by the human body performing the prescribed action collected by the visual motion capture method is (x1, y1, z1), the position information of the skeletal joint point 1 indicated by the motion data generated by the human body performing the prescribed action collected by the optical motion capture method is (x2, y2, z2), and the position information of the skeletal joint point 1 indicated by the motion data generated by the human body performing the prescribed action collected by the inertial motion capture method is (x3, y3, z3). According to the weights corresponding to each sensor, it can be determined that the motion data obtained after data fusion of skeletal joint point 1 indicates that the x-axis coordinate of the position information of skeletal joint point 1 is (0.3x1+0.3x2+0.4x3) / 3, the y-axis coordinate is (0.3y1+0.3y2+0.4y3) / 3, and the z-axis coordinate is (0.3z1+0.3z2+0.4z3) / 3.

[0105] Optionally, the weight of each of the at least two motion capture methods may be determined according to the acquisition environment in which the human body is located.

[0106] That is, the acquisition environment of the human body may include whether the human body is blocked. If the human body is blocked when collecting motion data, a lower weight may be assigned to the visual motion capture method and the optical motion capture method than to the inertial sensor.

[0107] For example, if a human body is waving and the position of the human body is obstructed for a period of time, since the visual motion capture method and the optical motion capture method are picky about the collection environment with occlusion, there will be a large error in the collection of motion data of the obstructed human body in the presence of occlusion. Therefore, the weights of the visual motion capture method and the optical motion capture method need to be appropriately reduced.

[0108] In one possible case, the motion capture device can inform the computing device of the duration or degree of occlusion of the human body, and the computing device can assign the weights of data fusion to each motion capture method according to the duration or degree of occlusion. Among them, the weights of the visual motion capture method and the optical motion capture method can be negatively correlated with the duration or degree of occlusion; the weight of the inertial motion capture method can be positively correlated with the duration or degree of occlusion.

[0109] That is to say, the longer the human body whose motion data is collected is blocked or the greater the degree of blockage, the smaller the weights that need to be allocated to the visual motion capture method and the optical motion capture method. Conversely, the shorter the human body whose motion data is collected is blocked or the smaller the degree of blockage, the greater the weights that need to be allocated to the visual motion capture method and the optical motion capture method. The longer the human body whose motion data is collected is blocked or the greater the degree of blockage, the greater the weight that needs to be allocated to the inertial motion capture method. Conversely, the shorter the human body whose motion data is collected is blocked or the smaller the degree of blockage, the smaller the weight that needs to be allocated to the inertial motion capture method.

[0110] Optionally, the weight of each of the at least two motion capture methods may be adjusted and determined according to user instructions.

[0111] That is to say, the user can send user instructions to the computing device to determine the weight of each motion capture method according to the requirements of the current acquisition environment for the motion capture method to be used.

[0112] For example, if the user determines that the acquisition environment of the current motion capture process has less occlusion of the human body, the user can input a user instruction to the computing device, and the user instruction includes the weight value of the motion data collected by each motion capture method when the computing device performs data fusion on the motion data. The computing device can calculate the fused motion data according to the weight indicated by the user.

[0113] S12, the computing device determines the skeleton tree information of the human body according to the motion data of the human body.

[0114] In an embodiment of the present application, after acquiring the motion data of the human body, the computing device can determine the skeleton tree information of the human body according to the position information of each bone joint point indicated by the motion data.

[0115] S13, the computing device calculates the bone length information of the human body according to the position information of the bone joints of the human body.

[0116] Since the computing device obtains the position information of the skeletal joints of the human body, the bone length information of each bone of the human body can be calculated according to the respective position information of two skeletal joints connected by the bones.

[0117] The skeleton may be used to connect a first joint point and a second joint point in a human body.

[0118] That is, the bone length information of the bone may be the bone length between the first joint point and the second joint point. For example, if the first joint point is the left elbow joint point, the second joint point may be the left wrist joint point or the left shoulder joint adjacent to the left elbow joint point, the bone may be the left forearm bone between the left elbow joint point and the left wrist joint point, and the bone may also be the left upper arm bone between the left elbow joint point and the left shoulder joint point.

[0119] In a possible implementation, the bone length of the bone is determined based on the position information of the first joint point and the position information of the second joint point.

[0120] That is to say, the computing device can fuse the motion data collected by at least two motion capture methods, and use the fused motion data to determine the position information of the first joint and the second joint. Since the position information is collected by fusing multiple motion capture methods, it is more accurate than the position information collected by a single motion capture method. Therefore, it can be ensured that the bone length of the bone calculated by using the more accurate position information of the first joint and the second joint is more accurate.

[0121] Exemplarily, the position information of the first joint point can be the three-dimensional coordinates of the first joint point in the three-dimensional space, and the position information of the second joint point can also be the three-dimensional coordinates of the second joint point in the three-dimensional space. The bone length of the bone connecting the first joint point and the second joint point can be calculated based on the three-dimensional coordinates of the first joint point and the three-dimensional coordinates of the second joint point.

[0122] S14, the computing device uses the encoder of the first encoding-decoding network structure to encode the skeleton tree information and bone length information of the human body into a length latent variable.

[0123] The neural network model may be a graph convolutional neural network model. The neural network model may include a first encoding-decoding network and a second encoding-decoding network structure. Each encoding-decoding network may include an encoder using convolutional propagation and a decoder using deconvolutional propagation.

[0124] In one possible implementation, a computing device obtains the skeleton tree information and bone length information of each human body. During each training process, the skeleton tree information and bone length information of a single human body are input into the encoder in the first encoding-decoding network. The encoder in the first encoding-decoding network jointly encodes the skeleton tree information and bone length information of the human body through convolution operations to generate a length latent variable.

[0125] Among them, the encoder may include a convolution operation expression, and the skeleton tree information and the bone length information of the human body are convolved through the convolution operation expression and then jointly encoded to generate a length latent variable.

[0126] In a possible implementation, the length latent variable is input into the decoder of the first encoding-decoding network, and the decoder performs a deconvolution operation to obtain decoded skeleton tree information and decoded bone length information. The computing device updates the convolution operation expression in the encoder and the deconvolution operation expression in the decoder in the first encoding-decoding network with the goal that the error between the decoded skeleton tree information and the skeleton tree information of the human body is less than a first specified threshold, and the error between the decoded bone length information and the bone length information of the human body is less than a second threshold.

[0127] For example, Figure 6 Schematic diagram of a first encoding-decoding network training update involved in an embodiment of the present application. Figure 6As shown, for the first encoding-decoding network, after the skeleton tree information S of the human body and the bone length information A of the human body are input into the encoder in the first encoding-decoding network, after the convolution operation of the convolution operation expression of the encoder, the skeleton tree information S of the human body and the bone length information A of the human body can be jointly encoded as a length latent variable, and the length latent variable can be output by the deconvolution operation of the deconvolution operation expression in the decoder of the first encoding-decoding network. The skeleton tree information S of the human body and the decoded skeleton tree information S' are calculated by loss function to obtain a first loss function, and the bone length information A of the human body and the decoded bone length information A' are calculated by loss function to obtain a second loss function. The convolution operation expression and the deconvolution operation expression are updated based on the first loss function and the second loss function. It should be understood that the first loss function is a functional expression of the error between the skeleton tree information S of the human body and the decoded skeleton tree information S', and the second loss function is a functional expression of the error between the bone length information A of the human body and the decoded bone length information A'. The loss function that appears later in the embodiment of the present application also has a similar meaning.

[0128] S15, the computing device uses the encoder of the second encoding-decoding network structure to encode the human body's motion data into motion latent variables.

[0129] Among them, the second encoding-decoding network structure includes an encoder and a decoder. The computing device obtains the motion data of each human body performing a specified action. In each training process, the motion data of a single human body (the skeleton tree information of the human body input for this training and the bone length information of the human body belong to the same human body) is input into the encoder in the second encoding-decoding network. The encoder in the second encoding-decoding network encodes the bone motion data of the human body through convolution operations to generate motion latent variables.

[0130] S16, the computing device inputs the length latent variable and the action latent variable into the decoder of the second encoding-decoding network structure to obtain decoded action data.

[0131] In a possible implementation, the length latent variable and the action latent variable are input into a decoder of the second encoding-decoding network, and the decoder performs a deconvolution operation to obtain decoded action data.

[0132] S17, the computing device trains the neural network model with the goal of making the error between the decoded motion data and the human body's motion data less than a preset threshold.

[0133] In one possible implementation, the computing device updates the convolution operation expression in the encoder and the deconvolution operation expression in the decoder in the second encoding-decoding network with the goal of making the error of the decoded motion data less than a preset threshold.

[0134] For example, Figure 7 Schematic diagram of a second encoding-decoding network training update involved in an embodiment of the present application. Figure 7 As shown, for the second encoding-decoding network, after the human body's motion data D is input into the encoder in the second encoding-decoding network, the human body's motion data D can be encoded into motion latent variables through the convolution operation of the encoder's convolution operation expression, and the motion latent variables and the received length latent variables are input into the decoder of the second encoding-decoding network together, and the deconvolution operation expression is used to perform the deconvolution operation, and the decoded motion data D' can be output. The human body's motion data D and the decoded motion data D' are subjected to the loss function calculation to obtain the third loss function. The convolution operation expression and the deconvolution operation expression in the second encoding-decoding network are updated based on the third loss function.

[0135] Since bone length information is added during the training process of the above-mentioned neural network model, that is, the bone length information is used as an additional input to describe the action semantic information, even when there is a large difference in the bone length between the source object and the character to be driven, the action data after action redirection output by the neural network model can still ensure a high degree of matching with the bone model of the character to be driven, so that the character to be driven can restore the action of the source object well.

[0136] Through the above S11 to S17, the neural network model is trained. After the model training is completed, the computing device can store the trained neural network model for subsequent model application. The neural network model can be used on the computing device after being trained on other devices, or it can be used on the computing device after being trained. The device that performs the training process of the neural network model is not limited here.

[0137] In one possible implementation, the action data of the source object, the skeleton tree information of the character to be driven, and the bone length information of the character to be driven are input into a trained neural network model to obtain the action data of the character to be driven that is output by the neural network model.

[0138] For example, the action data of the character to be driven outputted by the neural network model can be used in the live broadcast of the digital human, that is, mapping the actions of the real person performance on the character to be driven, and broadcasting the corresponding actions by showing the character to be driven. Alternatively, driving the character to be driven to perform actions can also be used in the process of game modeling, by collecting real person actions, and constructing a model animation of the character to be driven performing actions displayed in the game.

[0139] For example, Figure 8 1 is a flow chart of an action redirection process involved in an embodiment of the present application. Figure 8 As shown, in the model training stage, based on the optical motion capture device, the inertial motion capture device and the visual motion capture device, the motion data is collected when the human body (motion capture object) performs the prescribed action. Among them, the prescribed action can be an action in the prescribed action library (such as walking, running, waving, putting hands together, etc.). After sending the collected action data to the computing device, the computing device can perform data fusion on the action data collected by multiple motion capture methods to obtain the fused action data. The computing device can evaluate the action standard degree of the fused action data, and if the standard degree does not meet the requirements, the motion capture object can be prompted to adjust the posture of performing the prescribed action, and the part of the action data in which the error with the standard action data exceeds the specified threshold is fed back, and the motion capture object is guided to adjust to the standard action indicated by the standard action data. Among them, the collection process can be to use AI visual motion capture technology to evaluate the standard degree of the action, feedback the standard degree to the user, and guide the motion capture object to perform standard actions. If the standard degree meets the requirements, the relatively accurate human bone length information and human skeleton tree information can be calculated through the fused motion data, and the fused motion data, human bone length information and human skeleton tree information are input into the neural network model, with the goal of the error between the motion data output by the neural network model and the motion data of the human body being less than a preset threshold, the neural network model is trained. Each time the neural network model is trained, data corresponding to a human body is input into the neural network model, and the training of the neural network model is completed through multiple trainings, that is, data corresponding to multiple human bodies.

[0140] During the model application stage, the computing device can obtain the action data of the source object's execution action. The computing device also needs to obtain the bone length information and skeleton tree information of the character to be driven, and input the action data of the source object, the bone length information and the skeleton tree information of the character to be driven into the trained neural network model to obtain the action data of the character to be driven, thereby realizing the personalized redirection process and driving the character to be driven to perform the action.

[0141] To summarize, the motion redirection method provided in the present application inputs the bone length information as an additional supervisory signal into the neural network model, so that the neural network model adjusts the motion data of the source object into motion data that matches the skeleton of the character to be driven based on the bone length information. The adjusted motion data corresponds to a higher degree of motion restoration, thereby ensuring the effect of motion redirection.

[0142] Fig. 9 FIG. 1 shows an architecture diagram of an action redirection system provided by an exemplary embodiment of the present application. Fig. 9 As shown, the motion redirection system includes various motion capture devices and computing devices, wherein the various motion capture devices may include optical motion capture devices, visual motion capture devices, and inertial motion capture devices. The computing device may be used to obtain motion data collected by various motion capture devices and process the motion data. The motion redirection system may include an optical motion capture module 310, a video motion capture module 320, an inertial motion capture module 330, a multi-sensor fusion module 340, and a personalized redirection module 350.

[0143] The optical motion capture module 310 may include a process of collecting motion data of a motion capture object in reality through an infrared sensor, a process of sensor calibration, a process of feature point detection, and a process of motion reconstruction.

[0144] The video motion capture module 320 may include a process of collecting motion data of a motion capture object in reality through a visual sensor, a process of capturing facial expressions of a target object in reality, a process of capturing gestures of a target object in reality, and a process of redirecting the captured motion. Motion redirection may be a process of mapping the motion performed by a motion capture object in reality to a virtual object, or a process of transferring the motion performed by a virtual object to another virtual object.

[0145] The inertial motion capture module 330 may include a process of collecting motion data of a motion capture object in reality through an inertial navigation sensor, a process of collecting motion data of a motion capture object through a gyroscope, a process of integrating and calculating the angular velocity collected by the inertial sensor, and a process of calibrating the posture of the motion capture object according to the data collected by the inertial sensor.

[0146] The multi-sensor fusion module 340 can be used to receive the motion data of the motion capture object collected by various sensors, and perform data fusion on the motion data of the motion capture object collected by various sensors. In combination with the accuracy of the motion data collected by different sensors in different collection environments, the weight of the motion data collected by different sensors during data fusion can be adjusted. For example, if the accuracy of the motion data collected by the inertial sensor is better than the motion data collected by other sensors in the current collection environment, the weight of the motion data collected by the current inertial sensor during data fusion can be set to be higher than the weight corresponding to other sensors. After fusing the motion data, calibrated motion data can be obtained, and posture estimation can be performed according to the calibrated motion data. Then, the motion data of the standard motion corresponding to the motion can be identified and matched with the calibrated motion data to determine whether the error between the motion performed by the motion capture object and the standard motion meets the requirements. If it does not meet the requirements, the posture needs to be fed back to the motion capture object, that is, the motion capture object needs to be prompted to adjust the motion posture to re-execute the target motion. After re-acquiring each motion data collected by the multi-sensor and fusing the motion data, bone estimation can be performed according to the calibrated motion data, that is, calculating the bone length.

[0147] The personalized redirection module 350 can be used to implement the process of action redirection in combination with the bone length. The computing device can use the fused action data generated by the acquired source object performing the action, the bone length information of the character to be driven, and the skeleton tree information of the character to be driven as the input of the neural network model. After the calculation and processing of the neural network model, the neural network model can output the action data of the character to be driven to perform the action, thereby completing the process of action redirection.

[0148] The above mainly introduces the scheme of the embodiment of the present application from the perspective of the method. It can be understood that in order to realize the above functions, the action redirection device includes at least one of the hardware structure and software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0149] The embodiment of the present application can divide the action redirection device into functional units according to the above method example. For example, each functional unit can be divided according to each function, or two or more functions can be integrated into one processing unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units. It should be noted that the division of units in the embodiment of the present application is schematic and is only a logical functional division. There may be other division methods in actual implementation.

[0150] For example, Fig.10 The schematic diagram of the structure of an action redirection device 500 provided by an exemplary embodiment of the present application is shown. The action redirection device 500 is applied to a computing device, or the action redirection device 500 can be a computing device. The action redirection device 500 includes:

[0151] An acquisition module 510 is used to acquire action data generated by the source object performing an action, wherein the type of the source object includes a real person and / or a digital person, and the action is a body movement of the source object;

[0152] The acquisition module 510 is further used to acquire the skeleton model of the character to be driven;

[0153] A determination module 520 is used to determine the skeleton tree information and the bone length information of the character to be driven according to the skeleton model of the character to be driven, wherein the skeleton tree information of the character to be driven is used to describe the skeleton tree structure of the character to be driven, and the bone length information of the character to be driven includes the length of the skeleton of the character to be driven;

[0154] The processing module 530 is used to obtain the action data of the character to be driven to perform the action by inputting the action data of the source object, the skeleton tree information of the character to be driven, and the bone length information of the character to be driven into a neural network model, wherein the neural network model is used to adjust the positions of some or all of the bone joint points in the action data of the source object to positions that match the bone model of the character to be driven based on the bone length information of the character to be driven.

[0155] In a possible implementation, the neural network model includes a first encoding-decoding network structure and a second encoding-decoding network structure, and the device further includes:

[0156] A collection module, used to obtain motion data generated by a human body performing a prescribed motion, wherein the motion data of the human body includes position information of skeletal joints of the human body;

[0157] A training module is used to determine the skeleton tree information of the human body according to the motion data of the human body, and the skeleton tree information of the human body is used to describe the skeleton tree structure of the human body; calculate the bone length information of the human body according to the position information of the skeletal joints of the human body, and the bone length information includes the length of the bones of the human body; use the encoder of the first encoding-decoding network structure to encode the skeleton tree information and the bone length information of the human body into a length latent variable; use the encoder of the second encoding-decoding network structure to encode the motion data of the human body into a motion latent variable; input the length latent variable and the motion latent variable into the decoder of the second encoding-decoding network structure to obtain the decoded motion data; and train the neural network model with the goal of the error between the decoded motion data and the motion data of the human body being less than a preset threshold.

[0158] In a possible implementation, the motion data of the human body is fused motion data, and the acquisition module is further used to obtain multiple motion data of the human body performing the specified action collected through multiple motion capture methods; and fuse the multiple motion data to obtain the fused motion data.

[0159] In a possible implementation, the multiple motion capture modes include at least two of inertial motion capture, visual motion capture, and optical motion capture.

[0160] In a possible implementation manner, the device further includes:

[0161] An indication module is used to obtain original motion data of a human body before obtaining motion data generated by the human body performing a prescribed action, wherein the original motion data includes position information of skeletal joints when the human body last performed the prescribed action; determine an error between the original motion data of the human body and standard motion data of the prescribed action, wherein the error is used to indicate the standard degree of execution of the prescribed action by the human body; and output indication information when the error is greater than a specified threshold value, wherein the indication information is used to guide the human body to adjust its limb posture when performing the prescribed action.

[0162] For the detailed description of the above optional methods, please refer to the above method embodiments, which will not be repeated here. In addition, the explanation of any action redirection device provided above and the description of the beneficial effects can refer to the above corresponding method embodiments, which will not be repeated here.

[0163] Among them, the acquisition module 510, the determination module 520 and the processing module 530 can all be implemented by software, or can be implemented by hardware. Exemplarily, the following takes the acquisition module 510 as an example to introduce the implementation of the acquisition module 510. Similarly, the implementation of the determination module 520 and the processing module 530 can refer to the implementation of the acquisition module 510.

[0164] As an example of a software functional unit, the acquisition module 510 may include code running on a computing instance. Among them, the computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the above-mentioned computing instance may be one or more. For example, the acquisition module 510 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region (region) or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including a data center or multiple data centers with similar geographical locations. Among them, usually a region may include multiple AZs.

[0165] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set up in each VPC to achieve interconnection between VPCs through the communication gateway.

[0166] As an example of a hardware functional unit, the acquisition module 510 may include at least one computing device, such as a server, etc. Alternatively, the acquisition module 510 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0167] The multiple computing devices included in the acquisition module 510 can be distributed in the same region or in different regions. The multiple computing devices included in the acquisition module 510 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the acquisition module 510 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0168] It should be noted that, in other embodiments, the acquisition module 510 can be used to execute any step in the action redirection method, the determination module 520 can be used to execute any step in the action redirection method, and the processing module 530 can be used to execute any step in the action redirection method. The steps that the acquisition module 510, the determination module 520, and the processing module 530 are responsible for implementing can be specified as needed. The acquisition module 510, the determination module 520, and the processing module 530 respectively implement different steps in the action redirection method to achieve all the functions of the action redirection device. The present application also provides a computing device 100. Fig.11 As shown, the computing device 100 includes: a bus 102, a processor 104, a memory 106, and a communication interface 108. The processor 104, the memory 106, and the communication interface 108 communicate with each other through the bus 102. The computing device 100 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in the computing device 100.

[0169] The bus 102 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.11 The bus 104 is represented by only one line, but does not mean that there is only one bus or one type of bus. The bus 104 may include a path for transmitting information between various components of the computing device 100 (eg, the memory 106, the processor 104, and the communication interface 108).

[0170] The processor 104 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0171] The memory 106 may include a volatile memory, such as a random access memory (RAM). The processor 104 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0172] The memory 106 stores executable program codes, and the processor 104 executes the executable program codes to respectively implement the functions of the acquisition module 510, the determination module 520 and the processing module 530, thereby implementing the action redirection method. That is, the memory 106 stores instructions for executing the action redirection method.

[0173] Alternatively, the memory 106 stores executable codes, and the processor 104 executes the executable codes to respectively implement the functions of the aforementioned action redirection device, thereby implementing the action redirection method. That is, the memory 106 stores instructions for executing the action redirection method.

[0174] The communication interface 108 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 100 and other devices or a communication network.

[0175] The embodiment of the present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0176] like Fig.12 As shown, the computing device cluster includes at least one computing device 100. The memory 106 in one or more computing devices 100 in the computing device cluster may store the same instructions for executing the action redirection method.

[0177] In some possible implementations, the memory 106 of one or more computing devices 100 in the computing device cluster may also store partial instructions for executing the action redirection method. In other words, the combination of one or more computing devices 100 may jointly execute instructions for executing the action redirection method.

[0178] It should be noted that the memory 106 in different computing devices 100 in the computing device cluster may store different instructions, which are respectively used to execute part of the functions of the action redirection device. That is, the instructions stored in the memory 106 in different computing devices 100 may implement the functions of one or more modules among the acquisition module 510, the determination module 520 and the processing module 530.

[0179] In some possible implementations, one or more computing devices in the computing device cluster may be connected via a network, which may be a wide area network or a local area network. Fig.13 A possible implementation is shown. Fig.13 As shown, two computing devices 100A and 100B are connected via a network. Specifically, the network is connected via a communication interface in each computing device. In this type of possible implementation, the memory 106 in the computing device 100A stores instructions for executing the functions of the acquisition module 510. At the same time, the memory 106 in the computing device 100B stores instructions for executing the functions of the determination module 520 and the processing module 530.

[0180] Fig.13 The connection method between the computing device clusters shown may be based on the consideration that the action redirection method provided in the present application requires a large amount of storage and calculation of data, and therefore it is considered that the functions implemented by the determination module 520 and the processing module 530 are handed over to the computing device 100B for execution.

[0181] It should be understood that Fig.13 The functions of the computing device 100A shown in FIG. 1 may also be completed by multiple computing devices 100. Similarly, the functions of the computing device 100B may also be completed by multiple computing devices 100.

[0182] The present application embodiment also provides another computing device cluster. The connection relationship between the computing devices in the computing device cluster can be similar to that of Fig.12 and Fig.13 The connection mode of the computing device cluster is different in that the memory 106 in one or more computing devices 100 in the computing device cluster may store the same instructions for executing the action redirection method.

[0183] In some possible implementations, the memory 106 of one or more computing devices 100 in the computing device cluster may also store partial instructions for executing the action redirection method. In other words, the combination of one or more computing devices 100 may jointly execute instructions for executing the action redirection method.

[0184] It should be noted that the memory 106 in different computing devices 100 in the computing device cluster may store different instructions for executing part of the functions of the action redirection system. That is, the instructions stored in the memory 106 in different computing devices 100 may implement the functions of one or more devices in the action redirection device.

[0185] The embodiment of the present application also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the action redirection method.

[0186] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct a computing device to perform an action redirection method, or instructs a computing device to perform an action redirection method.

[0187] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. An action redirection method, characterized in that: The method is used to drive a character to be driven according to a source object, and the method comprises: Acquiring action data generated by the source object performing an action, wherein the type of the source object includes a real person and / or a digital person, and the action is a body movement of the source object; Obtaining a skeleton model of the character to be driven; Determine skeleton tree information and skeleton length information of the character to be driven according to the skeleton model of the character to be driven, wherein the skeleton tree information of the character to be driven is used to describe the skeleton tree structure of the character to be driven, and the skeleton length information of the character to be driven includes the length of the skeleton of the character to be driven; The action data of the character to be driven performing the action is obtained by inputting the action data of the source object, the skeleton tree information of the character to be driven, and the bone length information of the character to be driven into a neural network model, wherein the neural network model is used to adjust the positions of some or all of the bone joint points in the action data of the source object to positions matching the bone model of the character to be driven based on the bone length information of the character to be driven.

2. The method according to claim 1, characterized in that The neural network model includes a first encoding-decoding network structure and a second encoding-decoding network structure, and the method further includes: Acquiring motion data generated by a human body performing a prescribed motion, wherein the motion data of the human body includes position information of skeletal joints of the human body; Determine skeleton tree information of the human body according to the motion data of the human body, where the skeleton tree information of the human body is used to describe the skeleton tree structure of the human body; Calculating the bone length information of the human body according to the position information of the bone joint points of the human body, wherein the bone length information includes the length of the bones of the human body; Using the encoder of the first encoding-decoding network structure, the skeleton tree information and the bone length information of the human body are jointly encoded into a length latent variable; Using the encoder of the second encoding-decoding network structure, encoding the human body motion data into motion latent variables; Inputting the length latent variable and the action latent variable into the decoder of the second encoding-decoding network structure to obtain decoded action data; The neural network model is trained with the goal that the error between the decoded motion data and the motion data of the human body is less than a preset threshold.

3. The method according to claim 2, characterized in that The human body motion data is fused motion data, and the acquisition of motion data generated by the human body performing a prescribed motion includes: Acquire multiple sets of motion data of the human body performing the prescribed motion collected by multiple motion capture methods; The multiple pieces of motion data are fused to obtain the fused motion data.

4. The method according to claim 3, characterized in that The multiple motion capture modes include at least two of inertial motion capture, visual motion capture, and optical motion capture.

5. The method according to any one of claims 2 to 4, characterized in that Before obtaining the motion data generated by the human body performing the prescribed motion, the method further includes: Acquiring original motion data of the human body, wherein the original motion data includes position information of skeletal joints when the human body last performed the prescribed motion; Determine an error between the original motion data of the human body and the standard motion data of the prescribed motion, wherein the error is used to indicate a standard degree to which the human body performs the prescribed motion; When the error is greater than a specified threshold, an instruction message is output, wherein the instruction message is used to guide the human body to adjust the body posture when performing the prescribed action.

6. An action redirection device, characterized in that: The device is used to drive the character to be driven according to the source object, and the device comprises: An acquisition module, used for acquiring action data generated by the source object performing an action, wherein the type of the source object includes a real person and / or a digital person, and the action is a body movement of the source object; The acquisition module is further used to acquire the skeleton model of the character to be driven; A determination module, used to determine the skeleton tree information and the bone length information of the character to be driven according to the skeleton model of the character to be driven, wherein the skeleton tree information of the character to be driven is used to describe the skeleton tree structure of the character to be driven, and the bone length information of the character to be driven includes the length of the skeleton of the character to be driven; A processing module is used to obtain action data of the character to be driven performing the action by inputting the action data of the source object, the skeleton tree information of the character to be driven, and the bone length information of the character to be driven into a neural network model, wherein the neural network model is used to adjust the positions of some or all of the bone joints in the action data of the source object to positions matching the bone model of the character to be driven based on the bone length information of the character to be driven.

7. The device according to claim 6, characterized in that The neural network model includes a first encoding-decoding network structure and a second encoding-decoding network structure, and the device also includes: A collection module, used to obtain motion data generated by a human body performing a prescribed motion, wherein the motion data of the human body includes position information of skeletal joints of the human body; A training module is used to determine the skeleton tree information of the human body according to the motion data of the human body, and the skeleton tree information of the human body is used to describe the skeleton tree structure of the human body; calculate the bone length information of the human body according to the position information of the skeletal joints of the human body, and the bone length information includes the length of the bones of the human body; use the encoder of the first encoding-decoding network structure to encode the skeleton tree information and the bone length information of the human body into a length latent variable; use the encoder of the second encoding-decoding network structure to encode the motion data of the human body into a motion latent variable; input the length latent variable and the motion latent variable into the decoder of the second encoding-decoding network structure to obtain the decoded motion data; and train the neural network model with the goal of the error between the decoded motion data and the motion data of the human body being less than a preset threshold.

8. The device according to claim 7, characterized in that The motion data of the human body is fused motion data. The acquisition module is further used to obtain multiple motion data of the human body performing the specified motion acquired through multiple motion capture methods; and fuse the multiple motion data to obtain the fused motion data.

9. The device according to claim 8, characterized in that The multiple motion capture modes include at least two of inertial motion capture, visual motion capture, and optical motion capture.

10. The device according to any one of claims 7 to 9, characterized in that The device also includes: An indication module is used to obtain original motion data of a human body before obtaining motion data generated by the human body performing a prescribed action, wherein the original motion data includes position information of skeletal joints when the human body last performed the prescribed action; determine an error between the original motion data of the human body and standard motion data of the prescribed action, wherein the error is used to indicate the standard degree of execution of the prescribed action by the human body; and output indication information when the error is greater than a specified threshold value, wherein the indication information is used to guide the human body to adjust its limb posture when performing the prescribed action.

11. A computing device cluster, characterized in that: It includes at least one computing device, each computing device includes: a processor and a memory, the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the action redirection method as described in any one of claims 1 to 5.

12. A computer-readable storage medium, characterized in that: The method comprises computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the action redirection method according to any one of claims 1 to 5.

13. A computer program product, characterized in that The computer program product includes instructions, and when the instructions are executed by a computing device cluster, the computing device cluster performs the action redirection method according to any one of claims 1 to 5.

Images