Action correction method and related equipment

Abstract

The invention provides an action correction method, and the method comprises the steps: obtaining first action data which is action data of a virtual image; determining target action semantics, wherein the target action semantics indicate the meaning of the action expression of the expected virtual image; determining a constraint condition corresponding to the target action semantics, wherein the constraint condition indicates a constraint on action data related to the target action semantics; and correcting the first action data according to a constraint condition corresponding to the target action semantics, so that the action of the virtual image expresses the target action semantics. According to the method, the situation that the actual action semantics of the virtual image is inconsistent with the meaning expressed by the action of the expected virtual image can be avoided, the accuracy of action generation of the virtual image is improved, and the virtual image is more vivid and vivid.

Description

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on October 17, 2023, with application number 202311350312.0 and invention name “A method, device and system for generating action”, the entire contents of which are incorporated by reference in this application. Technical Field

[0002] The present application relates to the field of artificial intelligence (AI) technology, and in particular to an action correction method, apparatus, computing device cluster, computer-readable storage medium, and computer program product. Background Art

[0003] Avatar refers to a graphical representation of an object constructed through artificial intelligence (AI) technology. Avatars can be used in different scenarios. For example, in live broadcast scenarios, online conference scenarios, or news broadcast scenarios, avatars can be digital people corresponding to real people (also called virtual digital people).

[0004] Virtual images such as digital humans can usually be driven. Specifically, motion capture technology can be used to collect real-life motion data, and the real-life motion data can be used to drive the virtual image, so as to redirect the real-life motion to the digital human motion, so that the virtual image can provide vivid and realistic motion effects similar to real-life motion.

[0005] However, due to the difference in body shape between the performer and the model to be driven, as well as the insufficient accuracy of the collected motion data, the movements of the virtual image redirected using the above motion data are difficult to accurately express the motion semantics of the real person, and the effect is not good. Summary of the invention

[0006] The present application provides an action correction method, which can correct the action data of a virtual image so that the action of the virtual image can express the target action semantics. The present application also provides an action correction device, a computing device cluster, a computer-readable storage medium, and a computer program product corresponding to the method.

[0007] In the first aspect, the present application provides a motion correction method, which can be executed by a motion correction device, which can be a software device, which can be deployed in a computing device cluster, and the computing device cluster executes the program code of the software device, thereby executing the motion correction method of the present application. In some examples, the motion correction device can be a hardware device, for example, the motion correction device can be a computing device cluster with motion correction functions such as digital human motion repair, and the above hardware device executes the motion correction method of the present application when it is running.

[0008] Specifically, the action correction device obtains first action data, which is the action data of the virtual image. The virtual image includes at least one body part. The action data of the virtual image includes kinematic parameters of at least one body part. The action correction device determines the target action semantics, wherein the target action semantics indicates the meaning of the action expression of the desired virtual image. Then, the action correction device determines the constraint conditions corresponding to the target action semantics, wherein the constraint conditions indicate the constraints on the action data related to the target action semantics. The action data related to the target action semantics includes kinematic parameters of at least one body part. The constraints include the values ​​of the kinematic parameters and / or the value ranges of the kinematic parameters. Then, the action correction device corrects the first action data according to the constraint conditions corresponding to the target action semantics so that the action of the virtual image expresses the target action semantics.

[0009] In this method, by determining the target action semantics, the constraint conditions corresponding to the target action semantics can be further determined, and the action data of the virtual image can be corrected using the constraint conditions so that the action data of the corrected virtual image can express the target action semantics, thereby avoiding the situation where the actual action semantics of the virtual image is inconsistent with the meaning of the expected action expression of the virtual image, improving the accuracy of virtual image action generation, and making the virtual image more realistic and vivid.

[0010] In some possible implementations, the first action data is obtained by driving the virtual image with the second action data, and the second action data is obtained by collecting the action of the first object. The action correction device can determine the action semantics represented by the action of the first object as the target action semantics based on the action of the first object.

[0011] In this method, the virtual image can be driven by the action of the first object. By taking the action semantics of the action representation of the first object as the target action semantics, the action semantics of the action representation of the modified virtual image are made the same as the action semantics of the action representation of the first object. In this way, when there is a large difference in body size between the first object and the virtual image, the consistency of the action semantics is ensured.

[0012] In some possible implementations, the motion correction device may obtain video data of the motion of the first object, perform motion semantic recognition on the video data, and determine the target motion semantics. During the motion of the first object, the method collects video data, and uses the video data to determine the target motion semantics, thereby improving the accuracy of the target motion semantics.

[0013] In some possible implementations, the action correction device may query an action semantics constraint library based on the target action semantics, wherein the action semantics constraint library includes a plurality of action semantics constraints, and determine the constraints corresponding to the target action semantics from the plurality of action semantics constraints.

[0014] The method provides an action semantics constraint library. By taking the target action semantics as a query condition, it is possible to search among multiple action semantics constraints in the action semantics constraint library, thereby determining the constraint corresponding to the target action semantics, thereby facilitating the management of the action semantics constraint conditions.

[0015] In some possible implementations, the action semantics constraint library can be constructed in the following manner: receiving third action data input by a user and first action semantics matching the third action data, wherein the third action data can be obtained by collecting the action of a second object, and then generating constraints corresponding to the first action semantics based on the third action data.

[0016] In this method, users can upload the collected action data by themselves. By analyzing the action data uploaded by users, constraints can be automatically generated and matched with the action semantics input by users, without the need for users to manually extract and configure constraints. By providing users with an open action semantic constraint library construction interface, users can add constraints corresponding to the action semantics that need to be paid attention to in the action semantic constraint library according to actual needs (such as actual usage scenarios).

[0017] In some possible implementations, action data of key body parts can be determined based on the key body parts marked in the third action data, where the key body parts include at least one body part of the second object, and then constraints corresponding to the first action semantics are generated based on the action data of the key body parts.

[0018] In this method, the user can mark key body parts in the third action data to indicate the body parts that need to be paid attention to in the first action semantics. By analyzing the action data of key body parts, constraints can be generated so that the constraints can more accurately determine whether the action data can express the first action semantics.

[0019] In some possible implementations, the action semantics constraint library can be constructed in the following manner: receiving the second action semantics input by the user and the constraints of the action data related to the second action semantics, and generating the constraint conditions corresponding to the second action semantics based on the constraints of the action data related to the second action semantics.

[0020] In this method, the user can directly configure the constraint conditions of the action semantics, so that the user can quickly and conveniently add constraint conditions to the action semantic constraint library without collecting third action data.

[0021] In some possible implementations, at least one body part may include a bone point, and the kinematic parameters of at least one body part may include at least one of the following: a rotation angle of the bone point, displacement change information of the bone point, and a distance between at least two bone points. In this method, by focusing on the kinematic parameters of the bone point, the collected motion data can more accurately reflect the motion of the object or the virtual image, and the constraint conditions can achieve a more accurate correction effect.

[0022] In some possible implementations, the action correction device can determine the expected position of the virtual image based on the constraints corresponding to the target action semantics, and determine the current position of the virtual image based on the first action data, and then correct the first action data based on the current position of the virtual image and the expected position of the virtual image.

[0023] In this method, the idea of ​​inverse kinematics algorithm is adopted to determine the expected position through constraints, and then combined with the current position of the virtual image, the motion data of the virtual image is corrected to improve the motion correction effect.

[0024] In some possible implementations, the first action data and the second action data may be time series data, such as action data of a time period when the first object performs a certain action, or action data of a time period when the virtual image performs a certain action after the virtual image is driven.

[0025] In some possible implementations, the motion correction device may extract features from the video data to obtain motion features of the first object, and process the motion features of the first object using a motion semantics recognition model to obtain target motion semantics. The motion semantics recognition model may be a neural network model based on an encoder-decoder structure.

[0026] In some possible implementations, after the action correction device determines the target action semantics, the action correction device can judge the target action semantics, and based on the judgment result, determine the target action semantics that meet the set conditions, and then determine the constraint conditions corresponding to the target action semantics that meet the conditions.

[0027] In a second aspect, the present application provides an action correction device, the device comprising:

[0028] An acquisition module, configured to acquire first motion data, wherein the first motion data is motion data of a virtual image, the virtual image includes at least one body part, and the motion data of the virtual image includes kinematic parameters of the at least one body part;

[0029] A determination module, configured to determine a target action semantics, wherein the target action semantics indicates a desired meaning of the action expression of the avatar;

[0030] The determination module is further used to determine a constraint condition corresponding to the target action semantics, wherein the constraint condition indicates a constraint on action data related to the target action semantics, wherein the action data related to the target action semantics includes kinematic parameters of the at least one body part, and wherein the constraint includes a value of the kinematic parameter and / or a value range of the kinematic parameter;

[0031] The correction module is used to correct the first action data according to the constraint conditions corresponding to the target action semantics, so that the action of the virtual image expresses the target action semantics.

[0032] In some possible implementations, the first motion data is obtained by driving the virtual image with second motion data, the second motion data is obtained by collecting the motion of the first object, and the determining module is specifically configured to:

[0033] According to the action of the first object, the action semantics represented by the action of the first object is determined as the target action semantics.

[0034] In some possible implementations, the determining module is specifically configured to:

[0035] Acquire video data of an action of the first object;

[0036] Perform action semantic recognition on the video data to determine target action semantics.

[0037] In some possible implementations, the determining module is specifically configured to:

[0038] According to the target action semantics, querying an action semantics constraint library, wherein the action semantics constraint library includes a plurality of action semantics constraint conditions;

[0039] From the constraints of the multiple action semantics, determine the constraints corresponding to the target action semantics.

[0040] In some possible implementations, the action semantic constraint library is constructed in the following manner:

[0041] Receiving third action data input by a user and first action semantics matching the third action data, wherein the third action data is obtained by collecting an action of a second object;

[0042] Generate constraint conditions corresponding to the first action semantics according to the third action data.

[0043] In some possible implementations, generating, according to the third action data, a constraint condition corresponding to the semantics of the first action includes:

[0044] Determining, according to the key body parts marked in the third motion data, motion data of the key body parts, wherein the key body parts include at least one body part of the second object;

[0045] According to the action data of the key body part, a constraint condition corresponding to the first action semantics is generated.

[0046] In some possible implementations, the action semantic constraint library is constructed in the following manner:

[0047] Receiving a second action semantics input by a user and a constraint of action data related to the second action semantics;

[0048] According to the constraint of the action data related to the second action semantics, a constraint condition corresponding to the second action semantics is generated.

[0049] In some possible implementations, the at least one body part includes a skeleton point, and the kinematic parameter of the at least one body part includes at least one of the following:

[0050] The rotation angle of the skeleton point;

[0051] Displacement change information of the skeleton points;

[0052] The distance between at least two of the skeleton points.

[0053] In some possible implementations, the correction module is specifically used to:

[0054] determining a desired position of the virtual image according to the constraint conditions corresponding to the semantics of the target action, and determining a current position of the virtual image according to the first action data;

[0055] The first motion data is modified according to the current position of the avatar and the desired position of the avatar.

[0056] In a third aspect, the present application provides a computing device cluster. The computing device cluster includes at least one computing device, and the at least one computing device includes at least one processor and at least one memory. The at least one processor and the at least one memory communicate with each other. The at least one processor is used to execute instructions stored in the at least one memory, so that the computing device or the computing device cluster executes the action correction method described in the first aspect or any implementation of the first aspect.

[0057] In a fourth aspect, the present application provides a computer-readable storage medium, in which instructions are stored, and the instructions instruct a computing device or a computing device cluster to execute the action correction method described in the above-mentioned first aspect or any one of the implementations of the first aspect.

[0058] In a fifth aspect, the present application provides a computer program product comprising instructions, which, when executed on a computing device or a computing device cluster, enables the computing device or the computing device cluster to execute the action correction method described in the first aspect or any one of the implementations of the first aspect.

[0059] Based on the implementations provided in the above aspects, this application can also be further combined to provide more implementations. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the technical method of the embodiments of the present application, the drawings required for use in the embodiments are briefly introduced below.

[0061] Figure 1 A schematic diagram of the architecture of a motion correction device provided in an embodiment of the present application;

[0062] Figure 2 A flowchart of an action correction method provided in an embodiment of the present application;

[0063] Figure 3 A flowchart of another motion correction method provided in an embodiment of the present application;

[0064] Figure 4 A schematic diagram of the structure of a computing device provided in an embodiment of the present application;

[0065] Figure 5 A schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application;

[0066] Figure 6 A schematic diagram of the structure of another computing device cluster provided in an embodiment of the present application;

[0067] Figure 7 A schematic diagram of the structure of another computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0068] The terms "first" and "second" in the embodiments of the present application are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features.

[0069] First, some technical terms involved in this application are introduced.

[0070] A virtual image refers to a graphical representation of an object constructed through artificial intelligence (AI) technology. In different scenarios, the virtual image can be different. For example, in a live broadcast scenario, the object can be the anchor, and the virtual image can be the digital person corresponding to the anchor. For another example, in an online meeting scenario, the object can be a participant, and the virtual image can be the digital person corresponding to the participant. For another example, in a news broadcast scenario, the object can be a broadcaster, and the virtual image can be the digital person corresponding to the broadcaster.

[0071] Motion generation or motion retargeting refers to the use of motion data to drive a virtual image, migrating a motion data source (such as motion capture data) to different virtual character models, so that the driven virtual image has a motion representation consistent with the object (such as head motion representation or limb motion representation), thereby enabling the virtual image to provide vivid and realistic motion effects. Motion data can be obtained by collecting the motion of the object through motion capture technology. Commonly used motion capture technologies in the industry include mechanical motion capture technology, optical motion capture technology, inertial motion capture technology, video motion capture technology, etc. Among them, optical motion capture technology refers to pasting markers on specific parts of the object (such as the hips, elbows and other joints of the human body), using optical motion capture cameras to detect markers in real time from different angles, and transmitting the detection data to the data processing equipment in real time. The data processing equipment calculates the specific spatial coordinates, direction, motion trajectory and other motion data of the marker based on the detection data. Video motion capture technology refers to the use of multiple cameras to achieve multi-angle shooting of the object to obtain motion data.

[0072] However, lightweight motion capture technology (such as video motion capture technology) is difficult to achieve accurate collection of motion data, and the collected motion data may have a lot of jitter. In optical motion capture technology, due to the use of optical motion capture cameras, relatively accurate motion data of the object can be captured, but if there is a large size difference between the object and the virtual image, after the virtual image is driven by the object's motion data, it is difficult for the virtual image to accurately express the object's motion semantics. For example, when the object's action is to make a heart shape with both hands in front of the chest, the action of the virtual image may be to make a circle with both hands in front of the chest, or to cross the arms in front of the chest. The object's action is difficult to accurately transmit to the virtual image.

[0073] To address the above issues, the industry can usually use loss estimation technology to correct motion data. Specifically, for some specific scenarios, supplementary data can be obtained and used to estimate losses. The loss value can be used to iteratively optimize the object's motion data to improve the quality of the object's motion data. For example, for a scene where the foot touches the ground, the supplementary data may include the speed change data of the object's foot, the position data of the object's body parts, etc. The supplementary data can be used to estimate the penetration loss and the foot sliding loss, and the object's motion data can be corrected.

[0074] However, the loss estimation technology corrects the motion data of the object based on the loss estimation, and it is difficult to ensure the correctness of the motion semantics. At the same time, the loss estimation is to calculate the gap between the ideal motion data of the virtual image (also called the ideal value) and the real motion data of the virtual image (also called the real value), and to reduce the gap by optimizing the motion data of the object, thereby improving the quality of the motion data. If the size difference between the object and the virtual image is large, the accuracy of the ideal value will be reduced, thus affecting the correction effect of the loss estimation technology.

[0075] In addition, the industry can also use motion editing technology to correct motion data. Specifically, digital content creation (DCC) tools usually provide motion editing functions, and technicians (such as animators) can use sub-functions such as motion curve editing in the motion editing function to manually adjust the jitter of motion data. However, motion editing technology requires technicians to manually correct motion data, and the correction efficiency is low. Moreover, motion editing technology is only applicable to offline scenarios and cannot be applied in online scenarios (such as scenarios of real-time driving of virtual images).

[0076] With the continuous development of machine learning technology, methods for correcting motion data using optimization models have emerged. The above-mentioned optimization model can be a neural network model trained using original motion data and corrected motion data. However, the optimization model requires a large amount of high-quality corrected motion data as training data, and the generalization of the optimization model is poor, making it difficult to apply motion data correction in different scenarios.

[0077] In view of this, the present application provides a motion correction method. The method can be executed by a motion correction device, wherein the motion correction device can be a software device, and the software device can be deployed in a computing device cluster, and the computing device cluster executes the program code of the software device, thereby executing the motion correction method of the present application. In some examples, the motion correction device can be a hardware device, for example, the motion correction device can be a computing device cluster with motion correction functions such as digital human motion repair, and the above hardware device executes the motion correction method of the present application when it is running.

[0078] Specifically, the action correction device obtains first action data, which is the action data of the virtual image. The virtual image includes at least one body part. The action data of the virtual image includes kinematic parameters of at least one body part. The action correction device determines the target action semantics, wherein the target action semantics indicates the meaning of the action expression of the desired virtual image. Then, the action correction device determines the constraint conditions corresponding to the target action semantics, wherein the constraint conditions indicate the constraints on the action data related to the target action semantics. The action data related to the target action semantics includes kinematic parameters of at least one body part. The constraints include the values ​​of the kinematic parameters and / or the value ranges of the kinematic parameters. Then, the action correction device corrects the first action data according to the constraint conditions corresponding to the target action semantics so that the action of the virtual image expresses the target action semantics.

[0079] In this method, by determining the target action semantics, the constraint conditions corresponding to the target action semantics can be further determined, and the action data of the virtual image can be corrected using the constraint conditions so that the action data of the corrected virtual image can express the target action semantics, thereby avoiding the situation where the actual action semantics of the virtual image is inconsistent with the meaning of the expected action expression of the virtual image, improving the accuracy of virtual image action generation, and making the virtual image more realistic and vivid.

[0080] The action correction method provided in the present application can be provided to users by being integrated into a business device. For example, the action correction device can be integrated into a business device, and the business device includes but is not limited to a live broadcast application or an online conference application. One integration method can be to integrate the action correction device into the above-mentioned application when developing a live broadcast application or an online conference application. In other words, the action correction device can be a functional module or sub-device of the business device. Another integration method can be to integrate it into the business device in the form of a plug-in. The plug-in can interact with the business device to implement the action correction function. The action correction method can also be provided to users by way of a cloud service. For example, a live broadcast application or an online conference application can call a cloud service to implement the action correction function.

[0081] In order to make the technical solution of the present application clearer and easier to understand, the device architecture of the present application is introduced below with reference to the accompanying drawings.

[0082] See also Figure 1 As shown in the structure diagram of the motion correction device, the motion correction device 10 is respectively connected to the motion acquisition device 20, the motion generation device 30 and the interaction device 40. The functions of each device will be introduced below.

[0083] The motion acquisition device 20 is used to acquire the motion of the object. Specifically, the motion acquisition device 20 includes a plurality of physical devices, and different physical devices can acquire the motion of the object in different ways. For example, the physical device may be a motion capture device, which can acquire the motion data of the object (such as the motion data of at least one body part) during the motion of the object. For another example, the physical device may be a single-lens reflex digital camera (SLR), an industrial camera or an ordinary camera, which can acquire video data during the motion of the object. In some possible implementations of the embodiments of the present application, the virtual image can be driven by the motion data of the first object, and therefore, the motion acquisition device 20 can acquire the motion of the first object and obtain the second motion data, so as to drive the virtual image using the second motion data.

[0084] The action generation device 30 is used to process the action data to realize the action generation of the virtual image. Specifically, the action generation device 30 includes a posture recognition module 301 and a model driving module 302. Among them, the posture recognition module 301 is used to perform posture recognition on the action data of the object. In some embodiments, the posture recognition module 301 can first perform data screening on the action data of the object, and realize posture recognition according to the posture recognition algorithm for the action data after data screening. The model driving module 302 is used to drive the virtual image. In some embodiments, the model driving module 302 can use a model driving algorithm (such as a redirection algorithm) to redirect the action data after posture recognition to the virtual image to realize action generation.

[0085] The action correction device 10 is used to correct the action of the virtual image. Specifically, the action correction device 10 includes an acquisition module 101, a determination module 102, a correction module 103 and an action semantic constraint library 104. Among them, the acquisition module 101 is used for the first action data. It can be understood that after the action generation of the virtual image is realized, the virtual image can perform the corresponding action. In the embodiment of the present application, the first action data can be the action data of the virtual image, wherein the virtual image includes at least one body part, and the action data of the virtual image includes the kinematic parameters of at least one body part.

[0086] The determination module 102 is used to determine the target action semantics. The target action semantics indicates the meaning of the action expression of the desired virtual image, for example, the target action semantics may be "making a heart shape with both hands on the head", "making a heart shape with one hand on the chest", etc. In some embodiments, the target action semantics may be the action semantics of the action representation of the first object. For example, the determination module 102 may obtain video data of the action of the first object from the action acquisition device 20, and determine the target action semantics by performing action semantic recognition on the video data.

[0087] The determination module 102 is also used to determine the constraint conditions corresponding to the target action semantics. The constraint conditions indicate the constraints on the action data related to the target action semantics, and the action data related to the target action semantics include the kinematic parameters of at least one body part, such as the kinematic parameters of a skeletal point, and the constraints include the kinematic values ​​and / or the value ranges of the kinematic parameters. In some embodiments, the determination module 102 can query the action semantics constraint library 104 according to the target action semantics, and determine the constraint conditions corresponding to the target action semantics from the constraint conditions of multiple action semantics in the action semantics constraint library 104.

[0088] The correction module 103 is used to correct the first action data according to the constraint conditions corresponding to the target action semantics. In some embodiments, the correction module 103 can determine at least one body part involved in the constraint conditions corresponding to the target action semantics, and determine the correction order of the at least one body part according to the body structure of the virtual image, and then correct the first action data according to the correction order so that the action of the virtual image expresses the target action semantics.

[0089] The interactive device 40 is used to render the virtual image and the scene so as to present the display video of the virtual image to the user. Specifically, the interactive device 40 includes a model rendering module 401 and a scene rendering module 402. Among them, the model rendering module 401 is used to render the virtual image, and the scene rendering module 402 is used to render the scene where the virtual image is located. For example, when the object is a broadcaster and the virtual image is a digital person corresponding to the broadcaster, the scene rendering module 402 can be used to render the broadcast station, the news broadcast background, etc. For another example, when the object is a participant and the virtual image is a digital person corresponding to the participant, the scene rendering module 402 can be used to render the conference room background. In this way, the visual display of the virtual image is achieved.

[0090] Furthermore, the embodiments of the present application support users to configure and expand the constraints in the action semantics constraint library 104. In some embodiments, the user can upload the third action data and the first action semantics that match the third action data, so that the constraints corresponding to the first action semantics are generated based on the third action data, and the automatic extraction of the action semantics and the automatic generation of the constraints are realized. In other embodiments, the user can upload the second action semantics and the constraints of the action data related to the second action semantics, so that the constraints corresponding to the second action semantics are generated based on the constraints of the action data related to the second action semantics.

[0091] It should be noted that, in some possible implementations, the action semantic constraint library 104 may not be deployed in the action correction device 10, that is, the action semantic constraint library 104 may be a database provided by an external device, including multiple action semantic constraint conditions, and the embodiments of the present application do not limit this.

[0092] based on Figure 1 The motion correction device 10 shown in the figure, the present application also provides a motion correction method. The motion correction method of the present application is introduced below in conjunction with an embodiment.

[0093] See also Figure 2 The flowchart of the action correction method shown in FIG. 1 includes the following steps:

[0094] S201: The motion correction device 10 obtains first motion data.

[0095] The first motion data is motion data of the virtual image. The embodiment of the present application does not limit the form of expression of the virtual image. For example, the virtual image can be a graphical representation of a real person, animal or other organism, or a graphical representation of a cartoon image.

[0096] The avatar includes at least one body part, and the motion data of the avatar includes kinematic parameters of the at least one body part. The kinematic parameters can be understood as parameters used to describe the motion state of at least one body part. In some embodiments, at least one body part may include a bone point, such as a top bone point located at the top of the head, a neck bone point located at the center of the neck, a left elbow bone point located at the elbow of the left arm, etc. The kinematic parameters of at least one body part may include at least one of the following: the rotation angle of the bone point, the displacement change information of the bone point, and the distance between at least two bone points.

[0097] In other words, during the movement of the virtual image, the motion correction device 10 can collect data on at least one body part of the virtual image, obtain kinematic parameters of at least one body part of the virtual image, and then obtain the first motion data.

[0098] In an embodiment of the present application, the first motion data can be obtained by driving the virtual image with the second motion data, and the second motion data can be obtained by collecting the motion of the first object. The first object can be an object with motion such as a real person. In other words, the first object can perform motion as a performer (for example, an action performance), and the motion of the first object can drive the virtual image, so that the virtual image performs the same or similar motion as the first object. For example, when the first object raises its right hand, the virtual image will also raise its right hand.

[0099] In specific implementation, the first action data can be obtained through the following steps:

[0100] A1: The motion collection device 20 collects the motion of the first object to obtain second motion data;

[0101] For example, the second motion data may be collected by a motion capture device in the motion acquisition device 20. In some possible implementations, a marker may be deployed in advance on at least one body part of the first object, for example, a marker may be deployed on at least one skeletal point of the first object, so that during the movement of the first object, the motion capture device may detect the marker and obtain the second motion data.

[0102] In some embodiments, the second motion data may include one or more of rotation angle data, displacement data, motion trajectory, motion speed, and motion acceleration. In addition, the second motion data may be time series data, such as motion data of a time period in which the first object performs a certain specific motion.

[0103] A2: The action acquisition device 20 sends the second action data to the action generation device 30;

[0104] A3: The action generating device 30 drives the virtual image according to the second action data to obtain the first action data;

[0105] A4: The motion generating device 30 sends the first motion data to the motion correcting device 10 .

[0106] After acquiring the second motion data, the motion acquisition device 20 may send the second motion data to the motion generation device 30. The motion generation device 30 may drive the virtual image according to the second motion data to generate the first motion data. Specifically, the motion generation device 30 may use a redirection algorithm to transfer the second motion data to the virtual image, for example, transferring the second motion data of the first bone point in the first object to the first bone point in the virtual image, thereby driving the virtual image to perform the corresponding action and realizing the motion generation.

[0107] It is understandable that, due to the possible size difference between the first object and the virtual image, or certain errors in the process of generating the action, it is difficult for the virtual image to completely and accurately perform the same action as the first object, that is, the first action data and the second action data are difficult to be completely identical. Therefore, in the embodiment of the present application, after driving the virtual image, the action generation device 30 can send the first action data to the action correction device 10 to perform subsequent action correction.

[0108] S202: The action correction device 10 determines the target action semantics.

[0109] The target action semantics may indicate the meaning of the action that the avatar is expected to express. For example, when the action is "making a heart shape with both hands above the head", the target action semantics may be "making a heart shape with both hands above the head". Similarly, the target action semantics may also be "making a heart shape with one hand in front of the chest", "congratulations with both hands", "giving a thumbs up with one hand", "giving a thumbs up with both hands", "applauding in front of the chest", etc.

[0110] In some possible implementations, the action correction device 10 can determine the action semantics of the action representation of the first object as the target action semantics based on the action of the first object. It can be understood that since the first action data can be obtained by driving the virtual image with the second action data, and the second action data can be obtained by collecting the action of the first object, it is expected that the meaning of the action expression of the virtual image can be the same as the action semantics of the action representation of the first object. For example, when the action semantics of the action representation of the first object is "making a heart shape with both hands on the head", the meaning of the action expression of the virtual image is also expected to be "making a heart shape with both hands on the head".

[0111] In a specific implementation, the action correction device 10 may obtain video data of the action of the first object, perform action semantic recognition on the video data, and determine the target action semantics.

[0112] The video data of the action of the first object refers to the video stream data collected during the process of the first object performing the action. In other words, during the process of the first object performing a certain action, the action acquisition device 20 can use equipment such as a motion capture device and a camera to simultaneously collect the second action data and the video data, the second action data is used to drive the virtual image, and the video data is used to perform action semantic recognition on the action of the first object.

[0113] The process of action semantic recognition is described below. Considering that the video data may include multiple actions of the first object, the action correction device 10 can extract parameters for each video frame of the video data, such as action parameter extraction and expression parameter extraction, and then use the action parameters and expression parameters to perform initial recognition to obtain a set of video frames belonging to the same action in the video data, and then perform action semantic recognition on the set of video frames belonging to the same action to improve the accuracy of action semantic recognition.

[0114] The action correction device 10 can perform action semantics recognition with the help of the action semantics recognition model. Specifically, the action correction device 10 can perform feature extraction on the video data to obtain the action features of the first object, and then the action correction device 10 can process the action features of the first object using the action semantics recognition model to obtain the target action semantics.

[0115] In some embodiments, the action semantics recognition model can be a neural network model based on an encoder-decoder structure. The action semantics recognition model can be obtained by training a neural network model using a vector expression constructed from action features and a vector expression constructed from action semantics matching the action features as training data.

[0116] At this time, the action feature of the first object can be a vector sequence composed of the action features of each video frame in the video data. For example, the action feature of each video frame can be the coordinates of at least one skeleton point of the first object, and the action feature of the first object can be a vector sequence composed of the coordinates of at least one skeleton point of the first object. The action feature of the first object is used as the input of the action semantics recognition model. The action semantics recognition model can perform action semantics recognition based on the action feature of the first object and output the word vector of the target action semantics. In this way, the action correction device 10 can determine the action semantics of the action representation of the first object.

[0117] S203: The action correction device 10 determines the constraint conditions corresponding to the target action semantics.

[0118] The constraint condition corresponding to the target action semantics indicates the constraint on the action data related to the target action semantics. The action data related to the target action semantics may include kinematic parameters of at least one body part, and the constraint may include the value of the kinematic parameter and / or the value range of the kinematic parameter.

[0119] That is to say, the constraint conditions corresponding to the target action semantics may be the value of the kinematic parameters of at least one body part and / or the value range of the kinematic parameters. When the action data satisfies the value and / or value range, it can be understood that the action data can express the target action semantics. Otherwise, the action data cannot express the target action semantics.

[0120] In some possible implementations, the action correction device 10 can query the action semantics constraint library according to the target action semantics, wherein the action semantics constraint library includes multiple action semantics constraint conditions, and the action correction device 10 can determine the constraint conditions corresponding to the target action semantics from the multiple action semantics constraint conditions. In other words, the action correction device 10 can use the target action semantics as a query condition, search in the action semantics constraint library, and obtain the constraint conditions corresponding to the target action semantics.

[0121] In some embodiments, considering that the user may not perform action correction on all actions, after the action correction device 10 determines the target action semantics, the action correction device 10 may judge the target action semantics and determine the target action semantics that meet the set conditions based on the judgment result.

[0122] For example, the action correction device 10 can query the action semantics constraint library according to the target action semantics. If no matching constraint conditions are found, it indicates that the target action semantics does not need to be corrected in the current scene, and the target action semantics does not meet the set conditions. For another example, the action correction device 10 can perform key action semantics recognition on the target action semantics. If the recognition result is non-key action semantics, the target action semantics does not meet the set conditions.

[0123] For target action semantics that do not meet the set conditions, the action correction device 10 does not need to perform subsequent action correction, so the interaction device 40 can directly render the virtual image and scene, thereby avoiding resource waste of the action correction device 10.

[0124] S204: The action correction device 10 corrects the first action data according to the constraint conditions corresponding to the target action semantics, so that the action of the virtual image expresses the target action semantics.

[0125] In some possible implementations, the motion correction device 10 may use an inverse kinematics (IK) algorithm to perform motion correction. Specifically, the motion correction device 10 may determine the desired position of the avatar according to the constraint conditions corresponding to the target motion semantics, and determine the current position of the avatar according to the first motion data, and then correct the first motion data according to the current position of the avatar and the desired position of the avatar.

[0126] The current position of the virtual image may include the current position of at least one body part in the virtual image. Specifically, the motion correction device 10 may determine the current position of at least one body part in the virtual image based on the kinematic parameters of at least one body part in the virtual image. Similarly, the expected position of the virtual image may include the expected position of at least one body part in the virtual image. Since the constraint condition indicates the constraint on the motion data related to the target motion semantics, and the motion data related to the target motion semantics may include the kinematic parameters of at least one body part, the motion correction device 10 may determine the expected position of at least one body part in the virtual image based on the body part appearing in the constraint condition. For example, when the constraint condition is that the upper arm rotation angle is greater than y and less than z, at least one body part in the virtual image may be an upper arm skeletal point, and the expected position may be any position where the upper arm rotation angle is greater than y and less than z.

[0127] Considering that different virtual images may have different body structures, in a specific implementation, the action correction device 10 may determine the correction order according to the body structure of the virtual image, and then correct the first action data according to the correction order. In some embodiments, when the virtual image is a graphical representation of a real person, the body structure of the virtual image may be a tree structure, that is, each bone point in the virtual image may constitute a tree structure, and the action correction device 10 may determine the correction order of at least one body part according to the position of at least one body part in the virtual image in the tree structure. For example, the action correction device 10 may determine the body part (such as the finger bone point) located at the child node position of the tree structure as the body part to be corrected first, and determine the body part (such as the upper arm bone point) located at the root node position of the tree structure as the body part to be corrected last. In this way, the first action data of at least one body part is corrected in sequence according to the correction order.

[0128] In this way, the motion correction device 10 can determine the joint point parameters of at least one body part in the virtual image based on the current position of at least one body part in the virtual image and the expected position of at least one body part in the virtual image, using the kinematic equation, and then determine the motion trajectory of the virtual image, so that at least one body part in the virtual image moves to the expected position, thereby realizing the correction of the first motion data.

[0129] Furthermore, the motion correction device 10 can also perform smoothing on the motion data after the motion correction to obtain the final corrected first motion data. After obtaining the corrected first motion data, the following steps can also be performed:

[0130] B1: The motion correction device 10 sends the corrected first motion data to the motion generation device 30;

[0131] B2: the action generating device 30 generates the action of the virtual image according to the corrected first action data;

[0132] B3: the action generating device 30 sends the action of the virtual image to the interaction device 40;

[0133] B4: The interactive device 40 renders the virtual image and scene to generate a display video.

[0134] It is understandable that after the action correction device 10 corrects the first action data, the action generation device 30 needs to re-drive the virtual image according to the corrected first action data. For example, the action generation device 30 can transfer the modified first action data to the virtual image to generate the action of the virtual image. By rendering the virtual image and the scene through the interactive device 40, the action of the virtual image in the display video presented to the user can accurately express the semantics of the target action, thereby improving the user's experience of watching the virtual image display video.

[0135] Based on the above description, the action correction method in the embodiment of the present application determines the target action semantics, and then determines the constraint conditions corresponding to the target action semantics, and uses the constraint conditions to correct the action data of the virtual image, so that the corrected action data can express the target action semantics, ensuring that the action semantics of the virtual image are the same as the expected action semantics, thereby enhancing the semantic expression ability of the virtual image.

[0136] In the action correction method provided in the embodiment of the present application, the action correction device 10 can query the target action semantics in the action semantic constraint library to determine the constraint conditions. The construction process of the action semantic constraint library will be described below.

[0137] See also Figure 3 A flow chart of another motion correction method, in which the second motion data of the first object can drive the virtual image to obtain the first motion data of the virtual image. In some embodiments, during the movement of the first object, video data is collected to identify motion semantics of the video data, determine the target motion semantics, and then query the constraint conditions in the motion semantic constraint library to determine the constraint conditions corresponding to the target motion semantics, and correct the first motion data according to the constraint conditions corresponding to the target motion semantics.

[0138] The embodiments of the present application support constructing action semantic constraints in the action semantic constraint library in different ways. In some embodiments, third action data input by a user and first action semantics matching the third action data can be received, and constraints corresponding to the first action semantics can be generated according to the third action data.

[0139] The third motion data can be obtained by collecting the motion of the second object, for example, by collecting the motion of the second object through a motion capture device in the motion acquisition device 20. By analyzing the third motion data, constraints can be automatically generated, and the generated constraints are matched with the motion semantics (i.e., the first motion semantics) input by the user to achieve the construction of constraints. In this way, constraints can be automatically generated based on the motion data input by the user, without the need for the user to manually extract and configure constraints.

[0140] In specific implementation, the user can mark the key body parts in the third action data, determine the action data of the key body parts based on the key body parts marked in the third action data, and then generate the constraint conditions corresponding to the first action semantics based on the action data of the key body parts.

[0141] Among them, the key body parts include at least one body part of the second object. In an embodiment of the present application, the key body parts can be understood as body parts that need to be paid attention to in the first action semantics. For example, the key body parts may include at least one skeletal point of the second object, and the action data of the key body parts may include the kinematic parameters of the at least one skeletal point, and the constraint conditions may be the values ​​of the kinematic parameters of the at least one skeletal point, or the range of values ​​of the kinematic parameters of the at least one skeletal point determined by taking the kinematic parameters of the at least one skeletal point as the reference value. In this way, by analyzing the action data of the key body parts, it is possible to generate constraint conditions, so that the constraint conditions can more accurately determine whether the action data can express the first action semantics.

[0142] Based on the above description, it can be known that by providing an open action semantic constraint library construction interface to the user, the user can add the constraint conditions corresponding to the action semantics that need to be paid attention to in the action semantic constraint library according to actual needs (such as actual usage scenarios). For example, in a live broadcast scene, the user can add the constraint conditions corresponding to the action semantics expressing "thank the audience" in the action semantic constraint library. For another example, in a news broadcast scene, the user can add the constraint conditions corresponding to the action semantics expressing "explain the news" in the action semantic constraint library. In this way, users are supported to customize the construction of action semantic constraint libraries in different scenarios, enriching the applicable occasions of the action correction method provided in the embodiment of the present application.

[0143] In other embodiments, the user may also directly configure the constraint conditions of the action semantics. Specifically, the second action semantics input by the user and the constraint of the action data related to the second action semantics may be received, and the constraint conditions corresponding to the second action semantics may be generated according to the constraint of the action data related to the second action semantics.

[0144] In other words, when the user has constraints on action data related to the second action semantics, the user can directly use the constraints on the action data related to the second action semantics to configure the constraint conditions of the second action semantics. In this way, the user can quickly and conveniently add constraints to the action semantics constraint library without collecting third action data.

[0145] In some possible implementations, before the action semantics constraint library is launched, the configuration personnel (e.g., animators) can use the above method to pre-configure the constraints. For example, animators can configure the constraints corresponding to the second action semantics for some common second action semantics according to the laws of human body movement. In this way, for action semantics with higher frequency of occurrence and more common, preset constraints are provided in the action semantics constraint library to improve the applicability of the action semantics constraint library.

[0146] The action correction method provided in the embodiment of the present application will be described below with reference to specific examples.

[0147] In the live broadcast scene, the anchor can express his gratitude to the audience by performing the action of "making a heart shape with both hands on the head". Therefore, in this scene, the action semantics of "making a heart shape with both hands on the head" deserves attention. First, the user can configure the constraint conditions corresponding to "making a heart shape with both hands on the head" in the action semantic constraint library. Specifically, the second object can perform the action of "making a heart shape with both hands on the head". By collecting the action of the second object, the third action data is obtained. The user can input the third action data and the second action semantics matching the third action data (i.e., "making a heart shape with both hands on the head"), and mark the key body parts in the third action data, such as the hand bone points, elbow bone points, shoulder bone points, etc. The action correction device 10 can determine the kinematic parameters (such as rotation angle and displacement change information) of the marked key body parts according to the marked key body parts in the third action data, and generate the constraint conditions corresponding to "making a heart shape with both hands on the head". For example, the constraint conditions may include that the distance between the two hands is less than x, the rotation angle of the upper arm is greater than y and less than z, the rotation angle of the lower arm is greater than k and less than j, and the rotation angle of the finger is greater than o and less than h.

[0148] In the process of actually driving the virtual image, the action correction device 10 can correct the action data of the virtual image. Specifically, the first object performs the action of "making a heart shape with both hands on the top of the head". By collecting the action of the first object, the second action data is obtained. The action generation device 30 can use the second action data to drive the virtual image and obtain the first action data. Then, the action correction device 10 determines the target action semantics (i.e., "making a heart shape with both hands on the top of the head"), and determines the constraint conditions corresponding to "making a heart shape with both hands on the top of the head" by querying the action semantic constraint library. By comparing the first action data with the constraint conditions corresponding to "making a heart shape with both hands on the top of the head", the first action data is corrected so that the corrected first action data can express the target action semantics of "making a heart shape with both hands on the top of the head".

[0149] In this way, users can configure corresponding constraints for action semantics that need to be paid attention to in different scenarios. When the target action semantics have constraints in the action semantic constraint library, the action data of the virtual image is corrected so that the virtual image can accurately express the action of the first object.

[0150] Based on the motion correction method of the above embodiment, the present application embodiment further provides a motion correction device 10 as described above. The motion correction device 10 is introduced below in conjunction with the accompanying drawings.

[0151] See also Figure 1 The structural diagram of the motion correction device 10 shown in FIG. 1 , the motion correction device 10 comprises:

[0152] The acquisition module 101 is used to acquire first motion data, wherein the first motion data is motion data of a virtual image, the virtual image includes at least one body part, and the motion data of the virtual image includes kinematic parameters of at least one body part;

[0153] A determination module 102 is used to determine the target action semantics, where the target action semantics indicates the meaning of the action expression of the desired avatar;

[0154] The determination module 102 is further used to determine a constraint condition corresponding to the target action semantics, wherein the constraint condition indicates a constraint on action data related to the target action semantics, the action data related to the target action semantics includes kinematic parameters of at least one body part, and the constraint includes a value of the kinematic parameter and / or a value range of the kinematic parameter;

[0155] The correction module 103 is used to correct the first action data according to the constraint conditions corresponding to the target action semantics, so that the action of the virtual image expresses the target action semantics.

[0156] The above-mentioned file acquisition module 101, determination module 102 and correction module 103 can be implemented by hardware modules or by software modules. Among them, the acquisition module 101, determination module 102 and correction module 103 can be implemented by a computing device or a computing engine on a computing device. Below, the determination module 102 is taken as an example for description.

[0157] When implemented by software, the determination module 102 may be an application or application module running on a computing device or a computing device cluster, such as a computing engine. The application may be provided to users in the form of a virtualized service. Virtualized services may include virtual machine (VM) services, bare metal server (BMS) services, and container services. Among them, the VM service may be a service that virtualizes a virtual machine (VM) resource pool on multiple physical hosts (such as computing devices) through virtualization technology to provide users with VMs on demand for use. The BMS service is a service that virtualizes a BMS resource pool on multiple physical hosts to provide users with BMS on demand for use. The container service is a service that virtualizes a container resource pool on multiple physical hosts to provide users with containers on demand for use. VM is a simulated virtual computer, that is, a logical computer. BMS is a high-performance computing service that can be elastically scalable, and its computing performance is no different from that of a traditional physical machine, and it has the characteristics of secure physical isolation. Containers are a kernel virtualization technology that can provide lightweight virtualization to achieve the purpose of isolating user space, processes, and resources. It should be understood that the VM service, BMS service and container service in the above-mentioned virtualization services are merely specific examples. In actual applications, the virtualization service may also be other lightweight or heavyweight virtualization services, which are not specifically limited here.

[0158] When implemented by hardware, the determination module 102 may include at least one computing device, such as a server, etc. Alternatively, the determination module 102 may also be a device implemented by an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), etc. The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0159] In some possible implementations, the first motion data is obtained by driving the virtual image with the second motion data, and the second motion data is obtained by collecting the motion of the first object. The determination module 102 is specifically configured to:

[0160] According to the action of the first object, the action semantics represented by the action of the first object is determined as the target action semantics.

[0161] In some possible implementations, the determination module 102 is specifically configured to:

[0162] Obtaining video data of an action of a first object;

[0163] Perform action semantic recognition on video data to determine the target action semantics.

[0164] In some possible implementations, the determination module 102 is specifically configured to:

[0165] According to the target action semantics, query the action semantics constraint library, wherein the action semantics constraint library includes a plurality of action semantics constraint conditions;

[0166] From the constraints of multiple action semantics, determine the constraints corresponding to the target action semantics.

[0167] In some possible implementations, the action semantic constraint library 104 is constructed in the following manner:

[0168] Receiving third action data input by a user and first action semantics matching the third action data, wherein the third action data is obtained by collecting an action of the second object;

[0169] Generate constraint conditions corresponding to the first action semantics according to the third action data.

[0170] In some possible implementations, generating a constraint condition corresponding to the first action semantics according to the third action data includes:

[0171] Determining motion data of the key body parts according to the key body parts marked in the third motion data, wherein the key body parts include at least one body part of the second object;

[0172] According to the action data of the key body parts, constraint conditions corresponding to the first action semantics are generated.

[0173] In some possible implementations, the action semantic constraint library 104 is constructed in the following manner:

[0174] receiving a second action semantics input by a user and a constraint of the action data related to the second action semantics;

[0175] According to the constraint of the action data related to the second action semantics, a constraint condition corresponding to the second action semantics is generated.

[0176] In some possible implementations, at least one body part includes a skeleton point, and the kinematic parameter of at least one body part includes at least one of the following:

[0177] The rotation angle of the bone point;

[0178] Displacement change information of bone points;

[0179] The distance between at least two bone points.

[0180] In some possible implementations, the correction module 103 is specifically configured to:

[0181] determining a desired position of the virtual image according to the constraint conditions corresponding to the semantics of the target action, and determining a current position of the virtual image according to the first action data;

[0182] The first motion data is modified according to the current position of the avatar and the desired position of the avatar.

[0183] The present application also provides a computing device 400. Figure 4 As shown, computing device 400 includes: bus 402, processor 404, memory 406 and communication interface 408. Processor 404, memory 406 and communication interface 408 communicate through bus 402. Computing device 400 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in computing device 400.

[0184] The bus 402 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 The bus 402 is represented by only one line, but it does not mean that there is only one bus or one type of bus. The bus 402 may include a path for transmitting information between various components of the computing device 400 (eg, the memory 406, the processor 404, and the communication interface 408).

[0185] The processor 404 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0186] The memory 406 may include a volatile memory, such as a random access memory (RAM). The memory 406 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD) or a solid state drive (SSD). The memory 406 stores executable program code, and the processor 404 executes the executable program code to implement the aforementioned action correction method. Specifically, the memory 406 stores instructions for the action correction device 10 to execute the action correction method.

[0187] The communication interface 408 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 400 and other devices or a communication network.

[0188] The embodiment of the present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0189] like Figure 5 As shown, the computing device cluster includes at least one computing device 400. The memory 406 in one or more computing devices 400 in the computing device cluster may store the same motion correction apparatus 10 for executing instructions of the motion correction method.

[0190] In some possible implementations, one or more computing devices 400 in the computing device cluster may also store some instructions of the action correction device 10 for executing the action correction method. In other words, the combination of one or more computing devices 400 may jointly execute the instructions of the action correction device 10 for executing the action correction method.

[0191] It should be noted that the memory 406 in different computing devices 400 in the computing device cluster can store different instructions for executing partial functions of the action correction device 10.

[0192] Figure 6 A possible implementation is shown. Figure 6As shown, two computing devices 400A and 400B are connected via a communication interface 408. The memory in computing device 400A stores instructions for executing the functions of acquisition module 101. The memory in computing device 400B stores instructions for executing the functions of determination module 102 and correction module 103. In other words, the memories 406 of computing devices 400A and 400B jointly store instructions for the action correction apparatus 10 to execute the action correction method.

[0193] Figure 6 The connection mode between the computing device clusters shown may be that considering that the action correction method provided in the present application needs to obtain the first action data and correct the first action data, it is considered that the functions implemented by the acquisition module 101 are performed by the computing device 400A, and the functions implemented by the determination module 102 and the correction module 103 are performed by the computing device 400B.

[0194] It should be understood that Figure 6 The functions of the computing device 400A shown in FIG. 4 may also be completed by multiple computing devices 400. Similarly, the functions of the computing device 400B may also be completed by multiple computing devices 400.

[0195] In some possible implementations, one or more computing devices in the computing device cluster may be connected via a network, which may be a wide area network or a local area network. Figure 7 A possible implementation is shown. Figure 7 As shown, two computing devices 400C and 400D are connected via a network. Specifically, the network is connected via a communication interface in each computing device. In this type of possible implementation, the memory 406 in the computing device 400C stores instructions for executing the functions of the acquisition module 101. At the same time, the memory 406 in the computing device 400D stores instructions for executing the functions of the determination module 102 and the correction module 103.

[0196] Figure 7 The connection mode between the computing device clusters shown can be considered that the action correction method provided by the present application needs to obtain the first action data and correct the first action data. Therefore, it is considered that the functions implemented by the acquisition module 101 are performed by the computing device 400C, and the functions implemented by the determination module 102 and the correction module 103 are performed by the computing device 400D. It should be understood that Figure 7 The functions of the computing device 400C shown in FIG. 4 may also be completed by multiple computing devices 400. Similarly, the functions of the computing device 400D may also be completed by multiple computing devices 400.

[0197] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to execute the above-mentioned application to the action correction device for executing the action correction method.

[0198] The embodiment of the present application also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the above-mentioned action correction method.

[0199] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A motion correction method, characterized in that: The method comprises: Acquire first motion data, where the first motion data is motion data of a virtual image, the virtual image includes at least one body part, and the motion data of the virtual image includes kinematic parameters of the at least one body part; Determining target action semantics, wherein the target action semantics indicates a meaning that the action of the avatar is expected to express; Determining a constraint condition corresponding to the target action semantics, the constraint condition indicating a constraint on action data related to the target action semantics, the action data related to the target action semantics including kinematic parameters of the at least one body part, the constraint including a value of the kinematic parameter and / or a value range of the kinematic parameter; According to the constraint conditions corresponding to the target action semantics, the first action data is modified so that the action of the virtual image expresses the target action semantics.

2. The method according to claim 1, characterized in that The first action data is obtained by driving the virtual image with the second action data, and the second action data is obtained by collecting the action of the first object, and the determining of the target action semantics includes: According to the action of the first object, the action semantics represented by the action of the first object is determined as the target action semantics.

3. The method according to claim 2, characterized in that The determining, according to the action of the first object, the action semantics represented by the action of the first object as the target action semantics includes: Acquire video data of an action of the first object; Perform action semantic recognition on the video data to determine target action semantics.

4. The method according to any one of claims 1 to 3, characterized in that: The determining of the constraint conditions corresponding to the semantics of the target action includes: According to the target action semantics, querying an action semantics constraint library, wherein the action semantics constraint library includes a plurality of action semantics constraint conditions; From the constraints of the multiple action semantics, determine the constraints corresponding to the target action semantics.

5. The method according to claim 4, characterized in that The action semantic constraint library is constructed in the following way: Receiving third action data input by a user and first action semantics matching the third action data, wherein the third action data is obtained by collecting an action of a second object; Generate constraint conditions corresponding to the first action semantics according to the third action data.

6. The method according to claim 5, characterized in that Generating, according to the third action data, a constraint condition corresponding to the semantics of the first action, comprises: Determining, according to the key body parts marked in the third motion data, motion data of the key body parts, wherein the key body parts include at least one body part of the second object; According to the action data of the key body part, a constraint condition corresponding to the first action semantics is generated.

7. The method according to claim 4, characterized in that The action semantic constraint library is constructed in the following way: Receiving a second action semantics input by a user and a constraint of action data related to the second action semantics; According to the constraint of the action data related to the second action semantics, a constraint condition corresponding to the second action semantics is generated.

8. The method according to any one of claims 1 to 7, characterized in that: The at least one body part comprises a skeletal point, and the kinematic parameter of the at least one body part comprises at least one of the following: The rotation angle of the skeleton point; Displacement change information of the skeleton points; The distance between at least two of the skeleton points.

9. The method according to any one of claims 1 to 8, characterized in that: The step of modifying the first action data according to the constraint condition corresponding to the semantics of the target action includes: determining a desired position of the virtual image according to the constraint conditions corresponding to the semantics of the target action, and determining a current position of the virtual image according to the first action data; The first motion data is modified according to the current position of the avatar and the desired position of the avatar.

10. A motion correction device, characterized in that: The device comprises: An acquisition module, configured to acquire first motion data, wherein the first motion data is motion data of a virtual image, the virtual image includes at least one body part, and the motion data of the virtual image includes kinematic parameters of the at least one body part; A determination module, configured to determine a target action semantics, wherein the target action semantics indicates a meaning that the action of the virtual image is expected to express; The determination module is further used to determine a constraint condition corresponding to the target action semantics, wherein the constraint condition indicates a constraint on action data related to the target action semantics, wherein the action data related to the target action semantics includes kinematic parameters of the at least one body part, and wherein the constraint includes a value of the kinematic parameter and / or a value range of the kinematic parameter; The correction module is used to correct the first action data according to the constraint conditions corresponding to the target action semantics, so that the action of the virtual image expresses the target action semantics.

11. The device according to claim 10, characterized in that The first action data is obtained by driving the virtual image with the second action data, and the second action data is obtained by collecting the action of the first object, and the determination module is specifically used for: According to the action of the first object, the action semantics represented by the action of the first object is determined as the target action semantics.

12. The device according to claim 11, characterized in that The determination module is specifically used for: Acquire video data of an action of the first object; Perform action semantic recognition on the video data to determine target action semantics.

13. The device according to any one of claims 10 to 12, characterized in that The determination module is specifically used for: According to the target action semantics, querying an action semantics constraint library, wherein the action semantics constraint library includes a plurality of action semantics constraint conditions; From the constraints of the multiple action semantics, determine the constraints corresponding to the target action semantics.

14. The device according to claim 13, characterized in that The action semantic constraint library is constructed in the following way: Receiving third action data input by a user and first action semantics matching the third action data, wherein the third action data is obtained by collecting an action of a second object; Generate constraint conditions corresponding to the first action semantics according to the third action data.

15. The device according to claim 14, characterized in that Generating, according to the third action data, a constraint condition corresponding to the semantics of the first action, comprises: Determining, according to the key body parts marked in the third motion data, motion data of the key body parts, wherein the key body parts include at least one body part of the second object; According to the action data of the key body part, a constraint condition corresponding to the first action semantics is generated.

16. The device according to claim 13, characterized in that The action semantic constraint library is constructed in the following way: Receiving a second action semantics input by a user and a constraint of action data related to the second action semantics; According to the constraint of the action data related to the second action semantics, a constraint condition corresponding to the second action semantics is generated.

17. The device according to any one of claims 10 to 16, characterized in that The at least one body part comprises a skeletal point, and the kinematic parameter of the at least one body part comprises at least one of the following: The rotation angle of the skeleton point; Displacement change information of the skeleton points; The distance between at least two of the skeleton points.

18. The device according to any one of claims 10 to 17, characterized in that The correction module is specifically used for: determining a desired position of the virtual image according to the constraint conditions corresponding to the semantics of the target action, and determining a current position of the virtual image according to the first action data; The first motion data is modified according to the current position of the avatar and the desired position of the avatar.

19. A computing device cluster, characterized in that: The computing device cluster includes at least one computing device, and the at least one computing device includes at least one processor and at least one memory, wherein the at least one memory stores computer-readable instructions; the at least one processor executes the computer-readable instructions so that the computing device cluster executes the method as described in any one of claims 1 to 9.

20. A computer-readable storage medium, characterized in that: The method comprises computer-readable instructions; the computer-readable instructions are used to implement the method according to any one of claims 1 to 9.

21. A computer program product, characterized in that The method comprises computer-readable instructions; the computer-readable instructions are used to implement the method according to any one of claims 1 to 9.

Images