Digital human action control method and device, equipment, medium and product

Abstract

The application relates to a digital human action control method and device, equipment, medium and product, the method comprising: acquiring an action file for controlling the movement of a digital human, the action file comprising information frames corresponding to each image frame in a movement image of the digital human, the information frames storing position information of skeletal key points of the digital human; performing action anomaly detection according to the position information of the skeletal key points in each information frame, determining an action anomaly segment describing an action anomaly phenomenon, the action anomaly segment comprising one or more time-sequentially continuous information frames; correcting the action anomaly segment to overcome the action anomaly phenomenon; and driving the movement of the digital human by applying the action file to generate the movement image. The application performs action anomaly detection and optimization on the action file for driving the movement of the digital human, so that the corresponding movement image is more smooth and natural.

Description

Technical Field

[0001] This application relates to digital human virtual technology, and more particularly to a digital human motion control method, apparatus, device, medium, and product thereof. Background Technology

[0002] With the rise of the "metaverse," the introduction of "digital humans" into live streaming scenarios has become a standard feature of live streaming applications, with major manufacturers and live streaming platforms successively launching virtual "digital human" functions. Live streaming using "digital humans" has become a popular choice for many streamers. In live streaming scenarios, "digital humans" offer many new possibilities, with using external materials such as sound, motion, and music to drive their movement becoming a novel and popular feature.

[0003] Currently, solutions that drive "digital humans" using external materials typically employ machine learning to generate motion files that control the "digital human's" movements. These motion files contain data for each frame of the "digital human's" movement, with each frame containing the coordinate information of various skeletal nodes. The coordinate information of each skeletal node is used to instruct the drawing of the "digital human's" posture. By continuously drawing multiple frames, the "digital human" can be driven to present a motion effect.

[0004] Because machine learning-generated motion files often contain biases, the actions displayed in the video images of the "digital humans" driven by them often exhibit many uncoordinated anomalies such as clipping, lack of smoothness, stillness, and rhythm issues, thus requiring the development of a corrective solution. Summary of the Invention

[0005] The purpose of this application is to solve the above-mentioned problems by providing a digital human motion control method and corresponding devices, equipment, non-volatile readable storage media, and computer program products.

[0006] According to one aspect of this application, a digital human motion control method is provided, comprising the following steps:

[0007] Obtain motion files corresponding to control the movement of the digital human. These motion files include information frames corresponding to each image frame in the motion image of the digital human. The information frames store the position information of the key points of the digital human's skeleton.

[0008] Based on the position information of the skeletal key points in each information frame, motion anomaly detection is performed to determine the existence of motion anomaly segments describing motion anomalies. The motion anomaly segments include one or more information frames that are sequentially continuous.

[0009] Correct the abnormal action segment to overcome the abnormal action phenomenon;

[0010] The motion file is used to drive the digital human to move, thereby generating the motion image.

[0011] Optionally, motion anomaly detection is performed based on the position information of skeletal key points in each information frame to determine the existence of motion anomaly segments describing abnormal motion phenomena, including:

[0012] Read at least one information frame from the action file as the target information frame, and determine the torso model space of the digital human based on the skeletal key points corresponding to the torso region of the digital human in the target information frame.

[0013] Based on the position information of the skeletal key points corresponding to the limb area in the target information frame, it is detected whether the limb area falls into the torso model space. When it falls into the torso model space, it is determined that there is an abnormal action phenomenon corresponding to clipping in the target information frame.

[0014] One or more time-series consecutive target information frames that exhibit abnormal motion phenomena are constructed as motion abnormality segments.

[0015] Optionally, the torso model space of the digital human is determined based on the skeletal key points corresponding to the torso region in the target information frame, including:

[0016] The triangular plane region is determined based on the two shoulder joint points corresponding to the torso region and the three skeletal key points corresponding to the torso base point in the target information frame.

[0017] Calculate and determine the geometric center point of the triangular plane region, and determine the coordinate information of the geometric center point in the three-dimensional coordinate system;

[0018] Based on the coordinate information of the geometric center point, the three-dimensional expansion is performed to determine the torso model space of the digital human.

[0019] Optionally, motion anomaly detection is performed based on the position information of skeletal key points in each information frame to determine the existence of motion anomaly segments describing abnormal motion phenomena, including:

[0020] The similarity of the motion feature vectors of every two temporally consecutive information frames in the motion file is continuously calculated. The motion feature vectors include the positional information of the skeletal key points of the digital human limbs in the corresponding information frames.

[0021] When the similarity exceeds a preset threshold, the abnormal action segment covering the two information frames is determined by expanding from the action file based on the two corresponding information frames.

[0022] Optionally, correcting the abnormal action segment to overcome the abnormal action phenomenon includes any one or more of the following steps:

[0023] When the abnormal action segment includes a single information frame, the position information of the skeletal key points falling into the torso model space is adjusted based on the torso model space corresponding to the information frame, so that the corresponding limb area does not overlap with the torso model space.

[0024] When the abnormal action segment includes a first number of multiple information frames, a set of information frames similar to the actions of each information frame in the abnormal action segment is matched from the material library, and the abnormal action segment in the action file is replaced with the set of information frames.

[0025] When the abnormal action segment includes a second number of information frames, the information frames in the action file that are before and after the abnormal action segment are used as reference frames. An interpolation operation is performed based on the two reference frames to generate an information frame set, and the abnormal action segment in the action file is replaced by the information frame set.

[0026] Optionally, a set of information frames similar to the actions of each information frame in the abnormal action segment can be matched from the material library, including:

[0027] Obtain the motion feature vector corresponding to each information frame in the abnormal motion segment, wherein the motion feature vector includes the position information of the skeletal key points of the digital human limb in the corresponding information frame;

[0028] The abnormal action fragment is compared with the information frame set of each material in the material library. The data distance between the action feature vectors of each two information frames corresponding to the time sequence is calculated based on the temporal correspondence of the information frames. The data distances of each information frame in the information frame set of each material are accumulated to obtain the similarity score of the material.

[0029] The set of information frames with the highest similarity score is determined as the set of information frames similar to the abnormal action segment.

[0030] Optionally, frame interpolation based on the two reference frames includes:

[0031] The similarity of the motion feature vectors of the two reference frames is detected. The motion feature vectors include the positional information of the skeletal key points of the digital human limbs in the corresponding information frames.

[0032] When the similarity reaches a preset threshold, a frame interpolation operation is performed on the two reference frames in a three-dimensional coordinate system.

[0033] When the similarity does not reach the preset threshold, the position information of the skeletal key points in the two reference frames is converted from three-dimensional coordinate information to Euler angles. Based on the Euler angles, the two reference frames are interpolated and then converted back to three-dimensional coordinate information.

[0034] According to another aspect of this application, a digital human motion control device is provided, comprising:

[0035] The file acquisition module is configured to acquire motion files corresponding to the control of the digital human's movement. The motion files include information frames corresponding to each image frame in the motion image of the digital human. The information frames store the position information of the key points of the digital human's skeleton.

[0036] An anomaly detection module is configured to perform motion anomaly detection based on the position information of skeletal key points in each information frame, and determine that there are motion anomaly segments describing motion anomalies, wherein the motion anomaly segments include one or more information frames that are sequentially continuous.

[0037] The motion correction module is configured to correct the abnormal motion segment in order to overcome the abnormal motion phenomenon.

[0038] The file application module is configured to use the motion file to drive the digital human to move, thereby generating the motion image.

[0039] According to another aspect of this application, a digital human motion control device is provided, including a central processing unit and a memory, wherein the central processing unit is used to invoke and run a computer program stored in the memory to execute the steps of the digital human motion control method described in this application.

[0040] According to another aspect of this application, a non-volatile readable storage medium is provided, which stores a computer program implemented according to the digital human motion control method in the form of computer-readable instructions, wherein the computer program, when invoked by a computer, executes the steps included in the method.

[0041] According to another aspect of this application, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the method described in any embodiment of this application.

[0042] Compared to existing technologies, this application performs motion anomaly detection on pre-generated motion files used to drive digital human motion, identifies the information frames corresponding to the abnormal motion segments, corrects the information frames, and eliminates many uncoordinated motion anomalies in digital human motion, such as clipping, lack of smoothness, stillness, and rhythm. This process does not rely on machine learning models and can efficiently and quickly optimize motion files. After the motion files obtained from this application drive the digital human motion, the motion images of the digital human are smoother and more natural. It avoids relying entirely on the reasoning ability learned by machine learning models, effectively controls the technical implementation cost, and achieves efficient and high-quality processing results. When the motion files optimized by the technical solution of this application are used to control the motion images of digital humans, a more natural and smooth digital human motion effect can be obtained. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a schematic diagram of the network architecture of an exemplary application environment for this application;

[0045] Figure 2 This is a schematic diagram illustrating an exemplary digital human and the distribution of its skeletal key points according to this application;

[0046] Figure 3 This is a flowchart illustrating one embodiment of the digital human motion control method of this application.

[0047] Figure 4 This is a flowchart illustrating the process of determining the abnormal action segment corresponding to the modeling in an embodiment of this application;

[0048] Figure 5 This is a schematic diagram illustrating the process of determining the torso model space of a digital human in an embodiment of this application;

[0049] Figure 6 This is a flowchart illustrating the process of determining the abnormal action segment corresponding to non-smooth action in an embodiment of this application.

[0050] Figure 7 This is a flowchart illustrating the process of matching information frames for abnormal action segments in an embodiment of this application.

[0051] Figure 8 This is a flowchart illustrating the frame interpolation operation performed under different circumstances in the embodiments of this application;

[0052] Figure 9 This is a schematic block diagram of the digital human motion control device of this application;

[0053] Figure 10 This is a schematic diagram of the structure of a digital human motion control device used in this application. Detailed Implementation

[0054] Please see Figure 1The network architecture adopted in an exemplary application scenario of this application includes a terminal device 80, a media server 81, and an application server 82. The application server 82 can be used to deploy a live streaming service, and the media server 81 can be used to generate a video stream corresponding to the digital human based on an action file. When a user accesses the live streaming service provided by the application server 82 from their terminal device 80 and begins to control the output of the video stream corresponding to the digital human, they can specify an action file for controlling the movement of the digital human to generate the video stream. Then, the media server 81 generates the corresponding video stream based on the action file, with the help of a process generated by a computer program that implements the digital human motion control method of this application. Under the coordination of the application server 82, the video stream is pushed to the live streaming room, thereby providing a live streaming service for the digital human through a live streaming network.

[0055] In another exemplary application scenario of this application, the terminal device can implement the functions corresponding to the media server. After the user specifies the action file, the process generated by the computer program that implements the digital human motion control method of this application is run, and a corresponding digital human video stream is generated according to the action file and pushed to a remote computer device, thereby enabling further applications related to calls and live streaming.

[0056] In exemplary industrial applications, the motion file can be generated by a machine learning model pre-trained to a convergent state based on motion-related information provided by music, voice, text, dance videos, etc. The motion file contains information frames corresponding to each image frame in the digital human video stream to be generated. Each information frame stores and describes the image information of the digital human in the corresponding image frame. This image information is described by the position information of all preset skeletal key points of the digital human. The specific data format can be standardized to adapt to the output format of the machine learning model or arbitrarily preset. For example, in one embodiment, when the motion file represents each information frame, it can provide the coordinate values ​​of each preset skeletal key point in the image coordinate system, adapting to the case of 3D modeling and generation of the digital human. The image coordinate system can be a 3D coordinate system, so that the position information of all skeletal key points in each information frame can be represented as a vector in the motion file, arranged row by row. Thus, the entire motion file is described as a matrix with time sequence as the row index. As can be seen, as long as the data structure inside the action file is defined in advance to represent the position information generated by the machine learning model or other arbitrary technical implementation method based on the action association information to control the skeletal key points of the digital human to produce corresponding movements, the action file defined in this application can be used in this application.

[0057] Key skeletal features of a digital human, such as Figure 2As shown in the example, these key points are mainly distributed in the various skeletal joints of the digital human's body and limbs. The displacement of these key skeletal points in the image coordinate system can usually drive the corresponding body parts of the digital human to produce corresponding motion effects. By progressively adjusting the positional information of the same set of key skeletal points in multiple information frames, the corresponding image frames that produce gradual changes in motion of the digital human can be controlled accordingly. By playing the video stream composed of these image frames in an orderly manner, the motion effects of the digital human body parts corresponding to the corresponding key skeletal points can be visually presented.

[0058] The digital human can be pre-modeled so that, according to the position information of each skeletal key point in the information frame, the image frame position of the body parts corresponding to each skeletal key point can be adjusted so that the corresponding digital human image frame can be generated through rendering. In addition, according to actual needs, when generating the digital human image frame, corresponding background images or other foreground images can be added to it. Those skilled in the art can implement this flexibly.

[0059] Please see Figure 3 According to a digital human motion control method provided in this application, in one embodiment, it includes the following steps:

[0060] Step S1100: Obtain the motion file corresponding to control the movement of the digital human. The motion file includes information frames corresponding to each image frame in the motion image of the digital human. The information frames store the position information of the key points of the skeletal structure of the digital human.

[0061] For a known motion file, as mentioned earlier, it stores information frames used to control the digital human to generate a video corresponding to it. Each information frame describes the positional information of various skeletal key points in the three-dimensional model of the digital human, such as... Figure 2 As shown in the example, there are 24 skeletal key points in a 3D digital human model. Therefore, the motion file stores the position information of these 24 skeletal key points in different image frames of the video stream. The set of image information consisting of the position information of all 24 skeletal key points corresponding to each image frame is regarded as an information frame. Usually, there is a one-to-one correspondence between information frames and image frames according to the time sequence.

[0062] In one embodiment, the position information can be described by the relative position information between skeletal keypoints. For example, taking a certain skeletal keypoint on the body of the digital human's 3D model as a reference point, the position information of this reference point is represented as [0,0,0], and the position information of other skeletal keypoints can be represented as [Δx]. i,j ,Δy i,j ,Δz i,j], where Δ represents the coaxial relative offset, i represents the sequence number of the information frame, and j represents the sequence number of the skeletal keypoint. In another embodiment, the position information can also be described by the absolute position information of a three-dimensional coordinate system, for example, the position information of a skeletal keypoint can be represented as [x i,j ,y i,j ,z i,j ].

[0063] In practice, the data structure within the action file can be represented in any way. A pre-defined data structure can be used to represent the positional information of all skeletal keypoints in all information frames of the action file. Regardless of the method used to represent the positional information of the skeletal keypoints, as long as it can be parsed and utilized by this application, it is acceptable.

[0064] Since the digital human is a 3D model obtained by 3D modeling, the position information can generally be represented in a 3D coordinate system. Therefore, the position information representation can be quickly mapped to the 3D model of the digital human, which facilitates the actual image transformation and adjustment of the corresponding skeletal key points in the 3D model of the digital human.

[0065] When the motion file is obtained using a machine learning model, corresponding input information is prepared according to the input information format specified during the training of the machine learning model. This input information is then fed into the machine learning model, which infers the location information of the skeletal key points of the digital human in each image frame of the corresponding video stream. This information is then stored as a corresponding motion file according to the data format specification of the motion file, and can be called upon. For example:

[0066] In one embodiment, the machine learning model can extract audio features from audio data provided by music files or voice data, and then generate position information of key points of the digital human skeleton corresponding to the corresponding dance movements based on the audio features, store the corresponding motion files, and use them to control the digital human to perform dance movements, thereby realizing intelligent choreography services.

[0067] In another embodiment, the machine learning model can detect and determine the position information of the skeletal key points of the human object performing dance movements in each image frame based on the image features corresponding to multiple image frames in the dance video, store and generate corresponding motion files, and use them to control the digital human to perform dance movements, thereby realizing the service of having a real person perform dance and transferring the dance movements to the digital human.

[0068] In another embodiment, the machine learning model can extract textual semantic features from the action description text expressed in natural language, generate positional information of the key points of the digital human skeleton corresponding to the corresponding dance action based on the textual semantic features, store it as a corresponding action file, and use it to control the digital human to perform dance actions, thereby realizing associative choreography service.

[0069] By extension, in other possible scenarios, the machine learning model can also be pre-trained to serve martial arts movements, gymnastics movements, game movements, etc. It can extract deep semantic information based on the movement association information provided by the music, voice, text, dance video and other materials, and then generate the position information of the skeletal key points of the corresponding digital human based on the deep semantic information, store the corresponding action file, and use it to control the digital human to perform the corresponding actions.

[0070] Step S1200: Perform motion anomaly detection based on the position information of the skeletal key points in each information frame, and determine that there are motion anomaly segments describing motion anomalies, wherein the motion anomaly segments include one or more information frames that are sequentially continuous.

[0071] Because motion files generated by machine learning models or other methods may contain inaccurate descriptions of the location information of key skeletal points of a digital human, when the corresponding video stream is generated by controlling the movement of the digital human according to the motion file, abnormal phenomena such as clipping, uneven movement or rhythm, or stillness may occur when the video stream is played. Therefore, before applying the motion file, motion anomaly detection can be performed on the motion file. By detecting and identifying one or more information frames corresponding to motion anomalies, these information frames can be identified as motion anomaly segments for centralized optimization.

[0072] In one embodiment, the presence of anomalies describing clipping in the action file can be detected based on the positional information of the skeletal key points in each information frame. Specifically, this can be achieved by detecting whether one or more information frames, after application, cause clipping effects when the digital human performs actions. Clipping, as referred to here, means the intersection of different parts of the digital human's body in three-dimensional space. For example, when the digital human's arm passes through the torso model space, it is a clipping phenomenon. The torso model space can be defined as the three-dimensional region containing the digital human's chest cavity. It is easy to understand that since the digital human's movements are mainly large-scale movements of limbs such as hands and feet, detecting clipping can be done by detecting whether the positional information of the skeletal key points of the hands, feet, etc., are contained within the three-dimensional space of the torso. When one or more information frames are found to have such clipping, these frames are considered anomalous information frames. Successive anomalous information frames are marked as the same anomalous action segment, allowing for targeted processing later.

[0073] In another embodiment, the presence of motion anomalies caused by non-smoothness in the motion file, such as inconsistent motion descriptions or rhythms, can be detected based on the positional information of skeletal keypoints in the information frames. Specifically, this can be achieved by detecting whether one or more information frames, after application, cause the digital human to exhibit non-smooth motion when performing actions. Typically, since temporally adjacent information frames should exhibit a gradual change in the positional information of the same skeletal keypoint, the similarity between the vectors of every two adjacent information frames can be detected to determine whether there is non-smoothness between the later information frame and the earlier information frame. One or more temporally consecutive information frames with low similarity are identified as corresponding motion anomaly segments for targeted optimization.

[0074] In another embodiment, the presence of motion anomalies corresponding to motion stillness in the motion file can be detected based on the position information of the skeletal key points in the information frame. Specifically, this can be determined by detecting the change in the similarity between the vectors of one or more later information frames relative to the earlier reference information frame. When the cumulative similarity of multiple later information frames is lower than a preset threshold, it can be determined that all information frames from the reference information frame to the last detected information frame constitute a motion anomaly segment, and subsequent optimization can be performed on this motion anomaly segment.

[0075] The above are different embodiments for implementing motion anomaly detection in different situations. One or more of these embodiments can be applied, with the aim of detecting abnormal motion segments in the motion file for targeted optimization. As can be seen from the above embodiments, the abnormal motion segment includes one or more information frames. These information frames are all considered to contain abnormal information frames describing abnormal motion phenomena and require targeted optimization.

[0076] Step S1300: Correct the abnormal action segment to overcome the abnormal action phenomenon;

[0077] For detected abnormal motion segments, technical means can be used to correct the entire abnormal motion segment, or to correct the position information of the skeletal key points in the abnormal information frames of the abnormal motion segment, so as to achieve the correction and optimization of the abnormal motion segment.

[0078] In one embodiment, the following correction step S1310 can be used: when the abnormal action segment includes a single information frame, the position information of the skeletal key points falling into the torso model space is adjusted based on the torso model space corresponding to the information frame, so that the corresponding limb area does not overlap with the torso model space:

[0079] For animation anomalies consisting of only a single information frame, to minimize computational overhead, individual corrections can be made to the location information of specific skeletal keypoints within that frame. For example, a key skeletal point in a limb causing clipping can be corrected by moving its location outside the torso model space. Similarly, if animation smoothness or stillness is caused by a single information frame, the anomalies can be overcome by correcting the location information of the specific skeletal keypoints within that frame.

[0080] In another embodiment, the following correction step S1320 can be used: when the abnormal motion segment includes a first number of multiple information frames, a set of information frames similar to the actions of each information frame in the abnormal motion segment is matched from the material library, and the abnormal motion segment in the motion file is replaced with the set of information frames.

[0081] For situations where the number of information frames in an abnormal motion segment is large, such as exceeding a preset first number, the entire abnormal motion segment can be replaced to correct it. The material used to replace the abnormal motion segment can be retrieved from a preset material library. This library can pre-collect and store sets of information frames corresponding to different actions, with each set constituting a material. When material needs to be retrieved from the library, the similarity between the vectors of the abnormal motion segment and each set of information frames in the library is determined. Then, the materials are sorted according to the similarity, and the set of information frames from the material with the highest similarity is selected to replace the corresponding abnormal motion segment, thus achieving correction. It should be noted that when calculating the vector similarity between the set of information frames and the abnormal motion segment, the similarity can be calculated frame by frame at the information frame level, then accumulated to obtain the overall similarity. The set of information frames most similar to the abnormal motion segment is then determined based on the overall similarity and used to correct the abnormal motion segment. This embodiment can not only be used to handle abnormal motion segments corresponding to clipping phenomena, but also to handle abnormal motion segments such as non-smooth motion or motion stillness.

[0082] In another embodiment, the following correction step S1330 can be used: when the abnormal action segment includes a second number of information frames, the information frames in the action file that are before and after the abnormal action segment are used as reference frames. An interpolation operation is performed based on the two reference frames to generate an information frame set, and the abnormal action segment in the action file is replaced with the information frame set.

[0083] For situations where there are few information frames in an abnormal motion segment, such as when the number of frames is lower than the first number but exceeds the preset second number, two information frames (one before and one after the abnormal motion segment) can be used as reference frames. Frame interpolation is then performed on these two reference frames to obtain smoothly filtered information frames between the two reference frames. These interpolated information frames replace the abnormal information frames in the abnormal motion segment. When performing the frame interpolation, either linear interpolation or spherical interpolation can be used, which can be flexibly selected by those skilled in the art. Similarly, this embodiment can not only be used to handle abnormal motion segments corresponding to motion anomalies caused by clipping, but also is applicable to handling abnormal motion segments such as non-smooth motion or motion stillness.

[0084] As can be seen, depending on the number of abnormal information frames that need to be corrected in the abnormal action segment, the aforementioned different correction steps S1310, S1320, and S1330 can be flexibly selected to correct the abnormal action segment. Any one or any multiple of the aforementioned correction steps can be used for correction processing, so that after the action file is applied, it can control the digital human to perform relatively smooth and natural actions and generate corresponding smooth image effects.

[0085] In a special case where multiple abnormal motion segments exist in the motion file, if two abnormal motion segments are confirmed to have similar movements through similarity detection, and a corresponding information frame is generated for one of the abnormal motion segments after a corresponding correction method has been determined, these information frames can be used to replace the other abnormal motion segment accordingly, thereby improving correction efficiency. The repetition of abnormal motion segments often occurs in motion files generated from music or dance videos, because both are essentially compiled based on music rhythm information. Music generally has repeated sections, and therefore, corresponding dance movements often repeat as well. Thus, when a previous abnormal motion segment exists in the motion file, one or more subsequent abnormal motion segments will often also exist. In this case, the method described in this embodiment can be used for processing.

[0086] As can be seen from the above description, when all abnormal action segments in the action file are corrected using the methods disclosed in one or more of the above embodiments, the action file is optimized. When the digital human is controlled to move according to the action file, the generated motion image can overcome the corresponding abnormal action phenomenon and the image playback effect is more natural and smooth.

[0087] Step S1400: Apply the motion file to drive the digital human to move, so as to generate the motion image.

[0088] After the motion file is optimized, the 3D model of the digital human can be driven to perform corresponding motion transformations frame by frame according to each information frame in the motion file. Then, the image frame corresponding to the information frame is rendered. Multiple such image frames are encoded into a video stream to generate a corresponding playable video file. Alternatively, the video stream can be pushed to a corresponding network address. After the video stream is played, the smooth motion effect of the digital human performing corresponding actions according to the motion file can be presented.

[0089] In one embodiment, when generating the image frames corresponding to each of the information frames, background images and / or decorative images can be added to the digital human to make the image more realistic.

[0090] In another embodiment, a live streamer in the live streaming room of a network live streaming service can specify a piece of music. The server obtains the corresponding motion file for the music, optimizes it using the method of this application, and generates corresponding image frames frame by frame based on each information frame in the optimized motion file to obtain the corresponding video stream. The audio stream of the music is then merged to form a live stream, which is pushed to the live streamer's live streaming room. After the viewers in the live streaming room obtain the live stream, the player in the live streaming room parses and plays it, thereby obtaining an audio and video playback effect that synchronizes the music with the digital human's movements, thus realizing a virtual live streaming service.

[0091] As can be seen from the above embodiments, this application performs motion anomaly detection on the pre-generated motion file used to drive the digital human's movement, determines the information frames corresponding to the abnormal motion segments, corrects the information frames, and eliminates many uncoordinated motion anomalies in the digital human's movement, such as clipping, lack of smoothness, stillness, and rhythm. In this process, there is no need to rely on machine learning models, which can efficiently and quickly optimize the motion file. After the motion file obtained from this application drives the digital human's movement, the motion image of the digital human is smoother and more natural. It avoids relying entirely on the reasoning ability learned by machine learning models, effectively controls the technical implementation cost, and achieves efficient and high-quality processing results. When the motion file optimized by the technical solution of this application is used to control the motion image of the digital human, a more natural and smooth digital human movement effect can be obtained.

[0092] Based on any embodiment of this application, please refer to Figure 4 Based on the position information of the skeletal key points in each information frame, motion anomaly detection is performed to determine the presence of motion anomaly segments describing abnormal motion phenomena, including:

[0093] Step S2100: Read at least one information frame from the action file as the target information frame, and determine the torso model space of the digital human based on the skeletal key points corresponding to the torso region of the digital human in the target information frame.

[0094] When performing clipping detection on the action file, each information frame can be detected as a target information frame. Starting from the first target information frame, if the detection determines that there is clipping, then the corresponding target information frame is an abnormal information frame.

[0095] To perform penetration detection, it is necessary to determine the torso region of the digital human. The torso region is generally defined as the area corresponding to the chest of the digital human, which can be achieved through methods such as... Figure 2As shown in the example of digital human skeleton key points, skeleton key points 16, 17, and 9 can be used to define the chest region. Centered on the chest region of the digital human, a torso model space can be extended in the corresponding three-dimensional space of the digital human. The size of the torso model space can be set to cover the entire chest space of the digital human. In addition, a margin corresponding to the clothing worn by the digital human can be reserved, so the torso model space can be appropriately expanded to accommodate this margin.

[0096] Step S2200: Detect whether the limb region falls into the torso model space based on the position information of the skeletal key points corresponding to the limb region in the target information frame. When it falls into the torso model space, determine that there is an abnormal action phenomenon corresponding to clipping in the target information frame.

[0097] The limb areas of a digital human, for example, such as Figure 2 As shown, it can be set to the area defined by the skeletal key points 17, 19, and 21 on the left arm and the skeletal key points 16, 18, and 20 on the right arm.

[0098] In a specific example embodiment, each arm can use the positional information corresponding to shoulder joint point 17 or 16, elbow joint point 18 or 19, and wrist joint point 20 or 21 to define multiple points, in order to further refine the positional information. Specifically, for the left arm in the figure, the corresponding limb region of the left arm can be defined by four points: elbow joint point 19, wrist joint point 21, the center point between shoulder joint point 17 and elbow joint point 19, and the center point between elbow joint point 19 and wrist joint point 21. Similarly, for the right arm in the figure, the corresponding limb region of the right arm can be defined by four points: elbow joint point 18, wrist joint point 20, the center point between shoulder joint point 16 and elbow joint point 18, and the center point between elbow joint point 18 and wrist joint point 20. The positional information corresponding to the above four points of the left and right arms can be constructed as action feature vectors in this application for similarity detection of actions between two information frames. Its specific application will be disclosed in other embodiments of this application, and will not be discussed here.

[0099] Therefore, the limb region corresponding to each arm can be represented as a feature vector composed of the positional information of four points, realizing the numerical representation of the limb region. The digital human torso model space has also completed the corresponding numerical representation in advance. Therefore, based on the numerical representation, it can be determined by calculation whether the limb region falls into the torso model space. When any part of the limb region determined according to a target information frame falls into the torso model space, it can be determined that the target information frame describes the fact of abnormal movement. Thus, the target information frame can be identified as an abnormal information frame.

[0100] For each information frame in the action file, it can be determined whether each information frame belongs to an abnormal information frame in the above manner, thereby completing the detection of all information frames.

[0101] It should be noted that the above is only an example of using the arm as the corresponding limb area to refer to the torso model space corresponding to the chest. The same principle can be applied to the limb areas corresponding to the legs and feet. For different flexibility requirements, those skilled in the art can adjust as needed based on the principles disclosed here.

[0102] Step S2300: Construct one or more time-series consecutive target information frames that exhibit abnormal motion phenomena into an abnormal motion segment.

[0103] Once an abnormal information frame is detected, it is constructed into a single action anomaly segment based on its temporal continuity. This allows for subsequent corrections to be performed on a segment-by-segment basis. It's easy to understand that the information frames within an action anomaly segment can include one or more frames, depending on the specific situation.

[0104] As can be seen from the above embodiments, when performing clipping detection, the limb regions with large amplitude and frequent movements and the torso model space with small amplitude and relatively infrequent movements are first determined based on the information frames in the action file. Then, the relative positional relationship between the limb regions and the torso model space is detected. The relative positional relationship is used to determine whether the corresponding information frame describes the clipping phenomenon. The amount of data required is small, the computational efficiency is high, and the detection can be completed quickly without relying on complex technical means for massive calculations. The detection effect is accurate. For scenarios with high real-time requirements, this embodiment is more likely to meet the timeliness requirements.

[0105] Based on any embodiment of this application, please refer to Figure 5 The torso model space of the digital human is determined based on the skeletal key points corresponding to the torso region in the target information frame, including:

[0106] Step S2110: Determine the triangular plane region based on the two shoulder joint points corresponding to the torso region and the three skeletal key points corresponding to the torso base point in the target information frame.

[0107] like Figure 2 The example of the digital human has two shoulder joint points 16 and 17, and a torso base point 9 located at the bottom of its chest (or any of the points labeled 6, 3, or 0 can be used as the torso base point). A triangular plane region can be obtained by connecting these three points.

[0108] Step S2120: Calculate and determine the geometric center point of the triangular plane region, and determine the coordinate information of the geometric center point in the three-dimensional coordinate system;

[0109] Based on the position information of the two shoulder joints and the torso base point described in the target information frame, the position information of the geometric center point of the triangular plane region can be quickly calculated by applying the triangle center point formula. This position information can be represented as coordinate information in a three-dimensional coordinate system corresponding to the three-dimensional model of the digital human.

[0110] Step S2130: Expand the three-dimensional direction according to the coordinate information of the geometric center point to determine the torso model space of the digital human.

[0111] Since the digital human is a three-dimensional model, the torso model space corresponding to the digital human should be determined based on the three-dimensional model. Therefore, the geometric center point of the triangular plane region can be used as the base point of the torso model space. Then, according to the preset thickness, a cuboid space is extended along the three dimensions of the three-dimensional coordinate system. This cuboid space not only completely covers the corresponding position of the digital human's body, but also reserves a certain size for the digital human's clothing. Thus, the cuboid space that has been set up above is set as the torso model space of the digital human, and a numerical representation is completed. It can be used to detect whether the limb area falls into it, thereby identifying whether there is clipping in the information frame.

[0112] As can be seen from the above embodiments, the torso model space of a digital human can be quickly obtained by using geometric principles, which is highly efficient, consumes low system resources, and is more efficient to implement.

[0113] Based on any embodiment of this application, please refer to Figure 6 Based on the position information of the skeletal key points in each information frame, motion anomaly detection is performed to determine the presence of motion anomaly segments describing abnormal motion phenomena, including:

[0114] Step S3100: Calculate the similarity of action feature vectors for every two temporally consecutive information frames in the action file. The action feature vectors include the positional information of the skeletal key points of the digital human limbs in the corresponding information frames.

[0115] When detecting anomalies such as uneven movement or rhythm in the motion file, similarity detection between information frames can be used. Specifically, two adjacent information frames are obtained from the motion file as target information frames. The data distance between the vector representations of the position information of each skeletal keypoint of the target information frame that occurs later in time and the target information frame that occurs earlier in time is calculated. The data distance is converted into a similarity score. Then, whether the transition between the two target information frames is uneven is determined based on whether the similarity score exceeds a preset threshold.

[0116] In practice, if the position information of all the skeletal joints in both target information frames is used to calculate the data distance, the computational load is high. Therefore, in this embodiment, for each target information frame, the position information of the four points corresponding to the left arm and the four points corresponding to the right arm, a total of eight points, can be used to construct the corresponding action feature vector. Usually, the action feature vector represents the image position presented by the arm movement in the corresponding information frame.

[0117] The essence of the motion feature vector is to represent the positional information of the corresponding limb parts of the digital human used to perform the action. Therefore, when selecting the points constituting the motion feature vector, the previously disclosed embodiment can be used as a reference, that is, taking the arm as the main focus, determining the elbow joint point, the wrist joint point and the center point between the two, and the center point between the shoulder joint point and the elbow joint point, for a total of four points. When the selected limb is the leg or foot, points are selected accordingly on the leg or foot part of the digital human. Of course, appropriate modifications can be made to the construction of the motion feature vector exemplified above, such as selecting only the positional information corresponding to the shoulder joint point, elbow joint point, and wrist joint point to construct the motion feature vector. Please note that the method of constructing the motion feature vector in this embodiment is also applicable to any other embodiment of this application.

[0118] Since the action feature vector has represented the position information of each selected point in vector form, and both target information frames have their corresponding action feature vectors, any algorithm such as cosine similarity, dot product between vectors, Euclidean distance, Pearson correlation coefficient, Jaccard coefficient, etc., can be used to calculate the data distance between the two action feature vectors, and the data distance can be converted into a similarity representation.

[0119] Step S3200: When the similarity exceeds a preset threshold, the abnormal action segment covering the two information frames is determined from the action file based on the two corresponding information frames.

[0120] It is easy to understand that in the motion file, the similarity between vectors of each pair of adjacent information frames can be calculated in the above way. This similarity represents the degree of similarity between the limb parts in the two information frames when performing actions. Therefore, the higher the similarity, the smoother and more natural the action change between the two information frames. The lower the similarity, the less smooth the action change between the two information frames.

[0121] Therefore, a threshold can be preset to check whether there is an anomaly of non-smoothness between two adjacent information frames. The similarity between any two adjacent information frames is compared with the threshold. When the similarity exceeds the threshold, it can be confirmed that there is an anomaly of non-smoothness between the two information frames, and thus the information frame pair consisting of the two information frames can be marked as an action anomaly segment. If there are multiple consecutive information frame pairs, these consecutive information frame pairs can be merged together to construct the same action anomaly segment.

[0122] Considering the need for smoothing multiple information frames during motion optimization, one embodiment can preset a smoothing factor to specify the number of information frames to be extended. When a non-smooth information frame pair is identified, multiple information frames can be obtained by extending the pair in the direction of earlier timing. The extended information frames and the original information frame pair together constitute an action anomaly segment. For example, if information frames F(m) to F(m+1) are non-smooth, and a smoothing factor α is preset, then information frames between F(m-α) and F(m+1) can be constructed as action anomaly segments. Typically, the smoothing factor α is set to an integer greater than 1 and less than 10.

[0123] As can be seen from the above embodiments, when detecting abnormal phenomena corresponding to whether the motion is smooth or not in the motion file, the motion feature vector of each information frame is constructed based on the position information of a small number of representative skeletal key points, taking each two temporally adjacent information frames as units. The similarity between the two information frames is determined based on the data distance between the motion feature vectors, thereby realizing similarity detection. The similarity is used to determine whether there is a non-smooth phenomenon between the two information frames. The computation is small and the computation speed is fast, which can meet the practical requirements of real-time processing and the detection effect is relatively accurate.

[0124] Based on any embodiment of this application, please refer to Figure 7 The set of information frames that are similar to the actions of each information frame in the abnormal action segment are matched from the material library, including:

[0125] Step S1321: Obtain the action feature vector corresponding to each information frame in the abnormal action segment. The action feature vector includes the position information of the skeletal key points of the digital human limb in the corresponding information frame.

[0126] The method for constructing the action feature vector can follow the same approach as in the previous embodiments of this application. Accordingly, for any abnormal action segment determined according to any of the previous embodiments, the action feature vector corresponding to each information frame can be obtained.

[0127] Step S1322: The abnormal action segment is compared with the information frame set of each material in the material library. The data distance between the action feature vectors of each two information frames corresponding to the time sequence is calculated based on the temporal correspondence of the information frames. The data distance of each information frame in the information frame set of each material is accumulated to obtain the similarity score of the material.

[0128] As mentioned above, this application provides a material library for storing sets of information frames. Each set of information frames constitutes a corresponding material, and each set of information frames can be used to control the digital human to perform corresponding actions. It is easy to understand that, due to this preparation, when the materials in the material library are used to control the digital human to perform movements and generate corresponding images, better motion image representation can be obtained compared to abnormal motion fragments. The material library can be stored as a local list, or in a remote database or local cache. To improve computational efficiency during use, each information frame in each material in the material library has its corresponding motion feature vector pre-constructed and stored according to the method for constructing motion feature vectors described in this application.

[0129] When it is necessary to determine whether an abnormal motion segment and a target material in the media library represent similar actions, the temporal correspondence between each information frame in the abnormal motion segment and the information frames of the target material is first established. Then, the data distance between the motion feature vector of each information frame in the abnormal motion segment and the motion feature vector of the corresponding information frame in the target material is calculated. The method for calculating the data distance can be the same as described above, and any known algorithm can be used. By calculating the data distance between the corresponding information frames of the abnormal motion segment and the target material according to the temporal relationship, multiple corresponding data distances can be obtained along the temporal sequence. Summing these data distances yields a similarity score. Thus, the similarity score between the abnormal motion segment and the target material based on the established temporal correspondence can be determined.

[0130] In one embodiment, the temporal alignment relationship between information frames of the same target material and information frames of the motion-abnormal segment is adjusted. Multiple similarity scores can be obtained based on different alignment relationships. Thus, multiple similarity scores can be obtained based on different temporal alignment relationships between the same target material and the same motion-abnormal segment. The alignment relationship corresponding to the highest similarity score is selected, and the corresponding information frame sequence of the target material under this alignment relationship is determined as the valid information frame set of the target material. The highest similarity score is used as the similarity score corresponding to this valid information frame set, which can be used for final filtering. Therefore, it is easy to understand that the number of information frames in the target material can be greater than the number of information frames in the motion-abnormal segment. That is, the motion-abnormal segment can find the most similar local segment in a target material based on different temporal alignment relationships, determine the information frame set corresponding to this local segment, and use it as a candidate to replace the motion-abnormal segment.

[0131] For each piece of material in the aforementioned material library, an information frame set can be determined according to the above process, and its corresponding similarity score can be obtained. Therefore, using the material library and applying the similarity detection algorithm implemented in this application based on data distance calculation between action feature vectors, the information frame set corresponding to each piece of material in the material library and its corresponding similarity score can be determined when corresponding to the same abnormal action segment. It is easy to understand that the higher the similarity score, the closer the action image obtained by the corresponding information frame set when the control data human produces an action is to the action image corresponding to the abnormal action segment.

[0132] Step S1323: Determine the set of information frames with the highest similarity score as the set of information frames similar to the abnormal action segment.

[0133] Since each material in the material library has a set of information frames for the abnormal action segment and provides a corresponding similarity score, each material can be optimized based on the similarity score. Specifically, the set of information frames corresponding to the material with the highest similarity score is determined as the target set of information frames for replacing the abnormal action segment. Subsequently, the abnormal action segment in the action file can be replaced with this target set of information frames.

[0134] As can be seen from the above embodiments, it is possible to match the set of information frames most similar to the abnormal action fragment from the prepared material library based on the similarity between actions, and use them to replace the abnormal action fragment, thereby realizing the batch replacement of information frames in the abnormal action fragment. This is suitable for scenarios where the abnormal action fragment contains a large number of frames. It can not only improve the correction efficiency of the abnormal action fragment, but also ensure that a higher quality action file can be obtained by using high-quality materials.

[0135] Based on any embodiment of this application, please refer to Figure 8 The frame interpolation operation based on the two reference frames includes:

[0136] Step S1331: Detect the similarity of the action feature vectors of the two reference frames, wherein the action feature vectors include the position information of the skeletal key points of the digital human limb parts in the corresponding information frames;

[0137] For two reference frames F(m) and F(n) requiring frame interpolation, their motion feature vectors can be obtained in the same way as in the previous embodiment, which will not be repeated here. Based on the two motion feature vectors corresponding to the two reference frames, the similarity detection algorithm described above can be applied to determine the similarity between them. The similarity can characterize the degree of deviation in the motion effect between the two reference frames, so that different smoothing algorithms can be applied according to the different degrees of deviation to correct the abnormal motion segments in which the two reference frames are located. In order to detect the degree of deviation, a preset threshold can be set. When the similarity reaches the preset threshold, it indicates that the motion deviation is small and fast operation can be applied, and step S1332 is executed; otherwise, it indicates that the motion deviation is large and more accurate frame interpolation processing is required in step S1333.

[0138] Step S1332: When the similarity reaches a preset threshold, perform frame interpolation on the two reference frames in the three-dimensional coordinate system;

[0139] When the similarity reaches a preset threshold, the frame interpolation operation between the two reference frames can be performed based on the 3D coordinate system corresponding to the 3D model of the digital human. Specifically, interpolation is performed based on the positional information of the skeletal key points in the two reference frames. The interpolation method can be linear interpolation or spherical interpolation, which can be flexibly implemented. When the similarity is already high, interpolation based on the 3D coordinate system results in relatively effective interpolated information frames.

[0140] Step S1333: When the similarity does not reach the preset threshold, the position information of the skeletal key points in the two reference frames is converted from three-dimensional coordinate information to Euler angles. The two reference frames are then interpolated based on the Euler angles, and after the interpolation operation, they are converted back to three-dimensional coordinate information.

[0141] When the similarity does not reach a preset threshold, if interpolation is still performed using a 3D coordinate system as a reference, it usually leads to problems in the generated information frame after application, such as large directional changes or abnormal movements after linear interpolation. Therefore, for cases where the similarity does not reach the preset threshold and the movement deviation is large, the position information of the skeletal key points of the two reference frames can be converted into Euler angle representation first. Then, interpolation is performed on the Euler angle representation, and after the interpolation is completed, the position information is converted back to the 3D coordinate system representation. Similarly, the specific interpolation method can be linear interpolation or spherical interpolation, which can be flexibly implemented. Performing interpolation on the two reference frames based on Euler angles can effectively preserve the rotational characteristics of each skeletal key point, thereby ensuring that the generated information frame can accurately reflect the movement changes.

[0142] As can be seen from the above embodiments, when smoothing abnormal motion segments that represent non-smooth motion, different coordinate systems are used to perform frame interpolation operations to adapt to the magnitude of the motion deviation described by the two reference frames within the abnormal motion segment, thereby achieving an effective balance between fast interpolation and accurate interpolation.

[0143] Please see Figure 9 According to one aspect of this application, a digital human motion control device includes a file acquisition module 1100, an anomaly detection module 1200, a motion correction module 1300, and a file application module 1400. The file acquisition module 1100 is configured to acquire motion files corresponding to control the movement of the digital human. These motion files include information frames corresponding to each image frame in the motion image of the digital human, and the information frames store the position information of the skeletal key points of the digital human. The anomaly detection module 1200 is configured to perform motion anomaly detection based on the position information of the skeletal key points in each information frame, determining the existence of motion anomaly segments describing motion anomalies. These motion anomaly segments include one or more temporally consecutive information frames. The motion correction module 1300 is configured to correct the motion anomaly segments to overcome the motion anomalies. The file application module 1400 is configured to apply the motion files to drive the movement of the digital human to generate the motion image.

[0144] Based on any embodiment of this application, the anomaly detection module 1200 includes: a region determination unit, configured to read at least one information frame in the action file as a target information frame, and determine the torso model space of the digital human based on the skeletal key points corresponding to the torso region of the digital human in the target information frame; a clipping detection unit, configured to detect whether the limb region falls into the torso model space based on the position information of the skeletal key points corresponding to the limb region in the target information frame, and determine that the target information frame has a clipping-corresponding action anomaly when it falls into the torso model space; and a segment construction unit, configured to construct one or more temporally consecutive target information frames with action anomalies into action anomaly segments.

[0145] Based on any embodiment of this application, the region determination unit includes: a triangulation determination subunit, configured to determine a triangular plane region based on two shoulder joint points corresponding to the torso region and three skeletal key points corresponding to the torso base point in the target information frame; a center point determination subunit, configured to calculate and determine the geometric center point of the triangular plane region and determine the coordinate information of the geometric center point in the three-dimensional coordinate system; and a three-dimensional region determination subunit, configured to expand in three dimensions based on the coordinate information of the geometric center point to determine the torso model space of the digital human.

[0146] Based on any embodiment of this application, the anomaly detection module 1200 includes: a similarity detection unit, configured to continuously calculate the similarity of action feature vectors of every two temporally consecutive information frames in the action file, wherein the action feature vectors include the position information of skeletal key points of digital human limb parts in the corresponding information frames; and a segment acquisition unit, configured to determine, based on the corresponding two information frames, an abnormal action segment covering the two information frames from the action file when the similarity exceeds a preset threshold.

[0147] Based on any embodiment of this application, the motion correction module 1300 includes any one or more of the following units: a single-frame correction unit, configured to adjust the position information of the skeletal key points falling into the torso model space based on the torso model space corresponding to the information frame when the motion abnormal segment includes a single information frame, so that the corresponding limb area does not overlap with the torso model space; a clipping correction unit, configured to match a set of information frames similar to the actions of each information frame in the motion abnormal segment from the material library when the motion abnormal segment includes a first number of information frames, and replace the motion abnormal segment in the motion file with the set of information frames; and a smoothing correction unit, configured to use the information frames in the motion file that are before and after the motion abnormal segment as reference frames, perform frame interpolation based on the two reference frames to generate a set of information frames, and replace the motion abnormal segment in the motion file with the set of information frames.

[0148] Based on any embodiment of this application, the clipping correction unit includes: a vectorization subunit, configured to acquire motion feature vectors corresponding to each information frame in the motion anomaly segment, wherein the motion feature vectors include position information of skeletal key points of digital human limb parts in the corresponding information frames; a matching operation subunit, configured to calculate the data distance between the motion feature vectors of every two temporally corresponding information frames and the information frame sets of each material in the material library based on the temporal correspondence of the information frames, and accumulate the data distances of each information frame in the information frame set of each material to obtain the similarity score of that material; and a set optimization subunit, configured to determine the information frame set of the material with the highest similarity score as the information frame set similar to the motion anomaly segment.

[0149] Based on any embodiment of this application, the smoothing correction unit includes: a similarity operation subunit, configured to detect the similarity of action feature vectors of the two reference frames, wherein the action feature vectors include the position information of skeletal key points of digital human limb parts in the corresponding information frames; an efficient frame interpolation subunit, configured to perform frame interpolation operation on the two reference frames in a three-dimensional coordinate system when the similarity reaches a preset threshold; and a precise frame interpolation subunit, configured to convert the position information of skeletal key points in the two reference frames from three-dimensional coordinate information to Euler angles when the similarity does not reach the preset threshold, perform frame interpolation operation on the two reference frames based on Euler angles, and convert them back to three-dimensional coordinate information after the frame interpolation operation.

[0150] Another embodiment of this application provides a digital human motion control device. For example... Figure 10The diagram shows the internal structure of a digital human motion control device. This device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. The computer-readable, non-volatile storage medium stores an operating system, a database, and computer-readable instructions. The database can store information sequences, and when executed by the processor, these computer-readable instructions enable the processor to implement a digital human motion control method.

[0151] The processor of the digital human motion control device provides computing and control capabilities, supporting the operation of the entire device. The device's memory can store computer-readable instructions, which, when executed by the processor, cause the processor to perform the digital human motion control method of this application. The network interface of the device is used for communication with a terminal.

[0152] Those skilled in the art will understand that Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the digital human motion control device to which the present application is applied. A specific digital human motion control device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0153] In this embodiment, the processor is used to execute... Figure 9 The specific functions of each module are described, and the memory stores the program code and various data required to execute the above modules or sub-modules. The network interface is used to realize data transmission between user terminals or servers. The non-volatile readable storage medium in this embodiment stores the program code and data required to execute all modules in the digital human motion control device of this application, and the server can call the server's program code and data to execute the functions of all modules.

[0154] This application also provides a non-volatile readable storage medium storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps of the digital human motion control method of any embodiment of this application.

[0155] This application also provides a computer program product, including a computer program / instructions that, when executed by one or more processors, implement the steps of the method described in any embodiment of this application.

[0156] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The aforementioned storage medium can be a computer-readable storage medium such as a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM).

[0157] In summary, this application performs motion anomaly detection on pre-generated motion files used to drive digital human motion, efficiently and quickly optimizing the motion files. This results in smoother and more natural motion images of the digital human after the motion files are used to drive the digital human's motion. It avoids relying entirely on the reasoning ability learned by machine learning models, effectively controlling the cost of technical implementation, while achieving efficient and high-quality processing results. When the motion files optimized by the technical solution of this application are used to control the motion images of digital humans, a more natural and smooth digital human motion effect can be obtained. This can serve various application scenarios such as virtual digital human live streaming and virtual human music performances, paving the way for the metaverse technology era.

Claims

1. A method for controlling the motion of a digital human, characterized in that, include: Obtain motion files corresponding to control the movement of the digital human. These motion files include information frames corresponding to each image frame in the motion image of the digital human. The information frames store the position information of the key points of the digital human's skeleton. Action anomaly detection is performed based on the positional information of skeletal key points in each information frame to determine the existence of action anomaly segments describing action anomalies. These action anomaly segments include one or more temporally consecutive information frames. The process includes: reading at least one information frame from the action file as a target information frame; determining the torso model space of the digital human based on the skeletal key points corresponding to the torso region in the target information frame; detecting whether the limb region falls into the torso model space based on the positional information of the skeletal key points corresponding to the limb region in the target information frame; determining that the target information frame exhibits a clipping-related action anomaly based on the skeletal key points in the limb region of the target information frame; constructing one or more temporally consecutive target information frames with action anomalies as action anomaly segments; or, continuously calculating the similarity of action feature vectors between every two temporally consecutive information frames in the action file, where the action feature vectors include the positional information of skeletal key points of the digital human limb parts in the corresponding information frames; and when the similarity exceeds a preset threshold, expanding from the action file based on the corresponding two information frames to determine action anomaly segments covering those two information frames. Correcting the abnormal motion segment to overcome the abnormal motion phenomenon includes at least one of the following: when the abnormal motion segment includes a single information frame, adjusting the position information of the skeletal key points falling into the torso model space based on the torso model space corresponding to the information frame, so that the corresponding limb area does not overlap with the torso model space; when the abnormal motion segment includes a first number of multiple information frames, matching a set of information frames similar to the actions of each information frame in the abnormal motion segment from the material library, and replacing the abnormal motion segment in the motion file with the set of information frames; when the abnormal motion segment includes a second number of multiple information frames, using the information frames in the motion file that are before and after the abnormal motion segment as reference frames, performing frame interpolation based on the two reference frames to generate a set of information frames, and replacing the abnormal motion segment in the motion file with the set of information frames. The motion file is used to drive the digital human to move, thereby generating the motion image.

2. The digital human motion control method according to claim 1, characterized in that, The torso model space of the digital human is determined based on the skeletal key points corresponding to the torso region in the target information frame, including: The triangular plane region is determined based on the two shoulder joint points corresponding to the torso region and the three skeletal key points corresponding to the torso base point in the target information frame. Calculate and determine the geometric center point of the triangular plane region, and determine the coordinate information of the geometric center point in the three-dimensional coordinate system; Based on the coordinate information of the geometric center point, the three-dimensional expansion is performed to determine the torso model space of the digital human.

3. The digital human motion control method according to claim 1, characterized in that, The set of information frames that are similar to the actions of each information frame in the abnormal action segment is matched from the material library, including: Obtain the motion feature vector corresponding to each information frame in the abnormal motion segment, wherein the motion feature vector includes the position information of the skeletal key points of the digital human limb in the corresponding information frame; The abnormal action fragment is compared with the information frame set of each material in the material library. The data distance between the action feature vectors of each two information frames corresponding to the time sequence is calculated based on the temporal correspondence of the information frames. The data distances of each information frame in the information frame set of each material are accumulated to obtain the similarity score of the material. The set of information frames with the highest similarity score is determined as the set of information frames similar to the abnormal action segment.

4. The digital human motion control method according to any one of claims 1 to 3, characterized in that, The frame interpolation operation based on the two reference frames includes: The similarity of the motion feature vectors of the two reference frames is detected. The motion feature vectors include the positional information of the skeletal key points of the digital human limbs in the corresponding information frames. When the similarity reaches a preset threshold, a frame interpolation operation is performed on the two reference frames in a three-dimensional coordinate system. When the similarity does not reach the preset threshold, the position information of the skeletal key points in the two reference frames is converted from three-dimensional coordinate information to Euler angles. Based on the Euler angles, the two reference frames are interpolated and then converted back to three-dimensional coordinate information.

5. A digital human motion control device, characterized in that, include: The file acquisition module is configured to acquire motion files corresponding to the control of the digital human's movement. The motion files include information frames corresponding to each image frame in the motion image of the digital human. The information frames store the position information of the key points of the digital human's skeleton. An anomaly detection module is configured to perform motion anomaly detection based on the position information of skeletal key points in each information frame, and to determine the existence of motion anomaly segments describing motion anomalies. The motion anomaly segments include one or more temporally consecutive information frames. The module includes: a region determination unit, configured to read at least one information frame from the motion file as a target information frame, and determine the torso model space of the digital human based on the skeletal key points corresponding to the torso region of the digital human in the target information frame; and a clipping detection unit, configured to detect whether a limb region falls into the torso model space based on the position information of the skeletal key points corresponding to the limb region in the target information frame. When entering the torso model space, it is determined that the target information frame has a corresponding action anomaly phenomenon; the segment construction unit is configured to construct one or more temporally consecutive target information frames with action anomalies into action anomaly segments; or it includes: a similarity detection unit, configured to continuously calculate the similarity of action feature vectors of every two temporally consecutive information frames in the action file, the action feature vectors including the position information of the skeletal key points of the digital human limb parts in the corresponding information frames; the segment acquisition unit is configured to, when the similarity exceeds a preset threshold, expand from the action file based on the corresponding two information frames to determine the action anomaly segment covering the two information frames. The motion correction module is configured to correct the abnormal motion segments to overcome the abnormal motion phenomenon, including at least one of the following: a single-frame correction unit, configured to, when the abnormal motion segment includes a single information frame, adjust the position information of the skeletal key points falling into the torso model space based on the torso model space corresponding to the information frame, so that the corresponding limb area does not overlap with the torso model space; a clipping correction unit, configured to, when the abnormal motion segment includes a first number of multiple information frames, match a set of information frames from the material library that are similar to the motion of each information frame in the abnormal motion segment, and replace the abnormal motion segment in the motion file with the set of information frames; and a smoothing correction unit, configured to, when the abnormal motion segment includes a second number of multiple information frames, use the information frames in the motion file that are before and after the abnormal motion segment as reference frames, perform frame interpolation based on the two reference frames to generate a set of information frames, and replace the abnormal motion segment in the motion file with the set of information frames. The file application module is configured to use the motion file to drive the digital human to move, thereby generating the motion image.

6. The digital human motion control device according to claim 5, characterized in that, The region determination unit includes: The triangulation subunit is set to determine the triangular plane region based on the two shoulder joint points corresponding to the torso region and the three skeletal key points corresponding to the torso base point in the target information frame. The center point determination sub-unit is set to calculate and determine the geometric center point of the triangular plane region and determine the coordinate information of the geometric center point in the three-dimensional coordinate system. The three-dimensional region is defined by sub-units, which are set to expand in three dimensions based on the coordinate information of the geometric center point to determine the torso model space of the digital human.

7. The digital human motion control device according to claim 5, characterized in that, The molding correction unit includes: The vectorization subunit is configured to acquire the action feature vectors corresponding to each information frame in the abnormal action segment, wherein the action feature vectors include the position information of the skeletal key points of the digital human limb parts in the corresponding information frames; The matching operation subunit is configured to compare the abnormal action segment with the information frame set of each material in the material library, calculate the data distance between the action feature vectors of each two information frames corresponding to the time sequence based on the temporal correspondence of the information frames, and accumulate the data distances of each information frame in the information frame set of each material to obtain the similarity score of the material. The set of preferred subunits is set up to determine the set of information frames that are similar to the material with the highest similarity score as the set of information frames that are similar to the abnormal action segment.

8. The digital human motion control device according to any one of claims 5 to 7, characterized in that, The smoothing correction unit includes: The similarity operation subunit is configured to detect the similarity of the action feature vectors of the two reference frames, wherein the action feature vectors include the position information of the skeletal key points of the digital human limb parts in the corresponding information frames; The high-efficiency frame interpolation subunit is configured to perform frame interpolation operations on two reference frames in a three-dimensional coordinate system when the similarity reaches a preset threshold. The precise frame interpolation subunit is configured to convert the position information of the skeletal key points in the two reference frames from three-dimensional coordinate information to Euler angles when the similarity does not reach a preset threshold, perform frame interpolation operation on the two reference frames based on the Euler angles, and convert them back to three-dimensional coordinate information after the frame interpolation operation.

9. A digital human motion control device, comprising a central processing unit and a memory, characterized in that, The central processing unit is used to invoke and run a computer program stored in the memory to perform the steps of the method as described in any one of claims 1 to 4.

10. A non-volatile readable storage medium, characterized in that, It stores, in the form of computer-readable instructions, a computer program implemented according to any one of claims 1 to 4, which, when invoked by a computer, executes the steps included in the corresponding method.

Images