Human body motion sequence completion method, device, equipment, and storage medium

Abstract

The embodiments of the present application provide a method and apparatus, device, and storage medium for completing a human motion sequence, which belongs to the field of artificial intelligence technology. The method includes: obtaining an incomplete initial motion sequence of a target object; inputting the initial motion sequence into a pre-constructed motion sequence completion model, the motion sequence completion model includes a sequence repair sub-model and a context optimization sub-model; performing sequence repair on the initial motion sequence according to the sequence repair sub-model to obtain a repaired motion sequence; performing sequence optimization on the repaired motion sequence according to the context optimization sub-model to obtain a target motion sequence with complete completion of the target object. The embodiments of the present application can consider the missing of motion sequences from different dimensions, and at the same time consider the intrinsic connection between the joints of the human skeleton, so as to generate a better human motion sequence completion result.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method and apparatus, device, and storage medium for completing a human motion sequence. Background Art

[0002] Human motion sequence completion refers to the process of restoring a complete human motion sequence from a partially occluded or damaged human motion sequence. In real-world scenarios, due to object occlusion, sensor inaccuracy, and poor robustness of the data processing system, the video acquisition system can usually only capture part of the human body's motion, which is not conducive to subsequent motion recognition and motion prediction operations. Therefore, the research on human motion completion tasks has great application value in many real-world scenarios such as autonomous driving, unattended monitoring systems, and human-computer interaction.

[0003] At present, in order to solve the problem of human motion completion, the existing technology usually adopts a human motion sequence completion method that obtains latent vectors by sampling from Gaussian processes and uses graph neural networks for posture synthesis, thereby realizing a bidirectional transformation between latent space and skeleton space. Alternatively, a framework based on a generative adversarial network is constructed, and the corresponding generator and discriminator are designed based on the data features of the skeleton to use a repair framework to complete the action sequence. However, these methods have some disadvantages. The first method can only complete the missing parts in a single time dimension, resulting in unsatisfactory completion effects; the second method only considers the Euclidean structure of the skeleton data, ignoring the intrinsic connection between the joints of the human skeleton, resulting in unsatisfactory completion effects. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a human motion sequence completion method and apparatus, equipment, and storage medium, which can consider the missing of motion sequences from different dimensions and simultaneously consider the intrinsic connections between the joints of the human skeleton, thereby generating a human motion sequence completion result with better effect.

[0005] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a method for completing a human motion sequence, the method comprising:

[0006] Obtaining an incomplete initial motion sequence of the target object;

[0007] Inputting the initial motion sequence into a pre-built motion sequence completion model, wherein the motion sequence completion model includes a sequence repair sub-model and a context optimization sub-model;

[0008] Performing sequence repair on the initial motion sequence according to the sequence repair sub-model to obtain a repaired motion sequence;

[0009] The repaired motion sequence is sequence optimized according to the context optimization sub-model to obtain a target motion sequence with the target object completed.

[0010] In some embodiments, before inputting the initial motion sequence into a pre-built motion sequence completion model, the method further includes:

[0011] Constructing the motion sequence completion model specifically includes:

[0012] Acquire an incomplete initial sample motion sequence of a sample object and a complete target sample motion sequence of the sample object;

[0013] Performing sequence normalization on the initial sample motion sequence to obtain an initial sample motion image;

[0014] Performing image masking on the initial sample motion image according to a preset mask image to obtain a masked sample motion image;

[0015] Inputting the mask sample motion image into the sequence repair sub-model to perform sequence repair, thereby obtaining a repaired sample motion sequence;

[0016] Determining restoration loss data according to the restored sample motion sequence and the initial sample motion image, and adjusting parameters of the sequence restoration sub-model according to the restoration loss data;

[0017] Inputting the preset mask image and the restored sample motion sequence into the context optimization sub-model for sequence optimization to obtain an optimized sample sequence;

[0018] Determining three-dimensional optimization loss data according to the target sample motion sequence and the optimized sample sequence, and adjusting parameters of the context optimization sub-model according to the three-dimensional optimization loss data;

[0019] The motion sequence completion model is obtained according to the sequence repair sub-model and the context optimization sub-model after parameter adjustment.

[0020] In some embodiments, the sequence repair sub-model includes a repair encoder and a repair decoder, and inputting the mask sample motion image into the sequence repair sub-model for sequence repair to obtain a repaired sample motion sequence includes:

[0021] encoding the masked sample motion image according to the restoration encoder to obtain sample implicitly encoded data;

[0022] Decoding the sample implicitly encoded data according to the repair decoder to obtain a repaired sample motion image;

[0023] Performing inverse normalization on the restored sample motion image to obtain the restored sample motion sequence;

[0024] The determining of restoration loss data according to the restored sample motion sequence and the initial sample motion image comprises:

[0025] Restoration loss data is determined according to the restored sample motion image and the initial sample motion image.

[0026] In some embodiments, determining the restoration loss data according to the restored sample motion image and the initial sample motion image includes:

[0027] Calculating according to the restored sample motion image and the initial sample motion image to obtain reconstruction loss data;

[0028] Inputting the restored sample motion image and the initial sample motion image into a target activation layer of a pre-trained perceptual neural network respectively, to obtain first output data corresponding to the restored sample motion image and second output data corresponding to the initial sample motion image;

[0029] Calculate according to the loss weight parameter preset by the sample object in the target activation layer, the first output data and the second output data to obtain the perception loss data;

[0030] Calculate according to a preset generation loss function and the restored sample motion image to obtain generation loss data;

[0031] Calculate according to a preset discrimination loss function, the restored sample motion image and the initial sample motion image to obtain discrimination loss data;

[0032] The restoration loss data is obtained by performing weighted calculation according to the reconstruction loss data, the perception loss data, the generation loss data and the discrimination loss data.

[0033] In some embodiments, determining the three-dimensional optimization loss data according to the target sample motion sequence and the optimized sample sequence includes:

[0034] Acquire target three-dimensional coordinate data of each sample joint point in the target sample motion sequence, wherein the target three-dimensional coordinate data includes a target first-dimensional coordinate value, a target second-dimensional coordinate value, and a target third-dimensional coordinate value;

[0035] Obtaining optimized three-dimensional coordinate data of each sample joint point in the optimized sample sequence, wherein the optimized three-dimensional coordinate data includes an optimized first-dimensional coordinate value, an optimized second-dimensional coordinate value, and an optimized third-dimensional coordinate value;

[0036] Performing coordinate calculation on the target first-dimensional coordinate value and the optimized first-dimensional coordinate value to obtain a first-dimensional loss value;

[0037] Performing coordinate calculation on the target second-dimensional coordinate value and the optimized second-dimensional coordinate value to obtain a second-dimensional loss value;

[0038] Performing coordinate calculation on the target third-dimensional coordinate value and the optimized third-dimensional coordinate value to obtain a third-dimensional loss value;

[0039] The first-dimensional loss value, the second-dimensional loss value, and the third-dimensional loss value are averaged to obtain the three-dimensional optimization loss data.

[0040] In some embodiments, the context optimization sub-model includes a first graph convolution block connected in series, a second graph convolution block with a preset number of residual connections, and a first graph convolution layer, and the preset mask image and the restored sample motion sequence are input into the context optimization sub-model for sequence optimization to obtain an optimized sample sequence, including:

[0041] Performing convolution processing on the restored sample motion sequence according to the first graph convolution block to obtain a first optimized sequence;

[0042] According to the second graph convolution block, sequentially perform the preset number of convolution processes on the first optimization sequence, and use the last output sequence of the second graph convolution block as the second optimization sequence;

[0043] Performing convolution processing on the second optimized sequence according to the first graph convolution layer to obtain a third optimized sequence;

[0044] Performing image decomposition on the preset mask image to obtain a mask sequence;

[0045] performing sequence updating on the third optimized sequence according to the mask sequence;

[0046] The restored sample motion sequence and the updated third optimized sequence are added to obtain the optimized sample sequence.

[0047] In some embodiments, the first graph convolution block and the second graph convolution block have the same structure, and the first graph convolution block includes a second graph convolution layer, an activation layer, and a random inactivation layer, and the convolution processing of the restored sample motion sequence according to the first graph convolution block to obtain a first optimized sequence includes:

[0048] performing convolution processing on the restored sample motion sequence according to the second graph convolution layer to obtain a convolution sequence;

[0049] Performing activation processing on the convolution sequence according to the activation layer to obtain an activation sequence;

[0050] Regularization is performed on the activation sequence according to the random dropout layer to obtain the first optimized sequence.

[0051] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application proposes a human body motion sequence completion device, the device comprising:

[0052] A sequence acquisition module, used to acquire an incomplete initial motion sequence of a target object;

[0053] An input module, used for inputting the initial motion sequence into a pre-built motion sequence completion model, wherein the motion sequence completion model includes a sequence repair sub-model and a context optimization sub-model;

[0054] A sequence repair module, used for performing sequence repair on the initial motion sequence according to the sequence repair sub-model to obtain a repaired motion sequence;

[0055] A sequence optimization module is used to perform sequence optimization on the repaired motion sequence according to the context optimization sub-model to obtain a target motion sequence that completes the target object.

[0056] To achieve the above objective, a third aspect of an embodiment of the present application provides a computer device, including:

[0057] at least one memory;

[0058] at least one processor;

[0059] at least one computer program;

[0060] The at least one computer program is stored in the at least one memory, and the at least one processor executes the at least one computer program to implement the method described in the first aspect above.

[0061] To achieve the above-mentioned purpose, the fourth aspect of the embodiments of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program is used to enable a computer to execute the method described in the first aspect above.

[0062] The human motion sequence completion method, device, equipment, and storage medium proposed in the embodiment of the present application, first, obtain an incomplete initial motion sequence of the target object. The initial motion sequence is input into a pre-constructed motion sequence completion model, and the motion sequence completion model includes a sequence repair sub-model and a context optimization sub-model. The initial motion sequence is sequence repaired according to the sequence repair sub-model to obtain a repaired motion sequence. In order to consider the missing of the motion sequence from different dimensions and simultaneously consider the intrinsic connection between the joints of the human skeleton, the embodiment of the present application performs sequence optimization on the repaired motion sequence according to the context optimization sub-model to obtain a target motion sequence with complete completion of the target object. Therefore, the motion sequence completion model of the embodiment of the present application can consider the missing of the motion sequence from different dimensions and simultaneously consider the intrinsic connection between the joints of the human skeleton by combining the sequence repair sub-model and the context optimization sub-model, thereby being able to generate a better human motion sequence completion result. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 It is a schematic diagram of motion sequence completion in the time dimension of related technologies;

[0064] Figure 2 is a flow chart of a human motion sequence completion method provided by an embodiment of the present application;

[0065] Figure 3 is another flow chart of the human body motion sequence completion method provided by an embodiment of the present application;

[0066] Figure 4 yes Figure 3 Flow chart of step S340 in FIG.

[0067] Figure 5 yes Figure 3 Flow chart of step S350 in FIG.

[0068] Figure 6 yes Figure 3 Flow chart of step S360 in FIG.

[0069] Figure 7 It is a structural diagram of the context optimization sub-model provided in the embodiment of the present application;

[0070] Figure 8 yes Figure 6 Flow chart of step S610 in FIG.

[0071] Fig. 9 It is a schematic diagram of the structure of the graph convolution block provided in an embodiment of the present application;

[0072] Fig.10 yes Figure 3 Flow chart of step S370 in FIG.

[0073] Fig.11 is a schematic diagram of an embodiment of the present application that simultaneously completes a motion sequence in the time-space dimension;

[0074] Fig.12 is a schematic diagram of the overall model training of the motion sequence completion model of an embodiment of the present application;

[0075] Fig.13 is a structural schematic diagram of a human motion sequence completion device provided in an embodiment of the present application;

[0076] Fig.14 It is a schematic diagram of the hardware structure of the computer device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0077] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0078] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0079] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0080] First, some nouns involved in this application are analyzed:

[0081] Artificial Intelligence (AI): It is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science. AI attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robots, language recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0082] Motion sequence: refers to the sequence of actions or postures of an object (such as a human body) over a period of time. Motion sequences can describe the motion state and posture changes of an object at different time points. The motion sequence of an object can usually be obtained by using sensors, cameras or other motion capture devices. For example, the acceleration, angular velocity and posture information of an object can be obtained using an inertial measurement unit (IMU), and the position information of the object's joints can be obtained using a depth camera. The motion sequence of an object can be applied to application scenarios such as object motion recognition, posture estimation, and motion analysis. By analyzing the motion sequence of an object, the characteristics and patterns of the motion can be extracted, thereby realizing the understanding and analysis of the object's actions and behaviors. This is of great significance to fields such as human-computer interaction, virtual reality, and sports medicine.

[0083] Euclidean structure: refers to the geometric structure in three-dimensional Euclidean space. Euclidean structure is based on Euclidean geometry and involves points, lines, planes and the relationship between them. In three-dimensional Euclidean structure, points are the most basic elements used to represent positions. Lines are determined by two points and represent straight line segments connecting two points. Planes are determined by three or more points and represent an infinitely extended plane. Euclidean structure has a wide range of applications in geometry, computer graphics, computer vision and other fields. It is a geometric model based on Euclidean space and is used to describe and analyze the shape, position and movement of three-dimensional objects.

[0084] Non-Euclidean structure: refers to a geometric structure that does not satisfy the parallel axiom in Euclidean geometry. In non-Euclidean geometry, there are other geometric rules and properties that are different from Euclidean geometry.

[0085] Human motion sequence completion refers to the process of restoring a complete human motion sequence from a partially occluded or damaged human motion sequence. In real-world scenarios, due to object occlusion, sensor inaccuracy, and poor robustness of the data processing system, the video acquisition system can usually only capture part of the human body's motion, which is not conducive to subsequent motion recognition and motion prediction operations. Therefore, the research on human motion completion tasks has great application value in many real-world scenarios such as autonomous driving, unattended monitoring systems, and human-computer interaction.

[0086] At present, in order to solve the problem of human motion completion, the existing technology usually adopts a human motion sequence completion method that obtains latent vectors by sampling from a Gaussian process and uses a graph neural network for posture synthesis, thereby realizing a bidirectional transformation between the latent space and the skeleton space. Figure 1The figure is a schematic diagram of the motion sequence completion in the time dimension of the related technology. Figure 1 The thick line in the dashed box represents the completed motion sequence, and the thin line outside the dashed box represents the collected motion sequence. Alternatively, a framework based on generative adversarial networks is constructed, and the corresponding generator and discriminator are designed based on the data features of the skeleton to use the repair framework to complete the action sequence. However, these methods have some disadvantages. The first method can only complete the missing parts in a single time dimension, but cannot effectively solve the arbitrary missing parts in the spatiotemporal dimension, resulting in unsatisfactory completion effect; the second method only considers the Euclidean structure of the skeleton data, ignoring the intrinsic connection between the joints of the human skeleton, that is, it cannot effectively utilize the global context information, resulting in unsatisfactory completion effect.

[0087] Based on this, the embodiments of the present application provide a human motion sequence completion method and apparatus, equipment, and storage medium, which can consider the missing motion sequence from different dimensions and simultaneously consider the intrinsic connection between the joints of the human skeleton, thereby generating a human motion sequence completion result with better effect.

[0088] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0089] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0090] The human motion sequence completion method provided in the embodiment of the present application relates to the field of artificial intelligence technology. The human motion sequence completion method provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server side, and can also be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, and can also be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (Content Delivery Network, CDN) and cloud servers for basic cloud computing services such as big data and artificial intelligence platforms; the software can be an application that implements the human motion sequence completion method, etc., but is not limited to the above forms.

[0091] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network personal computers (Personal Computer, PC), minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0092] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0093] See also Figure 2 , Figure 2It is an optional flowchart of the human motion sequence completion method provided in an embodiment of the present application. Figure 2 The method may specifically include but is not limited to steps S110 to S140. Figure 2 These four steps are introduced in detail.

[0094] Step S210, obtaining an incomplete initial motion sequence of the target object;

[0095] Step S220, inputting the initial motion sequence into a pre-built motion sequence completion model, where the motion sequence completion model includes a sequence repair sub-model and a context optimization sub-model;

[0096] Step S230, performing sequence repair on the initial motion sequence according to the sequence repair sub-model to obtain a repaired motion sequence;

[0097] Step S240 , performing sequence optimization on the restored motion sequence according to the context optimization sub-model to obtain a target motion sequence that completes the target object.

[0098] In step S210 of some embodiments, in many real-life scenarios such as autonomous driving, unattended monitoring systems, and human-computer interaction, sometimes the human body is obscured by the surrounding environment or people nearby, and the video acquisition device can only capture part of the human body, and cannot obtain the overall movement of the human object, so human motion completion is required. For example, in the scenarios of autonomous driving and human-computer interaction, the camera may only be able to capture the upper body of the target object, thereby lacking the display of the motion sequence of the lower body of the target object. The target object refers to a real person who needs to be motion-completed. The initial motion sequence refers to the motion sequence of the target object that is obscured within a period of time captured by an ordinary camera or a depth camera.

[0099] In step S220 of some embodiments, in order to consider the missing of motion sequences from different dimensions and the intrinsic connection between the joints of the human skeleton at the same time, the embodiment of the present application constructs a motion sequence completion model including a two-stage repair optimization, and the motion sequence completion model includes a sequence repair sub-model and a context optimization sub-model. Among them, the sequence repair sub-model of the first stage can realize the preliminary repair of the motion sequence in two dimensions of time and space. The context optimization sub-model of the second stage can establish an implicit mapping and association between intact and damaged joints, and effectively optimize the results generated by the sequence repair sub-model of the first stage.

[0100] Therefore, the embodiment of the present application can simultaneously utilize the global modeling capability of the sequence repair sub-model and the modeling capability of the context optimization sub-model for the graph nodes in the first stage, and can complete the human body motion sequence in the form of occluded or damaged skeletons, thereby improving the accuracy of human body motion sequence completion.

[0101] See also Figure 3 , Figure 3 is another optional flow chart of the human motion sequence completion method provided by the embodiment of the present application. In some embodiments of the present application, before step S210, the human motion sequence completion method of the present application may further include the step of: constructing a motion sequence completion model. This step may specifically include but is not limited to steps S310 to S380. Figure 3 These eight steps are introduced in detail.

[0102] Step S310, obtaining an incomplete initial sample motion sequence of the sample object and a complete target sample motion sequence of the sample object;

[0103] Step S320, performing sequence normalization on the initial sample motion sequence to obtain an initial sample motion image;

[0104] Step S330, performing image masking on the initial sample motion image according to a preset mask image to obtain a masked sample motion image;

[0105] Step S340, inputting the masked sample motion image into the sequence restoration sub-model to perform sequence restoration, and obtaining a restored sample motion sequence;

[0106] Step S350, determining restoration loss data according to the restored sample motion sequence and the initial sample motion image, and adjusting parameters of the sequence restoration sub-model according to the restoration loss data;

[0107] Step S360, inputting the preset mask image and the restored sample motion sequence into the context optimization sub-model for sequence optimization to obtain an optimized sample sequence;

[0108] Step S370, determining three-dimensional optimization loss data according to the target sample motion sequence and the optimized sample sequence, and adjusting parameters of the context optimization sub-model according to the three-dimensional optimization loss data;

[0109] Step S380, obtaining a motion sequence completion model according to the sequence repair sub-model and the context optimization sub-model after parameter adjustment.

[0110] In step S310 of some embodiments, the sample object refers to a real person who needs to be motion completed. The initial sample motion sequence refers to a motion sequence when the sample object includes occlusion within a period of time captured by an ordinary camera or a depth camera. The target sample motion sequence refers to a motion sequence after the occluded part of the initial sample motion sequence is accurately completed. During the training process of the model, the target sample motion sequence is equivalent to a sample label of the initial sample motion sequence to determine the model's completion of the initial sample motion sequence.

[0111] It should be noted that the model training process includes multiple training samples, each of which includes an incomplete initial sample motion sequence of the sample object and a complete target sample motion sequence of the sample object. The training samples can be obtained by writing a web crawler or script program to crawl data in a targeted manner.

[0112] It should be noted that the initial sample motion sequence of the embodiment of the present application may be a sequence change of skeleton points extracted from a collected video data set of human motion using a key point detection algorithm.

[0113] In step S320 of some embodiments, in order to achieve preliminary repair of the motion sequence in both time and space dimensions, the embodiments of the present application use image repair to repair the damaged part of the image in the first stage. Specifically, first, the initial sample motion sequence is sequence normalized to obtain an initial sample motion image, denoted as N, that is, after sequence normalization, a single image corresponding to the motion skeleton sequence of the object is obtained. The embodiments of the present application can accelerate the training convergence speed of the model by normalizing the motion sequence.

[0114] It should be noted that the initial sample motion sequence of this application can be expressed as Among them, T represents the number of frames in the motion sequence in the time dimension, J represents the number of object joints in the spatial dimension, and 3 represents that each object joint includes values ​​in three coordinate dimensions in the Cartesian coordinate system x, y, and z. It can be seen that the value of each object joint corresponds to a three-dimensional coordinate in the Cartesian coordinate system.

[0115] It should be noted that the embodiment of the present application can use the maximum and minimum values ​​of each object joint point in each coordinate system dimension in the initial sample motion sequence to perform sequence normalization on the initial sample motion sequence.

[0116] The calculation of the normalized value corresponding to each object joint point in the initial sample motion sequence is shown in the following formula (1).

[0117]

[0118] Among them, i represents the Cartesian coordinate value of the corresponding object joint point in the initial sample motion sequence, N i represents the normalized value of each object joint point in the initial sample motion sequence in the initial sample motion image, S i Represents the motion sequence value of the corresponding object joint point in the corresponding frame in the initial sample motion sequence.

[0119] In step S330 of some embodiments, since the numbers of the object joints, such as the human body joints, are fixedly set, the horizontal axis in the motion image represents the time dimension of the motion sequence, and the vertical axis in the motion image represents the joint dimension of the motion sequence. Similarly, the horizontal axis of the preset mask image M also represents the time dimension of the motion sequence, and the vertical axis of the preset mask image also represents the joint dimension of the motion sequence. Therefore, the values ​​corresponding to the positions of the joints to be blocked in the preset mask image M are set to 0, and the values ​​corresponding to the remaining positions are set to 1. The preset mask image M and the corresponding element values ​​in the initial sample motion image N are multiplied to obtain a mask sample motion image, denoted as P. The values ​​corresponding to the positions of the joints to be blocked in the mask sample motion image P are 0, and the values ​​corresponding to the remaining positions are the same as the values ​​in the initial sample motion image N.

[0120] It should be noted that the embodiments of the present application can flexibly construct corresponding preset mask images according to the specific positions of the joints that need to be occluded in the skeleton, or different types of preset mask images can be generated according to different occlusion situations.

[0121] It should be noted that, in the training stage, the preset mask image can be set to a randomly generated style, and the ratio of the mask area of ​​the preset mask image to the overall image can be set to (0.15, 0.65). When, assuming that the horizontal axis and the vertical axis of the overall image are both represented as 1, according to the ratio (0.15, 0.65), it can be determined that the intervals corresponding to (0.15, 0.65) on the horizontal axis and the vertical axis can constitute a rectangular mask area.

[0122] In step S340 of some embodiments, the present application embodiment will input the mask sample motion image P and the initial sample motion image N into the sequence restoration sub-model, and the initial sample motion image N can be used as a label image of the mask sample motion image P for model training of the restoration sub-model. The restored sample motion sequence refers to the sequence output by the initial sample motion sequence S after passing through the first stage sequence restoration sub-model.

[0123] It should be noted that, since the input and output resolutions of the sequence repair sub-model can be set to 512×512×3 pixels, the dimensions of the initial sample motion sequence input in actual operation may be 25×75×3 pixels, which is different from the input sequence size set by the sequence repair sub-model. Therefore, before the initial sample motion sequence is input into the sequence repair sub-model for sequence repair, the initial sample motion sequence can be scaled according to the size scaling function to update the initial sample motion sequence. The size scaling function can use the cv2.resize function in the opencv library, which is not specifically limited here.

[0124] See also Figure 4, Figure 4 is an optional flowchart of step S340 provided in an embodiment of the present application. In some embodiments of the present application, the sequence repair sub-model includes a repair encoder and a repair decoder, and step S340 may specifically include but is not limited to steps S410 to S430. Figure 4 These three steps are introduced in detail.

[0125] Step S410, encoding the masked sample motion image according to the restoration encoder to obtain sample implicitly encoded data;

[0126] Step S420, decoding the sample implicitly encoded data according to the repair decoder to obtain a repaired sample motion image;

[0127] Step S430 , inverse normalize the restored sample motion image to obtain a restored sample motion sequence.

[0128] In steps S410 to S430 of some embodiments, the sequence repair submodel provided by the embodiments of the present application is a network structure composed of a repair encoder and a repair decoder. Specifically, the embodiment of the present application first encodes the masked sample motion image P after the image in the initial domain is masked through the repair encoder to obtain sample implicit coding data in the latent space. Then, the sample implicit coding data is decoded according to the repair decoder to decode the sample implicit coding data back to the initial domain. Afterwards, in order to be able to perform the second stage of motion sequence completion, the embodiment of the present application performs inverse normalization on the repaired sample motion image to convert the repaired sample motion image into the original skeleton data domain to obtain a repaired sample motion sequence, denoted as X.

[0129] It should be noted that the embodiments of the present application can use the Transformer-based Mask-Aware Transformer (MAT) network as the baseline network to construct a repair encoder and a repair decoder respectively. Since the MAT network is the best performing repair network at present, the MAT network used in the embodiments of the present application can use styleGAN as the basic structure, and use the improved Transformer structure to perform mask-aware repair processing, that is, the first stage of motion sequence completion. Among them, the MAT network used in the embodiments of the present application can flexibly design mask update strategies for large-area damage and missing parts, divide the mask area into multiple grids, and gradually update the mask according to the grid structure. In addition, due to the global modeling capability of the Transformer, the sequence repair sub-model can repair large-area missing parts.

[0130] In step S350 of some embodiments, in the model training process of the motion sequence completion model, the first stage sequence repair sub-model is first trained. Specifically, the repair loss data is determined according to the repaired sample motion sequence and the initial sample motion image, and the parameters of the sequence repair sub-model are adjusted according to the repair loss data. When the sequence repair sub-model after parameter adjustment meets the first preset end condition, the model training phase of the context optimization sub-model is entered.

[0131] It should be noted that the first preset end condition may be that the sequence repair accuracy of the currently adjusted sequence repair sub-model reaches the first repair accuracy threshold, such as 0.80, 0.85, etc.; or, the first preset end condition may be that the number of iterations of the currently adjusted sequence repair sub-model reaches the first stage iteration number threshold, etc., which is not specifically limited here.

[0132] The embodiment of the present application models a segment of object motion sequence into an image format, and uses a Transformer-based repair network to perform preliminary repair on the "image" with arbitrarily large-area masks applied in both time and space dimensions, which can better handle arbitrary deletions that occur simultaneously in the time dimension and the space dimension.

[0133] It should be noted that after obtaining the restored sample motion image, step S350 of determining the restoration loss data according to the restored sample motion sequence and the initial sample motion image includes: determining the restoration loss data according to the restored sample motion image and the initial sample motion image.

[0134] See also Figure 5 , Figure 5 is an optional flowchart of step S350 provided in an embodiment of the present application. In some embodiments of the present application, step S350 may specifically include but is not limited to steps S510 to S560. Figure 5 These six steps are introduced in detail.

[0135] Step S510, performing calculations based on the restored sample motion image and the initial sample motion image to obtain reconstruction loss data;

[0136] Step S520, inputting the restored sample motion image and the initial sample motion image into the target activation layer of the pre-trained perceptual neural network respectively, to obtain first output data corresponding to the restored sample motion image and second output data corresponding to the initial sample motion image;

[0137] Step S530, calculating according to the loss weight parameter preset in the target activation layer of the sample object, the first output data and the second output data to obtain the perception loss data;

[0138] Step S540, performing calculations according to a preset generation loss function and the restored sample motion image to obtain generation loss data;

[0139] Step S550, performing calculations according to a preset discrimination loss function, the restored sample motion image, and the initial sample motion image to obtain discrimination loss data;

[0140] Step S560, performing weighted calculation based on the reconstruction loss data, the perception loss data, the generation loss data and the judgment loss data to obtain the repair loss data.

[0141] In step S510 of some embodiments, in order to improve the sequence completion capability of the first stage, the embodiments of the present application adopt a combination of three loss functions in the model training stage of the sequence repair sub-model, namely, reconstruction loss, perception loss and adversarial loss. First, as shown in the following formula (2), calculation is performed based on the repaired sample motion image and the initial sample motion image to obtain reconstruction loss data.

[0142]

[0143] Among them, L rec Represents the reconstruction loss data, represents the restored sample motion image; x represents the initial sample motion image, ‖·‖ represents the L1 norm function, which means The process of subtracting the element data at all corresponding positions in x and then solving the sum of the absolute values.

[0144] In steps S520 to S530 of some embodiments, perceptual loss is a loss function used in computer vision tasks, which uses a pre-trained neural network model to measure the perceptual differences between images. The embodiment of the present application can improve the results of tasks such as image generation, super-resolution reconstruction, and image denoising by adjusting the parameters of the model in combination with perceptual loss, so that the generated image is more realistic, clear, and natural. The perceptual neural network used in the embodiment of the present application can be a visual geometry group (Visual Geometry Group, VGG) network, a convolutional neural network (Convolutional Neural Network, CNN), etc., which are not specifically limited here. The embodiment of the present application inputs the restored sample motion image and the initial sample motion image into the target activation layer of the pre-trained perceptual neural network, which can better extract the high-level features of the image and better capture the structure, texture and semantic information of the image, so as to use these features to calculate the difference between the restored sample motion image and the initial sample motion image. Among them, the first output data and the second output data are feature data. As shown in the following formula (3), the perceptual loss data is obtained by calculating according to the loss parameter corresponding to the target activation layer, the first output data, and the second output data.

[0145]

[0146] Among them, L p represents the perceptual loss data, i represents the Cartesian coordinate value of the object joint point corresponding to the frame in the initial sample motion sequence; represents the first output data, φ i (x) represents the second output data; η i Represents the loss weight parameter preset in the target activation layer for the motion sequence of the sample object. In the embodiment of the present application, corresponding weight parameters can be set for the loss of the motion sequence of the object at different target activation layers, thereby improving the sequence completion accuracy of the key area.

[0147] In steps S540 to S550 of some embodiments, when calculating the adversarial loss, the adversarial loss includes generation loss data based on the generator and discrimination loss data based on the discriminator. As shown in the following formula (4), the generation loss data is calculated based on the preset generation loss function and the restored sample motion image.

[0148]

[0149] Among them, L G Indicates the generation of loss data, represents the expected calculation of the restored sample motion image. Represents the output data of the restored sample motion image after passing through the generator.

[0150] As shown in the following formula (5), the discriminant loss data is obtained by calculation based on the preset discriminant loss function, the restored sample motion image and the initial sample motion image.

[0151]

[0152] Among them, L D represents the discriminant loss data, It represents the expected calculation of the initial sample motion image, and D(x) represents the output data of the initial sample motion image after passing through the generator.

[0153] In step S560 of some embodiments, in order to better train the sequence repair sub-model, the embodiments of the present application may perform weighted calculations based on the reconstruction loss data, the perception loss data, the generation loss data, and the discrimination loss data. Specifically, corresponding weight parameters are configured for the reconstruction loss data, the perception loss data, the generation loss data, and the discrimination loss data, and the loss data is multiplied by the corresponding weight parameters and then added to obtain the final repair loss data. Since the embodiments of the present application are to generate better motion completion results, the present application may set the weight parameters corresponding to the reconstruction loss data higher to improve the model's accuracy in repairing the reconstruction. For example, the weight parameters corresponding to the reconstruction loss data, the perception loss data, the generation loss data, and the discrimination loss data may be set to 0.8, 0.1, 0.05, and 0.05, respectively, without specific limitation herein.

[0154] In step S360 of some embodiments, in order to simultaneously consider the non-Euclidean structure of the human skeleton data, that is, to consider the intrinsic correlation between different skeleton points in modeling human body movements. In the embodiment of the present application, a context optimization sub-model is connected in series after the sequence repair sub-model of the motion sequence completion model to serve as an optimizer. Specifically, the preset mask image M and the repaired sample motion sequence X are input into the context optimization sub-model for sequence optimization to obtain an optimized sample sequence, denoted as Y. The context optimization sub-model can use the mask-aware residual connection structure to update the value of the repair result of the masked area, establish an implicit mapping and association between complete and damaged joints, and thus effectively optimize the results generated by the first-stage repair network.

[0155] See also Figure 6 , Figure 6 is an optional flow chart of step S360 provided in an embodiment of the present application. In some embodiments of the present application, step S360 may specifically include but is not limited to steps S610 to S660. Figure 6 These six steps are introduced in detail.

[0156] Step S610, performing convolution processing on the restored sample motion sequence according to the first graph convolution block to obtain a first optimized sequence;

[0157] Step S620, performing a preset number of convolution processes on the first optimization sequence in sequence according to the second graph convolution block, and taking the output sequence of the last second graph convolution block as the second optimization sequence;

[0158] Step S630, performing convolution processing on the second optimized sequence according to the first graph convolution layer to obtain a third optimized sequence;

[0159] Step S640, performing image decomposition on the preset mask image to obtain a mask sequence;

[0160] Step S650, updating the third optimized sequence according to the mask sequence;

[0161] Step S660: Add the restored sample motion sequence and the updated third optimized sequence to obtain an optimized sample sequence.

[0162] For example, Figure 7 A schematic diagram of the model structure of the context optimization sub-model is shown. The context optimization sub-model includes a first graph convolution block connected in series, a second graph convolution block with a preset number of residual connections, and a first graph convolution layer.

[0163] In steps S610 to S620 of some embodiments, when the repaired sample motion sequence X is input into the context optimization sub-model for optimization, the repaired sample motion sequence X is first transposed, that is, the dimensional order of the input data and the output data is transformed by transposition so as to match the input data dimension of the graph convolution block in the context optimization sub-model. Then, the repaired sample motion sequence is convolved according to the first graph convolution block to obtain a first optimized sequence. The first optimized sequence is input into the first second graph convolution block for convolution, and the first optimized sequence and the sequence output by the first second graph convolution block are added and calculated to obtain a new motion sequence. After that, the obtained new running sequence is input into the next second graph convolution block connected in series with the first second graph convolution block. After convolution processing of a preset number of second graph convolution blocks, the output sequence of the last second graph convolution block is used as the second optimized sequence.

[0164] It should be noted that the increase in the number of second graph convolution blocks will reduce the training speed of the model, but at the same time it can improve the completion accuracy of the model. Therefore, when a more efficient motion sequence completion model is needed, a larger preset number can be set; and when a motion sequence completion model with higher completion accuracy is needed, a smaller preset number can be set.

[0165] In steps S630 to S660 of some embodiments, after obtaining the second optimized sequence, in order to extract deeper feature information, the second optimized sequence is convolved according to the first graph convolution layer to obtain a third optimized sequence. When the context optimization sub-model is used for sequence optimization in the embodiment of the present application, a mask-aware residual connection module is added, that is, only the value of the area that has initially undergone mask processing is updated to further optimize the restoration result of the first stage. Specifically, the preset mask image is decomposed to obtain a mask sequence. Then, the values ​​of the corresponding joint points of the mask sequence and the third optimized sequence are numerically multiplied to update the third optimized sequence. Thereafter, the restored sample motion sequence and the updated third optimized sequence are added to obtain an optimized sample sequence Y.

[0166] See also Figure 8 , Figure 8 is an optional flowchart of step S610 provided in an embodiment of the present application. In some embodiments of the present application, step S610 may specifically include but is not limited to steps S810 to S830. Figure 8 These three steps are introduced in detail.

[0167] Step S810, performing convolution processing on the restored sample motion sequence according to the second graph convolution layer to obtain a convolution sequence;

[0168] Step S820, performing activation processing on the convolution sequence according to the activation layer to obtain an activation sequence;

[0169] Step S830, regularizing the activation sequence according to the random dropout layer to obtain a first optimized sequence.

[0170] For example, Fig. 9 A structural schematic diagram of a graph convolution block is shown. The first graph convolution block and the second graph convolution block have the same structure, and the first graph convolution block includes a second graph convolution layer, an activation layer, and a random inactivation layer.

[0171] In steps S810 to S830 of some embodiments, when the first graph convolution block and the second graph convolution block are specifically processed, the first graph convolution block is used as an example. Specifically, the restored sample motion sequence is first convolved according to the second graph convolution layer to obtain a convolution sequence. Then, the convolution sequence is activated according to the activation layer to obtain an activation sequence. The activation layer of the embodiment of the present application can be a nonlinear factor added to the convolution sequence to map the features of the convolution sequence to a high-dimensional nonlinear interval for interpretation, so as to solve problems that cannot be solved by linear models. Afterwards, the activation sequence is regularized according to the random deactivation layer to obtain a first optimized sequence. The random deactivation layer of the embodiment of the present application can deactivate some nodes, which can simplify the neural network structure and thus play a regularization role.

[0172] In step S370 of some embodiments, in the second stage of model training, the weight of each loss function of the sequence restoration submodel in the first stage is fixed, and then the three-dimensional optimization loss data is determined according to the target sample motion sequence and the optimized sample sequence, and the parameters of the context optimization submodel are adjusted according to the three-dimensional optimization loss data. In the second stage of model training of the embodiment of the present application, mask information can be injected into the optimization stage so that the graph convolution block can perceive the context information, thereby effectively modeling the relationship between different frames.

[0173] See also Fig.10 , Fig.10 is an optional flowchart of step S370 provided in an embodiment of the present application. In some embodiments of the present application, step S370 may specifically include but is not limited to steps S1010 to S1060. Fig.10 These six steps are introduced in detail.

[0174] Step S1010, obtaining target three-dimensional coordinate data of each sample joint point in the target sample motion sequence, wherein the target three-dimensional coordinate data includes a target first-dimensional coordinate value, a target second-dimensional coordinate value, and a target third-dimensional coordinate value;

[0175] Step S1020, obtaining optimized three-dimensional coordinate data of each sample joint point in the optimized sample sequence, wherein the optimized three-dimensional coordinate data includes optimizing the first-dimensional coordinate value, optimizing the second-dimensional coordinate value, and optimizing the third-dimensional coordinate value;

[0176] Step S1030, performing coordinate calculation on the target first-dimensional coordinate value and the optimized first-dimensional coordinate value to obtain a first-dimensional loss value;

[0177] Step S1040, performing coordinate calculation on the target second-dimensional coordinate value and the optimized second-dimensional coordinate value to obtain a second-dimensional loss value;

[0178] Step S1050, performing coordinate calculation on the target third-dimensional coordinate value and the optimized third-dimensional coordinate value to obtain a third-dimensional loss value;

[0179] Step S1060, calculating the mean of the first dimension loss value, the second dimension loss value and the third dimension loss value to obtain three-dimensional optimization loss data.

[0180] In steps S1010 to S1060 of some embodiments, in the optimization training stage of the motion sequence completion model, the embodiment of the present application flattens the three-dimensional data of the human body joints of the repaired sample motion sequence X and regards it as a graph node, thereby treating each frame along the time dimension as a feature of the graph node. Specifically, since each object joint point in the motion sequence of the embodiment of the present application includes the value of each coordinate dimension in the Cartesian coordinate system, the target three-dimensional coordinate data of each sample joint point in the target sample motion sequence is obtained, and the target three-dimensional coordinate data includes the target first-dimensional coordinate value, the target second-dimensional coordinate value, and the target third-dimensional coordinate value. The optimized three-dimensional coordinate data of each sample joint point in the optimized sample sequence is obtained, and the optimized three-dimensional coordinate data includes the optimized first-dimensional coordinate value, the optimized second-dimensional coordinate value, and the optimized third-dimensional coordinate value. In the training stage of the context optimization sub-model, the L2 distance loss function can be used when calculating the loss according to the three-dimensional joint point coordinates. Afterwards, the L2 distance loss function is used to calculate the coordinates of the target first-dimensional coordinate value and the optimized first-dimensional coordinate value to obtain the first-dimensional loss value. The L2 distance loss function is used to calculate the coordinates of the target second-dimensional coordinate value and the optimized second-dimensional coordinate value to obtain the second-dimensional loss value. The L2 distance loss function is used to calculate the coordinates of the target third-dimensional coordinate value and the optimized third-dimensional coordinate value to obtain the third-dimensional loss value. Finally, the first-dimensional loss value, the second-dimensional loss value, and the third-dimensional loss value are averaged to obtain the three-dimensional optimization loss data.

[0181] It should be noted that in the optimization training phase using the context optimization sub-model, the Adam optimizer can be used to perform model training based on the parameter settings of a learning rate of 2e-4, a batch size of 16 times, and a total number of iterations of 300,000 times.

[0182] In step S230 of some embodiments, based on the trained sequence repair sub-model, the initial motion sequence is first input into the sequence repair sub-model, so as to perform sequence repair on the initial motion sequence according to the sequence repair sub-model to obtain a repaired motion sequence.

[0183] In step S240 of some embodiments, based on the trained context optimization sub-model, the repaired motion sequence is input into the sequence repair sub-model to perform sequence optimization on the repaired motion sequence according to the context optimization sub-model to obtain a target motion sequence with complete completion of the target object.

[0184] For example, Fig.11 This is a schematic diagram of the embodiment of the present application for performing motion sequence completion in the time-space dimension simultaneously. Fig.11 The thick lines in the figure represent the connection lines of the object joint points in the completed motion sequence of the target object, and the thin lines represent the connection lines of the object joint points in the incomplete initial motion sequence of the target object initially obtained.

[0185] For example, Fig.12 The overall model training schematic diagram of the motion sequence completion model of the embodiment of the present application is shown. Specifically, first, an initial sample motion sequence with incomplete sample objects and a target sample motion sequence with complete sample objects are obtained. The initial sample motion sequence S is sequence normalized to obtain an initial sample motion image N. Then, the initial sample motion image N is image masked according to a preset mask image M to obtain a masked sample motion image P. In the first stage of the sequence repair sub-model, the masked sample motion image P is input into the repair encoder of the sequence repair sub-model for encoding processing to obtain sample implicit coding data. The sample implicit coding data is decoded according to the repair decoder to obtain a repaired sample motion image. The repaired sample motion image is inversely normalized to obtain a repaired sample motion sequence, and a repaired sample motion sequence X is obtained. The sequence repair sub-model can adjust the model parameters according to the repaired sample motion sequence X and the initial sample motion image N. Afterwards, in the second stage of the context optimization sub-model training, the repaired sample motion sequence X is sequence transposed, and the repaired sample motion sequence X is convolved according to the first graph convolution block to obtain a first optimized sequence. The first optimized sequence is input into the first second graph convolution block for convolution processing, and the first optimized sequence and the sequence output by the first second graph convolution block are added and calculated to obtain a new motion sequence. The obtained new running sequence is input into the next second graph convolution block connected in series with the first second graph convolution block. After convolution processing of a preset number of second graph convolution blocks, the output sequence of the last second graph convolution block is used as the second optimized sequence. In order to extract deeper feature information, the second optimized sequence is convolved according to the first graph convolution layer to obtain a third optimized sequence. The preset mask image is image decomposed to obtain a mask sequence. Then, the values ​​of the corresponding joint points of the mask sequence and the third optimized sequence are numerically multiplied to update the third optimized sequence. The repaired sample motion sequence and the updated third optimized sequence are added, and the added sequence is transposed again to obtain the optimized sample sequence Y. Afterwards, the parameters of the context optimization sub-model can be adjusted according to the optimized sample sequence Y initial sample motion sequence S.

[0186] A human motion sequence completion method provided in an embodiment of the present application proposes a two-stage "repair-optimization" object motion completion scheme. First, the object motion sequence in the form of a skeleton is modeled in the form of an image, and the missing part of the object motion sequence is preliminarily repaired using a sequence repair submodel. Then, the repair result is further optimized using a context optimization submodel composed of a graph convolutional network. Among them, the context optimization submodel proposed in the embodiment of the present application, by adding a mask-aware residual connection structure, inputs the mask to the position where the last parameter of the graph convolution block is updated, and only updates the value of the mask part, so that the intrinsic relationship between the object joints can be modeled, and the joints of the complete area and the occluded defect area are implicitly associated. Therefore, the embodiment of the present application normalizes the initial sample motion sequence S to the format of a single initial sample motion image N, obtains the mask sample motion image P by element-by-element multiplication with the preset mask image M, and inputs both the preset mask image M and the mask sample motion image P into the sequence repair submodel. Then, the mask sample motion image P is repaired using a sequence repair submodel composed of a repair encoder and a repair decoder. The embodiment of the present application uses an inverse normalization operation to reversely convert the repaired sequence into a repaired sample motion sequence X in the skeleton domain, and performs sequence optimization through the context optimization submodel designed in this application. Afterwards, the final optimized sample sequence Y and the initial sample motion sequence S are used together to calculate the three-dimensional optimization loss data. It can be seen that the network proposed in the embodiment of the present application can not only benefit from the advantage of the image restoration framework to generate arbitrary shapes, but also benefit from the graph convolutional network's modeling ability for graph nodes, thereby being able to generate better human motion sequence completion results.

[0187] See also Fig.13 , Fig.13 13 is a schematic diagram of the structure of a human motion sequence completion device provided in an embodiment of the present application, wherein the device comprises a sequence acquisition module 1310 , an input module 1320 , a sequence repair module 1330 and a sequence optimization module 1340 .

[0188] A sequence acquisition module 1310 is used to acquire an incomplete initial motion sequence of a target object;

[0189] An input module 1320 is used to input the initial motion sequence into a pre-built motion sequence completion model, where the motion sequence completion model includes a sequence repair sub-model and a context optimization sub-model;

[0190] A sequence repair module 1330 is used to perform sequence repair on the initial motion sequence according to the sequence repair sub-model to obtain a repaired motion sequence;

[0191] The sequence optimization module 1340 is used to perform sequence optimization on the restored motion sequence according to the context optimization sub-model to obtain a target motion sequence with a complete target object.

[0192] It should be noted that the human motion sequence completion device of the embodiment of the present application is used to implement the human motion sequence completion method of the above-mentioned embodiment. The human motion sequence completion device of the embodiment of the present application corresponds to the aforementioned human motion sequence completion method. Please refer to the aforementioned human motion sequence completion method for the specific processing process, which will not be repeated here.

[0193] The embodiment of the present application also provides a computer device, which includes: at least one memory, at least one processor, and at least one computer program, wherein the at least one computer program is stored in the at least one memory, and the at least one processor executes the at least one computer program to implement any one of the human motion sequence completion methods in the above embodiments. The computer device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0194] See also Fig.14 , Fig.14 The hardware structure of a computer device according to another embodiment is illustrated. The computer device includes:

[0195] The processor 1410 may be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0196] The memory 1420 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1420 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1420, and the processor 1410 calls and executes the human body motion sequence completion method of the embodiment of this application;

[0197] Input / output interface 1430, used to implement information input and output;

[0198] Communication interface 1440, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0199] Bus 1450 , which transmits information between various components of the device (e.g., processor 1410 , memory 1420 , input / output interface 1430 , and communication interface 1440 );

[0200] The processor 1410 , the memory 1420 , the input / output interface 1430 , and the communication interface 1440 are connected to each other in communication within the device via a bus 1450 .

[0201] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program, and the computer program is used to enable a computer to execute the human body motion sequence completion method in the above embodiment.

[0202] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0203] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0204] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0205] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0206] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0207] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0208] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0209] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0210] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0211] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0212] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0213] The preferred embodiments of the present invention are described with reference to the accompanying drawings, but the scope of the present invention is not limited by the above. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A method for completing a human motion sequence, characterized in that: The method comprises: Obtaining an incomplete initial motion sequence of the target object; A motion sequence completion model is constructed, wherein the motion sequence completion model includes a sequence repair submodel and a context optimization submodel; the construction of the motion sequence completion model specifically includes: obtaining an incomplete initial sample motion sequence of a sample object and a target sample motion sequence of the sample object; performing sequence normalization on the initial sample motion sequence to obtain an initial sample motion image; performing image masking on the initial sample motion image according to a preset mask image to obtain a masked sample motion image; inputting the masked sample motion image into the sequence repair submodel to perform sequence repair to obtain a repaired sample motion sequence; determining repair loss data according to the repaired sample motion sequence and the initial sample motion image, and adjusting parameters of the sequence repair submodel according to the repair loss data; inputting the preset mask image and the repaired sample motion sequence into the context optimization submodel to perform sequence optimization to obtain an optimized sample sequence; determining three-dimensional optimization loss data according to the target sample motion sequence and the optimized sample sequence, and adjusting parameters of the context optimization submodel according to the three-dimensional optimization loss data number adjustment; obtaining the motion sequence completion model according to the sequence repair submodel and the context optimization submodel after parameter adjustment; wherein the context optimization submodel includes a first graph convolution block connected in series, a second graph convolution block with a preset number of residual connections, and a first graph convolution layer, and the preset mask image and the repaired sample motion sequence are input into the context optimization submodel for sequence optimization to obtain an optimized sample sequence, including: performing convolution processing on the repaired sample motion sequence according to the first graph convolution block to obtain a first optimized sequence; performing convolution processing on the first optimized sequence in sequence according to the second graph convolution block for the preset number of times, and taking the output sequence of the last second graph convolution block as the second optimized sequence; performing convolution processing on the second optimized sequence according to the first graph convolution layer to obtain a third optimized sequence; performing image decomposition on the preset mask image to obtain a mask sequence; performing sequence update on the third optimized sequence according to the mask sequence; adding the repaired sample motion sequence and the updated third optimized sequence to obtain the optimized sample sequence; Inputting the initial motion sequence into a pre-built motion sequence completion model; Performing sequence repair on the initial motion sequence according to the sequence repair sub-model to obtain a repaired motion sequence; The repaired motion sequence is sequence optimized according to the context optimization sub-model to obtain a target motion sequence with the target object completed.

2. The method according to claim 1, characterized in that The sequence repair sub-model includes a repair encoder and a repair decoder. Inputting the mask sample motion image into the sequence repair sub-model for sequence repair to obtain a repaired sample motion sequence includes: encoding the masked sample motion image according to the restoration encoder to obtain sample implicitly encoded data; Decoding the sample implicitly encoded data according to the repair decoder to obtain a repaired sample motion image; Performing inverse normalization on the restored sample motion image to obtain the restored sample motion sequence; The determining of restoration loss data according to the restored sample motion sequence and the initial sample motion image comprises: Restoration loss data is determined according to the restored sample motion image and the initial sample motion image.

3. The method according to claim 2, characterized in that The determining of restoration loss data according to the restored sample motion image and the initial sample motion image comprises: Calculating according to the restored sample motion image and the initial sample motion image to obtain reconstruction loss data; Inputting the restored sample motion image and the initial sample motion image into a target activation layer of a pre-trained perceptual neural network respectively, to obtain first output data corresponding to the restored sample motion image and second output data corresponding to the initial sample motion image; Calculate according to the loss weight parameter preset by the sample object in the target activation layer, the first output data and the second output data to obtain the perception loss data; Calculate according to a preset generation loss function and the restored sample motion image to obtain generation loss data; Calculate according to a preset discrimination loss function, the restored sample motion image and the initial sample motion image to obtain discrimination loss data; The restoration loss data is obtained by performing weighted calculation according to the reconstruction loss data, the perception loss data, the generation loss data and the discrimination loss data.

4. The method according to claim 1, characterized in that: The determining of three-dimensional optimization loss data according to the target sample motion sequence and the optimized sample sequence includes: Acquire target three-dimensional coordinate data of each sample joint point in the target sample motion sequence, wherein the target three-dimensional coordinate data includes a target first-dimensional coordinate value, a target second-dimensional coordinate value, and a target third-dimensional coordinate value; Obtaining optimized three-dimensional coordinate data of each sample joint point in the optimized sample sequence, wherein the optimized three-dimensional coordinate data includes an optimized first-dimensional coordinate value, an optimized second-dimensional coordinate value, and an optimized third-dimensional coordinate value; Performing coordinate calculation on the target first-dimensional coordinate value and the optimized first-dimensional coordinate value to obtain a first-dimensional loss value; Performing coordinate calculation on the target second-dimensional coordinate value and the optimized second-dimensional coordinate value to obtain a second-dimensional loss value; Performing coordinate calculation on the target third-dimensional coordinate value and the optimized third-dimensional coordinate value to obtain a third-dimensional loss value; The first-dimensional loss value, the second-dimensional loss value, and the third-dimensional loss value are averaged to obtain the three-dimensional optimization loss data.

5. The method according to claim 1, characterized in that The first graph convolution block and the second graph convolution block have the same structure, and the first graph convolution block includes a second graph convolution layer, an activation layer, and a random inactivation layer, and the convolution processing of the restored sample motion sequence according to the first graph convolution block to obtain a first optimized sequence includes: performing convolution processing on the restored sample motion sequence according to the second graph convolution layer to obtain a convolution sequence; Performing activation processing on the convolution sequence according to the activation layer to obtain an activation sequence; Regularization is performed on the activation sequence according to the random dropout layer to obtain the first optimized sequence.

6. A human motion sequence completion device, characterized in that: The device comprises: A sequence acquisition module, used to acquire an incomplete initial motion sequence of a target object; An input module is used to input the initial motion sequence into a pre-constructed motion sequence completion model, wherein the motion sequence completion model includes a sequence repair sub-model and a context optimization sub-model; the input module is also used to construct a motion sequence completion model, which specifically includes: obtaining an incomplete initial sample motion sequence of a sample object and a target sample motion sequence with the sample object completed; performing sequence normalization on the initial sample motion sequence to obtain an initial sample motion image; performing image masking on the initial sample motion image according to a preset mask image to obtain a masked sample motion image; inputting the masked sample motion image into the sequence repair sub-model to perform sequence repair to obtain a repaired sample motion sequence; determining repair loss data according to the repaired sample motion sequence and the initial sample motion image, and adjusting parameters of the sequence repair sub-model according to the repair loss data; inputting the preset mask image and the repaired sample motion sequence into the context optimization sub-model to perform sequence optimization to obtain an optimized sample sequence; determining three-dimensional optimization loss data according to the target sample motion sequence and the optimized sample sequence, and adjusting parameters of the sequence repair sub-model according to the three-dimensional optimization loss data The parameters of the context optimization submodel are adjusted; the motion sequence completion model is obtained according to the sequence repair submodel and the context optimization submodel after the parameter adjustment; wherein the context optimization submodel includes a first graph convolution block connected in series, a second graph convolution block with a preset number of residual connections, and a first graph convolution layer; the preset mask image and the repaired sample motion sequence are input into the context optimization submodel for sequence optimization to obtain an optimized sample sequence, including: performing convolution processing on the repaired sample motion sequence according to the first graph convolution block to obtain a first optimized sequence; performing convolution processing on the first optimized sequence according to the second graph convolution block for the preset number of times in sequence, and taking the output sequence of the last second graph convolution block as the second optimized sequence; performing convolution processing on the second optimized sequence according to the first graph convolution layer to obtain a third optimized sequence; performing image decomposition on the preset mask image to obtain a mask sequence; performing sequence update on the third optimized sequence according to the mask sequence; adding the repaired sample motion sequence and the updated third optimized sequence to obtain the optimized sample sequence; A sequence repair module, used for performing sequence repair on the initial motion sequence according to the sequence repair sub-model to obtain a repaired motion sequence; A sequence optimization module is used to perform sequence optimization on the repaired motion sequence according to the context optimization sub-model to obtain a target motion sequence that completes the target object.

7. A computer device, characterized in that: include: at least one memory; at least one processor; at least one computer program; The at least one computer program is stored in the at least one memory, and the at least one processor executes the at least one computer program to implement: The method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer readable storage medium stores a computer program, and the computer program is used to cause a computer to execute: The method according to any one of claims 1 to 5.

Images