Multi-person interactive action generation method, device and electronic device

Abstract

The present invention relates to the field of computer vision technology, and in particular to a method, device and electronic device for generating multi-person interactive actions. The method comprises: obtaining an initial motion sequence and a motion text description of a target instance, wherein the target instance is a multi-person interactive instance; adding Gaussian noise to the initial motion sequence of the target instance to obtain a noisy initial motion sequence; training a diffusion model based on the motion sequence, spatial control conditions and motion text description of a non-target instance to obtain a trained diffusion model; using the noisy initial motion sequence and the motion text description of the target instance as inputs of the denoising process of the trained diffusion model to obtain the motion action and motion trajectory of the target instance. By adopting the above method, the present invention solves the problem that the multi-person interactive action generation method in the related art cannot ensure the accuracy of the interactive behavior while ensuring the rationality of the individual instance actions.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method, device and electronic equipment for generating multi-person interactive actions. Background Art

[0002] Multi-person interactive action generation is one of the hot research topics in computer vision and an important research direction for vision-based pattern recognition applications. Based on given conditions, multi-person interactive action generation will generate multi-person interactive action sequences, which not only explicitly encode the actions of each human instance, but also show the interactive behaviors of multiple human instances, accompanied by the motion trajectory of each instance. Multi-person interactive generation has a wide range of applications in the fields of robot control, filmmaking, virtual reality, and animation production, and has extremely high economic and application value.

[0003] At present, the mainstream methods of multi-person interactive action generation are divided into single-person adaptive methods and multi-person interactive methods. The former only trains a single-person generative model and adds rule strategies during multi-person interaction to enable the diffusion reverse process to generate multi-person actions; the latter adds multi-person constraints during training to ensure the rationality of the generated multi-person trajectories and actions.

[0004] However, the multi-person interactive action generation method in the above-mentioned related technologies cannot ensure the accuracy of the interactive behavior while ensuring the rationality of each human instance action. Summary of the invention

[0005] The present invention provides a multi-person interactive action generation method, device and electronic device, which are used to solve the problem that the multi-person interactive action generation method in the prior art cannot ensure the accuracy of the interactive behavior while ensuring the rationality of individual instance actions.

[0006] The present invention provides a method for generating a multi-person interactive action, comprising:

[0007] Acquire an initial motion sequence and a motion text description of a target instance, wherein the target instance is a multi-person interaction instance;

[0008] Adding Gaussian noise to the initial motion sequence of the target instance to obtain a noisy initial motion sequence;

[0009] Based on the motion sequence, spatial control conditions and motion text description of the non-target instance, a diffusion model is trained to obtain a trained diffusion model;

[0010] The noisy initial motion sequence and the motion text description of the target instance are used as inputs of the denoising process of the trained diffusion model to obtain the motion action and motion trajectory of the target instance.

[0011] According to the present invention, a method for generating a multi-person interactive action is provided, which obtains a motion text description of the target instance, specifically comprising:

[0012] Obtaining an initial motion text description of the target instance;

[0013] Splitting the initial movement text description to obtain multiple single-person movement text descriptions;

[0014] A hierarchical motion text description is constructed based on the multiple single-person motion text descriptions, and the hierarchical motion text description is used as the motion text description of the target instance.

[0015] According to the present invention, a method for generating multi-person interactive actions is provided, wherein the method trains a diffusion model based on a motion sequence, a spatial control condition and a motion text description of a non-target instance to obtain a trained diffusion model, specifically comprising:

[0016] Taking the noisy initial motion sequence and the motion text description of the target instance as inputs of a diffusion model;

[0017] Taking the motion sequence, spatial control conditions and motion text description of the non-target instance as input to the interaction network;

[0018] The input of the diffusion model and the input of the interactive network are subjected to feature interaction, and the input of the interactive network is embedded in the diffusion model to obtain a trained diffusion model.

[0019] According to the present invention, a method for generating multi-person interactive actions is provided, wherein the diffusion model and the interactive network are constructed through Mamda modules and have the same structure.

[0020] According to the present invention, a method for generating multi-person interactive actions is provided, wherein the method trains a diffusion model based on a motion sequence, a spatial control condition and a motion text description of a non-target instance to obtain a trained diffusion model, and further comprises:

[0021] In the process of feature interaction between the input of the diffusion model and the input of the interaction network, a hierarchical loss constraint is constructed to constrain the spatial position and motion action between each instance.

[0022] According to the present invention, a method for generating multi-person interactive actions is provided, wherein the noisy initial motion sequence and the motion text description of the target instance are used as inputs of the denoising process of the trained diffusion model to obtain the motion action and motion trajectory of the target instance, specifically comprising:

[0023] Using the noisy initial motion sequence as an input of a denoising process of the trained diffusion model to obtain a motion sequence of a first human instance in the target instance;

[0024] Using the noisy initial motion sequence of the second human body instance in the target instance and the motion sequence of the historical human body instance as inputs of the denoising process of the trained diffusion model to obtain the motion sequence of the second human body instance, and the motion sequence of the historical human body instance is the motion sequence of the first human body instance;

[0025] Using the motion sequence of the first human body instance and the motion sequence of the second human body instance as a new motion sequence of a historical human body instance to obtain a motion sequence of a next human body instance; until the motion sequence of each human body instance in the target instance is obtained;

[0026] According to the motion sequence of each human body instance in the target instance, the motion action and motion trajectory of the target instance are obtained.

[0027] The present invention also provides a multi-person interactive action generating device, comprising:

[0028] An instance acquisition unit, used to acquire an initial motion sequence and a motion text description of a target instance, wherein the target instance is a multi-person interaction instance;

[0029] A forward noise adding unit, used for adding Gaussian noise to the initial motion sequence of the target instance to obtain a noisy initial motion sequence;

[0030] A model training unit, used for training a diffusion model based on a motion sequence of a non-target instance, a spatial control condition and a motion text description, to obtain a trained diffusion model;

[0031] The reverse denoising unit is used to use the noisy initial motion sequence and the motion text description of the target instance as inputs of the denoising process of the trained diffusion model to obtain the motion action and motion trajectory of the target instance.

[0032] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the multi-person interactive action generating method as described above is implemented.

[0033] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the multi-person interactive action generation method as described above is implemented.

[0034] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the multi-person interactive action generating method as described above is implemented.

[0035] The multi-person interactive action generation method, device and electronic device provided by the present invention train a diffusion model through the motion sequence, spatial control conditions and motion text description of non-target instances to obtain a trained diffusion model; the noisy initial motion sequence and the motion text description of the target instance are used as inputs of the denoising process of the trained diffusion model to obtain the motion action and motion trajectory of the target instance, thereby ensuring the accuracy of the interactive behavior while ensuring the rationality of the individual instance actions. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 It is a flow chart of the method for generating multi-person interactive actions provided by the present invention.

[0038] Figure 2 It is a flow chart of the training diffusion model provided by the present invention.

[0039] Figure 3 It is a schematic diagram of the principle of the Transformer cross-attention interaction module provided by the present invention.

[0040] Figure 4 It is a schematic diagram of the principle of using the Mamba module as a feature encoder provided by the present invention.

[0041] Figure 5 It is a flow chart of obtaining the motion action and motion trajectory of a target instance provided by the present invention.

[0042] Figure 6 It is a structural schematic diagram of the multi-person interactive action generating device provided by the present invention.

[0043] Figure 7 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the description used in this way can be interchangeable where appropriate, so that the embodiment can be implemented in a sequence other than that illustrated or described in the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or modules is not necessarily limited to those steps or modules that are clearly listed, but may include other steps or modules that are not clearly listed or inherent to these processes, methods, products or devices. The naming or numbering of the steps that appear in the present application does not mean that the steps in the method flow must be executed in the time / logical sequence indicated by the naming or numbering. The process steps that have been named or numbered can change the execution order according to the technical purpose to be achieved, as long as the same or similar technical effects can be achieved. The division of modules in this application is a logical division. There may be other division methods when it is implemented in actual applications. For example, multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection between units can be electrical or other similar forms, which are not limited in this application. In addition, the modules or submodules described as separate components may or may not be physically separated, may or may not be physical units, or may be distributed in multiple circuit units, and some or all of the units may be selected according to actual needs to achieve the purpose of the present application.

[0046] Combine the following Figure 1-Figure 7 The specific contents of the present invention are described.

[0047] Figure 1 The flowchart of the method for generating multi-person interactive actions provided by the present invention includes steps S101-S104.

[0048] Step S101, obtaining an initial motion sequence and a motion text description of a target instance, where the target instance is a multi-person interaction instance.

[0049] In the embodiments of the present invention, the target instance refers to a specific multi-person interaction scenario that requires motion actions and motion trajectory generation. The initial motion sequence contains the motion data of each individual instance or object in the target instance in the initial state, such as parameters such as position, speed, acceleration, etc. The motion text description is used to express the action or interaction content expected to occur in the target instance.

[0050] Step S102: Add Gaussian noise to the initial motion sequence of the target instance to obtain a noisy initial motion sequence.

[0051] In the embodiment of the present invention, the initial motion sequence of the target instance is used as the input of the forward noise addition process of the diffusion model through a conventional diffusion model. In the forward noise addition process, Gaussian noise is gradually added, and after T time steps, the noisy initial motion sequence is obtained.

[0052] Step S103 , training a diffusion model based on the motion sequence, spatial control conditions and motion text description of the non-target instance to obtain a trained diffusion model.

[0053] In an embodiment of the present invention, during the training process of the diffusion model, the motion sequence of the non-target instance includes motion data of other multi-person interaction scenes that are different from the target instance, which is used to train the model, so that the diffusion model can learn a wider range of motion patterns and interaction rules. The spatial control conditions may include the layout of the scene, the position of the objects, the relative position of the characters, etc., which are used to ensure that the generated motion sequence meets specific spatial constraints. The motion sequence, spatial control conditions and motion text description of the non-target instance are collectively referred to as non-target instance data. The diffusion model learns the distribution of non-target instance data by gradually removing noise from the non-target instance data, so that data that meets a specific distribution can be generated given the noise data and conditions.

[0054] Step S104: The noisy initial motion sequence and the motion text description of the target instance are used as inputs of the denoising process of the trained diffusion model to obtain the motion action and motion trajectory of the target instance.

[0055] In a possible implementation, in step S101, obtaining a motion text description of a target instance specifically includes:

[0056] Step S1011, obtaining an initial motion text description of the target instance.

[0057] Step S1012, splitting the initial motion text description to obtain multiple single-person motion text descriptions.

[0058] Step S1013: construct a hierarchical motion text description based on multiple single-person motion text descriptions, and use the hierarchical motion text description as the motion text description of the target instance.

[0059] Specifically, in order to progressively characterize the motion text description of individual instances, the large language model is used to split the initial motion text description into single-person motion text descriptions of individual instances. A hierarchical motion text description is constructed. For the first individual instance, a single-person motion description is used; for subsequent individual instances, the single-person motion text descriptions of all previous individual instances are gradually merged to form a hierarchical motion text description.

[0060] In one embodiment of the present invention, the hierarchical motion text description is used as the motion text description of the target instance in step S104. The noised initial motion sequence is randomized noise. A specific prompt rule (i.e., single-person motion description) is defined for the order of prediction of each individual instance. First, the text of the single-person motion description is input into the large language model to obtain a text embedding description, and then the text embedding description is spliced ​​with the randomized noise to form the input of the denoising process of the trained diffusion model.

[0061] In a possible implementation, the text encoder in the diffusion model adopts a hybrid encoding method.

[0062] Specifically, in order to effectively process long and short texts, a hybrid encoding method of the T5 large language model and the CLIP text encoder is used. The encoding method is defined by the following formula:

[0063]

[0064] in, is the encoding method of the text encoder in the diffusion model, It is a text encoder selection module, that is, the corresponding text encoder is selected according to the length of the text. The present invention adopts CLIP text encoder for short text and T5 text encoder for long text. Exemplarily, during training, a random encoder is adopted with a probability of 0.1 to increase robustness, that is, a random text encoder is adopted for the input text with a probability of 0.1.

[0065] In a possible implementation, step S103, based on the motion sequence, spatial control conditions and motion text description of the non-target instance, a diffusion model is trained to obtain a trained diffusion model, such as Figure 2 As shown, it specifically includes steps S201-S203.

[0066] In the embodiment of the present invention, the target instance is a multi-person interaction instance, and the number of interaction individuals is P， Represents all human instances. The motion sequence of each human instance is expressed as:

[0067]

[0068] in represents the key point position, Indicates speed, is the local rotation, A sign that indicates whether the foot is touching the ground.

[0069] Step S201: taking the noisy initial motion sequence and the motion text description of the target instance as inputs of the diffusion model.

[0070] Specifically, when the noisy initial motion sequence and the motion text description of the target instance are used as the input of the diffusion model, key point representation or rotation representation can be selected. , is the number of frames, is the number of key points, M represents the length of the vector, is a set of spatial ranges. After feature mapping, the individual key point representation features can be expressed as Similarly, construct the torso representation , is the number of limbs. After feature interactive mapping, the trunk feature can be expressed as . D is the feature dimension, K The number of limbs.

[0071] Step S202: taking the motion sequence, spatial control conditions and motion text description of the non-target instance as input to the interactive network.

[0072] Specifically, while inputting the motion sequence of the non-target instance into the interactive network, the spatial control condition of the non-target instance is added. The spatial control condition of the non-target instance is sparse, which is specifically manifested in the sparsity between frames and the sparsity at the key point level of control. Spatial control condition , after being encoded by the linear layer, it will be added to the mapping of the historical motion sequence to make corresponding constraints on the motion of the current individual instance according to the historical trajectory information and spatial control conditions.

[0073] Step S203 , performing feature interaction between the input of the diffusion model and the input of the interactive network, and embedding the input of the interactive network into the diffusion model to obtain a trained diffusion model.

[0074] To achieve constraint control, the present invention uses the Transformer cross-attention interaction module to build the association between the control conditions of the target instance and the non-target instance. The interaction module takes the sequence input of the target instance and the sequence input of the non-target instance as the basic conditions, completes the interaction modeling through the cross-attention module, and finally bridges to the motion sequence modeling network of the target instance through linear layer mapping. Specifically, the interaction module adopts a bidirectional architecture, in which the forward direction is based on the encoding of the motion sequence of the non-target instance. As a query, the encoding of the target instance As key, value; reverse as target instance As a query, non-target instance As key and value. The interactive features will be split to form limb and key point motion sequences in turn, and added to the original features of the two branches to form the input of the next stage. The specific structure can be found in the attached Figure 3 .

[0075] In one possible implementation, the diffusion model and the interaction network are constructed through Mamda modules and have the same structure.

[0076] Specifically, the denoising network that performs the denoising process in the diffusion model consists of a hierarchical Mamba module architecture. Specifically, the input key point sequence is reconstructed to form a dual-branch basic architecture. Among them, the key point branch takes the key point sequence as input to form a feature representation The high-level trunk branch uses the human trunk vector as input to form a feature representation Both dual-branch networks use the Mamba module as the feature encoder, and finally use the splicing operation at the end of each stage to complete the feature aggregation. This feature will interact with the features of the historical instances to complete the feature supplement, as follows:

[0077]

[0078] in, is the feature after feature aggregation, m is the number of network stages. Before entering the next stage of feature encoding, the features will be split and the encoding will still be completed in the form of two branches. For the specific structure, please refer to the attached Figure 4 .

[0079] In a possible implementation, a diffusion model is trained based on the motion sequence, spatial control conditions and motion text description of non-target instances to obtain a trained diffusion model, which also includes: in the process of feature interaction between the input of the diffusion model and the input of the interaction network, a hierarchical loss constraint is constructed to constrain the spatial position and motion action between each instance.

[0080] Specifically, the spatial constraints between different individual instances are effectively strengthened, and hierarchical losses are used to constrain the spatial positions and limb positions between different individual instances. , predict the key points of the motion Key points to ideal movement The loss between them is expressed as:

[0081] ;

[0082] Similarly, for the limb spatial position of the ideal movement, the limb spatial position of the predicted movement Limb space for ideal movement The limb loss is expressed as:

[0083]

[0084] The update steps are as follows:

[0085]

[0086] in, and To correct the weight, For the t The movement sequence during round training, For the t +1 round of exercise sequence during training.

[0087] In a possible implementation, in step S104, the noisy initial motion sequence and the motion text description of the target instance are used as inputs of the denoising process of the trained diffusion model to obtain the motion action and motion trajectory of the target instance, such as Figure 5 As shown, it specifically includes steps S501-S504.

[0088] Step S501: Using the noisy initial motion sequence as an input to the denoising process of the trained diffusion model to obtain a motion sequence of a first human instance in the target instance.

[0089] Step S502, taking the noisy initial motion sequence of the second human instance in the target instance and the motion sequence of the historical human instance as the input of the denoising process of the trained diffusion model, to obtain the motion sequence of the second human instance, and the motion sequence of the historical human instance is the motion sequence of the first human instance.

[0090] Step S503, taking the motion sequence of the first human body instance and the motion sequence of the second human body instance as new motion sequences of historical human body instances to obtain the motion sequence of the next human body instance; until the motion sequence of each human body instance in the target instance is obtained.

[0091] Step S504, obtaining the motion action and motion trajectory of the target instance according to the motion sequence of each human body instance in the target instance.

[0092] In an embodiment of the present invention, the noisy initial motion sequence is used as the input of the denoising process of the trained diffusion model. The motion of each individual instance is split according to the input text. The motion sequence of the first individual instance is first derived. Each subsequent step requires not only the input of the current instance motion sequence noise, but also the addition of the motion sequence of the historical instance to achieve interaction. At the same time, text condition information is embedded in the two interaction branches respectively. The current instance completes reverse sampling and generation based on the above conditions to obtain the motion action and motion trajectory of the target instance.

[0093] The present invention can achieve the following beneficial effects by adopting the above method:

[0094] 1. Train the diffusion model through the motion sequence, spatial control conditions and motion text description of the non-target instance to obtain the trained diffusion model; use the noisy initial motion sequence and the motion text description of the target instance as the input of the denoising process of the trained diffusion model to obtain the motion action and motion trajectory of the target instance, while ensuring the rationality of each human instance action, ensuring the accuracy of the interactive behavior.

[0095] 2. The Mamba module is used as the encoder to build spatiotemporal associations, enhance the spatiotemporal interactions of long sequences while alleviating the network computational cost, and design the Transformer interaction module to build the spatiotemporal associations between the target instance and the historical instance.

[0096] 3. In the denoising process, hierarchical loss constraints are added to different individual instances to ensure the rationality of the actions of each individual instance and further ensure the accuracy of the interactive behavior.

[0097] 4. Construct a hierarchical motion text description and use it as the motion text description of the target instance. In the denoising process, the importance of multi-level individual motion text description to the rationality of multi-person interactive action generation is further considered.

[0098] The multi-person interactive action generating device provided by the present invention is described below. The multi-person interactive action generating device described below and the multi-person interactive action generating method described above can be referenced to each other.

[0099] Figure 6 A schematic diagram of the structure of a multi-person interactive action generation device provided by the present invention includes the following parts:

[0100] An instance acquisition unit 610 is used to acquire an initial motion sequence and a motion text description of a target instance, where the target instance is a multi-person interaction instance;

[0101] A forward noise adding unit 620, configured to add Gaussian noise to the initial motion sequence of the target instance to obtain a noisy initial motion sequence;

[0102] A model training unit 630 is used to train a diffusion model based on a motion sequence, a spatial control condition and a motion text description of a non-target instance to obtain a trained diffusion model;

[0103] The reverse denoising unit 640 is used to use the noisy initial motion sequence and the motion text description of the target instance as inputs of the denoising process of the trained diffusion model to obtain the motion action and motion trajectory of the target instance.

[0104] In a possible implementation, the instance acquisition unit 610 specifically includes a motion text description acquisition subunit. The motion text description acquisition subunit is used to acquire an initial motion text description of the target instance; split the initial motion text description to obtain multiple single-person motion text descriptions; construct a hierarchical motion text description based on the multiple single-person motion text descriptions, and use the hierarchical motion text description as the motion text description of the target instance.

[0105] In a possible implementation, the model training unit 630 is specifically configured to:

[0106] The noisy initial motion sequence and the motion text description of the target instance are used as inputs of the diffusion model;

[0107] The motion sequence of non-target instances, spatial control conditions, and textual descriptions of the motions are used as inputs to the interaction network;

[0108] The input of the diffusion model and the input of the interaction network are subjected to feature interaction, and the input of the interaction network is embedded in the diffusion model to obtain a trained diffusion model.

[0109] In one possible implementation, the diffusion model and the interaction network are constructed through Mamda modules and have the same structure.

[0110] In a possible implementation, the model training unit 630 is further used to construct hierarchical loss constraints during the feature interaction between the input of the diffusion model and the input of the interaction network, so as to constrain the spatial position and motion action between each instance.

[0111] In a possible implementation manner, the reverse denoising unit 640 is specifically configured to:

[0112] Using the noisy initial motion sequence as an input to the denoising process of the trained diffusion model to obtain the motion sequence of the first human instance in the target instance;

[0113] The noisy initial motion sequence of the second human instance in the target instance and the motion sequence of the historical human instance are used as inputs of the denoising process of the trained diffusion model to obtain the motion sequence of the second human instance, and the motion sequence of the historical human instance is the motion sequence of the first human instance;

[0114] The motion sequence of the first human body instance and the motion sequence of the second human body instance are used as the motion sequence of the new historical human body instance to obtain the motion sequence of the next human body instance; until the motion sequence of each human body instance in the target instance is obtained;

[0115] According to the motion sequence of each human instance in the target instance, the motion action and motion trajectory of the target instance are obtained.

[0116] Figure 7 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 7 As shown, the electronic device may include: a processor 710, a communications interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communications interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute a method for generating a multi-person interactive action, the method comprising:

[0117] Obtaining an initial motion sequence and a motion text description of a target instance, where the target instance is a multi-person interaction instance;

[0118] Adding Gaussian noise to the initial motion sequence of the target instance to obtain a noisy initial motion sequence;

[0119] Based on the motion sequence, spatial control conditions and motion text description of the non-target instance, a diffusion model is trained to obtain a trained diffusion model;

[0120] The noisy initial motion sequence and the motion text description of the target instance are used as the input of the denoising process of the trained diffusion model to obtain the motion action and motion trajectory of the target instance.

[0121] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0122] On the other hand, the present invention further provides a computer program product, the computer program product includes a computer program, the computer program can be stored in a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute a multi-person interactive action generation method provided by the above methods, the method comprising:

[0123] Obtaining an initial motion sequence and a motion text description of a target instance, where the target instance is a multi-person interaction instance;

[0124] Adding Gaussian noise to the initial motion sequence of the target instance to obtain a noisy initial motion sequence;

[0125] Based on the motion sequence, spatial control conditions and motion text description of the non-target instance, a diffusion model is trained to obtain a trained diffusion model;

[0126] The noisy initial motion sequence and the motion text description of the target instance are used as the input of the denoising process of the trained diffusion model to obtain the motion action and motion trajectory of the target instance.

[0127] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, a method for generating a multi-person interactive action provided by the above methods is implemented, the method comprising:

[0128] Obtaining an initial motion sequence and a motion text description of a target instance, where the target instance is a multi-person interaction instance;

[0129] Adding Gaussian noise to the initial motion sequence of the target instance to obtain a noisy initial motion sequence;

[0130] Based on the motion sequence, spatial control conditions and motion text description of the non-target instance, a diffusion model is trained to obtain a trained diffusion model;

[0131] The noisy initial motion sequence and the motion text description of the target instance are used as the input of the denoising process of the trained diffusion model to obtain the motion action and motion trajectory of the target instance.

[0132] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0133] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0134] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating multi-person interactive actions, characterized in that: include: Acquire an initial motion sequence and a motion text description of a target instance, wherein the target instance is a multi-person interaction instance; Adding Gaussian noise to the initial motion sequence of the target instance to obtain a noisy initial motion sequence; Based on the motion sequence, spatial control conditions and motion text description of the non-target instance, a diffusion model is trained to obtain a trained diffusion model; Using the noisy initial motion sequence and the motion text description of the target instance as inputs of the denoising process of the trained diffusion model to obtain the motion action and motion trajectory of the target instance; The method of training a diffusion model based on the motion sequence, spatial control conditions and motion text description of the non-target instance to obtain a trained diffusion model specifically includes: taking the noisy initial motion sequence and the motion text description of the target instance as inputs of the diffusion model; taking the motion sequence, spatial control conditions and motion text description of the non-target instance as inputs of the interactive network; performing feature interaction between the input of the diffusion model and the input of the interactive network, and embedding the input of the interactive network into the diffusion model to obtain a trained diffusion model; The method of training a diffusion model based on the motion sequence, spatial control conditions and motion text description of non-target instances to obtain a trained diffusion model also includes: in the process of feature interaction between the input of the diffusion model and the input of the interactive network, constructing a hierarchical loss constraint to constrain the spatial position and motion action between each instance.

2. The method for generating multi-person interactive actions according to claim 1, characterized in that: Obtaining a motion text description of the target instance, specifically including: Obtaining an initial motion text description of the target instance; Splitting the initial movement text description to obtain multiple single-person movement text descriptions; A hierarchical motion text description is constructed based on the multiple single-person motion text descriptions, and the hierarchical motion text description is used as the motion text description of the target instance.

3. The method for generating multi-person interactive actions according to claim 1, characterized in that: The diffusion model and the interaction network were constructed by Mamda modules and had the same structure.

4. The method for generating multi-person interactive actions according to claim 1, characterized in that: The step of using the noisy initial motion sequence and the motion text description of the target instance as inputs of the denoising process of the trained diffusion model to obtain the motion action and motion trajectory of the target instance specifically includes: Using the noisy initial motion sequence as an input of a denoising process of the trained diffusion model to obtain a motion sequence of a first human instance in the target instance; Using the noisy initial motion sequence of the second human body instance in the target instance and the motion sequence of the historical human body instance as inputs of the denoising process of the trained diffusion model to obtain the motion sequence of the second human body instance, and the motion sequence of the historical human body instance is the motion sequence of the first human body instance; Using the motion sequence of the first human body instance and the motion sequence of the second human body instance as a new motion sequence of a historical human body instance to obtain a motion sequence of a next human body instance; until the motion sequence of each human body instance in the target instance is obtained; According to the motion sequence of each human body instance in the target instance, the motion action and motion trajectory of the target instance are obtained.

5. A multi-person interactive action generating device, characterized in that: include: An instance acquisition unit, used to acquire an initial motion sequence and a motion text description of a target instance, wherein the target instance is a multi-person interaction instance; A forward noise adding unit, used for adding Gaussian noise to the initial motion sequence of the target instance to obtain a noisy initial motion sequence; A model training unit, used for training a diffusion model based on a motion sequence of a non-target instance, a spatial control condition and a motion text description, to obtain a trained diffusion model; A reverse denoising unit, used for taking the noisy initial motion sequence and the motion text description of the target instance as inputs of the denoising process of the trained diffusion model to obtain the motion action and motion trajectory of the target instance; The method of training the diffusion model based on the motion sequence, spatial control conditions and motion text description of the non-target instance to obtain the trained diffusion model specifically includes: taking the noisy initial motion sequence and the motion text description of the target instance as inputs of the diffusion model; taking the motion sequence, spatial control conditions and motion text description of the non-target instance as inputs of the interactive network; performing feature interaction between the input of the diffusion model and the input of the interactive network, and embedding the input of the interactive network into the diffusion model to obtain the trained diffusion model; The method of training a diffusion model based on the motion sequence, spatial control conditions and motion text description of non-target instances to obtain a trained diffusion model also includes: in the process of feature interaction between the input of the diffusion model and the input of the interactive network, constructing a hierarchical loss constraint to constrain the spatial position and motion action between each instance.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the multi-person interactive action generating method according to any one of claims 1 to 4 is implemented.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the multi-person interactive action generation method according to any one of claims 1 to 4 is implemented.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the multi-person interactive action generation method according to any one of claims 1 to 4 is implemented.

Images