Method and apparatus for generating virtual object action, and computer device

Abstract

The present application relates to a method and apparatus for generating a virtual object action, and a computer device. The method comprises: acquiring an action description text (202); by means of semantic analysis, parsing the action description text at a plurality of preset semantic hierarchies to obtain respective action description information of the plurality of semantic hierarchies, and acquiring a sampled noise signal for generating a virtual object action (204); respectively encoding the respective action description information of the plurality of semantic hierarchies to obtain respective action description representations of the plurality of semantic hierarchies (206); on the basis of an action description representation of a first semantic hierarchy among the plurality of semantic hierarchies, performing noise reduction processing of the first semantic hierarchy on the sampled noise signal to obtain an action feature vector outputted by means of the first semantic hierarchy (208); at each semantic hierarchy behind the first semantic hierarchy among the plurality of semantic hierarchies, on the basis of the action feature vector outputted by means of the previous semantic hierarchy and respective action description representations of at least two semantic hierarchies from the first semantic hierarchy to the present semantic hierarchy, performing noise reduction processing on the sampled noise signal to obtain action feature vectors having undergone cascaded noise reduction of the plurality of semantic hierarchies, wherein granularity levels of actions represented by the action feature vectors outputted by means of the noise reduction processing of the plurality of semantic hierarchies decrease from the highest semantic hierarchy to the lowest semantic hierarchy in a descending order (210); and decoding the action feature vectors having undergone the cascaded noise reduction to obtain the virtual object action (212).

Description

Virtual object action generation method, device and computer equipment

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to the Chinese patent application filed with the China Patent Office on August 3, 2023, with application number 202310970212.1, and application name “Virtual object action generation method, device, computer equipment and storage medium”, all contents of which are incorporated by reference into this application. Technical Field

[0003] The present application relates to the field of computer technology, and in particular to a method, apparatus, computer device, computer-readable storage medium, and computer program product for generating virtual object actions. Background Art

[0004] With the development of computer technology, text-driven virtual object action generation technology has emerged. This technology can use a piece of action description text that describes the virtual object to generate the virtual object action.

[0005] In traditional technology, the commonly used method for generating virtual object actions is to input the action description text as a control signal into a generative model (such as a generative adversarial network, variational autoencoder, diffusion model, etc.) to directly map the action description text into a virtual object action through the generative model.

[0006] However, traditional methods directly map action description text to virtual object actions, which can usually only generate coarse-grained virtual object actions and have the problem of inaccurate generated virtual object actions.

[0007] Summary of the Invention

[0008] Embodiments of the present application provide a method, apparatus, computer device, computer-readable storage medium, and computer program product for generating a virtual object action.

[0009] A method for generating a virtual object action, executed by a computer device, comprising:

[0010] Get the action description text used to describe the action of the virtual object;

[0011] Parsing the action description text at a plurality of preset semantic levels through semantic analysis to obtain action description information at each of the plurality of semantic levels, and acquiring a sampled noise signal for generating the action of the virtual object;

[0012] Encoding the action description information of each of the multiple semantic levels respectively to obtain the action description representations of each of the multiple semantic levels;

[0013] Based on the action description representation of a first semantic level among the multiple semantic levels, performing noise reduction processing of the first semantic level on the sampled noise signal to obtain an action feature vector output by the first semantic level;

[0014] At each semantic level after the first semantic level in the multiple semantic levels, based on the action feature vector output by the previous semantic level and the action description representations of at least two semantic levels from the first semantic level to the current semantic level, the sampled noise signal is subjected to denoising, thereby obtaining an action feature vector after cascade denoising of the multiple semantic levels; wherein the action granularity represented by the action feature vectors output by the denoising processing of the multiple semantic levels decreases in order from the highest semantic level to the lowest semantic level; and

[0015] The action feature vector after the cascade noise reduction is decoded to obtain the virtual object action.

[0016] A virtual object action generation device, the device comprising:

[0017] An acquisition module, used to acquire an action description text for describing an action of a virtual object;

[0018] a semantic parsing module, configured to parse the action description text at a plurality of preset semantic levels through semantic analysis, obtain action description information at each of the plurality of semantic levels, and acquire a sampled noise signal for generating the action of the virtual object;

[0019] An encoding module, configured to encode the action description information of each of the multiple semantic levels respectively to obtain action description representations of each of the multiple semantic levels;

[0020] A first noise reduction processing module is configured to perform noise reduction processing of a first semantic level among the multiple semantic levels on the sampled noise signal based on the action description representation of the first semantic level, so as to obtain an action feature vector outputted by the first semantic level;

[0021] A second noise reduction processing module is configured to perform noise reduction processing on the sampled noise signal at each semantic level after the first semantic level in the multiple semantic levels, based on the action feature vector output by the previous semantic level and the action description representations of at least two semantic levels from the first semantic level to the current semantic level, to obtain the action feature vector after cascade noise reduction of the multiple semantic levels; wherein the action granularity represented by the action feature vectors output by the noise reduction processing of the multiple semantic levels decreases in order from the highest semantic level to the lowest semantic level; and

[0022] A decoding module is used to decode the action feature vector after the cascade noise reduction to obtain the virtual object action.

[0023] A computer device includes a memory and one or more processors, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the one or more processors, the one or more processors execute the steps of the above-mentioned virtual object action generation method.

[0024] One or more non-volatile computer-readable storage media storing computer-readable instructions, on which computer-readable instructions are stored. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the above-mentioned virtual object action generation method.

[0025] A computer program product or computer program, which includes computer-readable instructions, which are stored in a computer-readable storage medium. One or more processors of a computer device read the computer-readable instructions from the computer-readable storage medium, and the one or more processors execute the computer-readable instructions, so that the computer device performs the steps of the above-mentioned virtual object action generation method.

[0026] The details of one or more embodiments of the present application are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the present application will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0028] FIG1 is a diagram illustrating an application environment of a method for generating a virtual object action according to an embodiment;

[0029] FIG2 is a schematic diagram of a flow chart of a method for generating a virtual object action in one embodiment;

[0030] FIG3 is a schematic diagram of noise reduction processing at the first semantic level in one embodiment;

[0031] FIG4 is a schematic diagram of obtaining a motion feature vector after cascade noise reduction in one embodiment;

[0032] FIG5 is a schematic diagram of a virtual object action sequence in one embodiment;

[0033] FIG6 is a schematic diagram of action description information at multiple semantic levels in one embodiment;

[0034] FIG7 is a schematic diagram of a hierarchical semantic graph according to one embodiment;

[0035] FIG8 is a schematic diagram of a hierarchical semantic graph in another embodiment;

[0036] FIG9 is a schematic diagram of generating adjusted virtual object motions by adjusting edge weights in one embodiment;

[0037] FIG10 is a schematic diagram of a noise reduction process for obtaining a motion feature vector output at the first semantic level in one embodiment;

[0038] FIG11 is a schematic diagram of predicting the added noise corresponding to the noise adding step in one embodiment;

[0039] FIG12 is a schematic diagram of the structure of a pre-trained action sequence generation model according to one embodiment;

[0040] FIG13 is an overall framework diagram of a method for generating virtual object actions in one embodiment;

[0041] FIG14 is a block diagram of a virtual object action generating device according to an embodiment;

[0042] FIG15 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0044] The virtual object action generation method provided in the embodiment of the present application can be applied to the application environment shown in Figure 1. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other servers. The server 104 obtains the action description text used to describe the virtual object action, and through semantic analysis, parses the action description text at multiple preset semantic levels to obtain the action description information of each of the multiple semantic levels, and obtains the sampling noise signal used to generate the virtual object action, and encodes the action description information of each of the multiple semantic levels respectively to obtain the action description representation of each of the multiple semantic levels. Based on the action description representation of the first semantic level in the multiple semantic levels, the sampling noise signal is subjected to the noise reduction processing of the first semantic level to obtain the action feature vector output by the first semantic level, and the action feature vector output by the first semantic level is obtained. At each semantic level after the first semantic level in the level, based on the action feature vector output by the previous semantic level and the action description representations of at least two semantic levels from the first semantic level to the current semantic level, the sampled noise signal is denoised to obtain an action feature vector after cascade denoising of multiple semantic levels; wherein, the action granularity level represented by the action feature vectors output by the denoising processing of multiple semantic levels decreases in order from the highest semantic level to the lowest semantic level, the action feature vectors after cascade denoising are decoded to obtain a virtual object action, and the virtual object action is pushed to the terminal 102 for display.

[0045] Terminal 102 may include, but is not limited to, various desktop computers, laptops, smartphones, tablet computers, IoT devices, and portable wearable devices. IoT devices may include smart speakers, smart TVs, smart air conditioners, and smart car devices. Portable wearable devices may include smart watches, smart bracelets, and head-mounted devices. Server 104 may be implemented as a standalone server or a server cluster consisting of multiple servers.

[0046] In one embodiment, as shown in FIG2 , a method for generating a virtual object action is provided. The method can be executed by a terminal or a server alone, or by a terminal and a server in collaboration. In the embodiment of the present application, the method is described by taking the application of the method to the server as an example, and includes the following steps:

[0047] Step 202: Obtain action description text for describing the action of the virtual object.

[0048] A virtual object refers to an object that can move within a virtual environment. Such an object can be a virtual character, virtual animal, or the like. For example, if the virtual environment is a three-dimensional virtual environment, a virtual object refers to a virtual character, virtual animal, or the like displayed within the three-dimensional virtual environment. The virtual object has its own shape and volume within the three-dimensional virtual environment and occupies a portion of the space within the three-dimensional virtual environment. The virtual environment is the environment provided by the client when it runs on a terminal. This virtual environment can be a simulation of the real world, a semi-simulated and semi-fictional environment, or a purely fictional environment. For example, the virtual environment can specifically be a three-dimensional virtual environment.

[0049] Among them, the virtual object action refers to the action of the virtual object when it is active in the virtual environment. For example, the virtual object action can be walking forward, standing up first and then walking forward, walking to the right, jumping forward, etc. The action description text refers to the text that describes the virtual object action. The action description text may include information such as action category, motion path, and action style. The action category refers to the category to which the virtual object action belongs. For example, the action category can be walking, running, jumping, etc. The motion path is used to indicate the direction of movement of the virtual object. For example, the motion path can be forward, left, right, etc. The action style is used to indicate the state of the virtual object when it is moving. For example, the action style can be happy, sad, etc. For example, the action description text can be a person walking forward, then turning left, and then continuing to walk to the right. The person here refers to the virtual object.

[0050] Specifically, when virtual object action generation is required, the server will obtain an action description text used to describe the virtual object action, so as to generate the virtual object action based on the action category, motion path, action style and other information in the action description text. In specific applications, the virtual object action generation in this application can be widely used in AR (Augmented Reality) / VR (Virtual Reality) content production, game content creation, 3D animation design and other scenarios to efficiently produce realistic and diverse virtual object actions.

[0051] In step 204 , the action description text is parsed at multiple preset semantic levels through semantic analysis to obtain action description information at each of the multiple semantic levels, and a sampled noise signal for generating a virtual object action is obtained.

[0052] Semantic analysis involves analyzing the meaning of each word in an action description to determine its structure and the part of speech of each word. For example, the structure of an action description might be (attributive) subject + (adverbial) predicate + (complement or attributive) + object. Another example is the part of speech of a word in an action description might be noun, verb, adverb, adjective, preposition, etc.

[0053] Among them, the semantic level refers to the angle used to describe the virtual object movement. Multiple semantic levels are used to describe the virtual object movement from multiple different angles. Different semantic levels focus on different angles. By using multiple semantic levels to describe the virtual object movement from multiple different angles, a comprehensive description of the virtual object movement can be achieved. In this embodiment, multiple semantic levels can be pre-set according to the actual application scenario. For example, the multiple semantic levels can specifically include an overall motion level, a local motion level, and an action detail level. The overall motion level is mainly used to describe the virtual object movement as a whole, the local motion level is mainly used to describe the virtual object movement through several local movements included in the virtual object movement, and the action detail level is mainly used to describe the virtual object movement through the details of several local movements.

[0054] Among them, the action description information at the semantic level refers to the information used to describe the virtual object action at the semantic level. For example, if the semantic level is the overall motion level, the action description information at the semantic level can specifically be information that describes the virtual object action as a whole. For another example, if the semantic level is the local action level, the action description information at the semantic level can specifically be verbs that represent several local actions included in the virtual object action. For another example, if the semantic level is the action detail level, the action description information at the semantic level can specifically be modifiers that modify the verbs that represent several local actions included in the virtual object action.

[0055] The sampled noise signal refers to a noise signal obtained by random sampling when generating a virtual object motion. For example, the sampled noise signal may specifically refer to a Gaussian noise signal obtained by random sampling when generating a virtual object motion.

[0056] Specifically, the semantic analysis method can be based on semantic role parsing. The server will parse the action description text at multiple preset semantic levels based on semantic role parsing to obtain action description information at multiple semantic levels, and obtain sampling noise signals for generating virtual object actions by random sampling. Among them, semantic roles refer to the different roles played by different sentence components (such as subject, object, time, place, etc.) in an event when describing an action event in a sentence. The names of these roles are usually the noun or verb part of a verb phrase. In this embodiment, semantic roles refer to the different roles played by different sentence components (such as subject, object, time, place, etc.) in the action description text. It should be noted that what semantic role a sentence component plays in a sentence depends on the predicate verb.

[0057] In specific applications, when parsing the action description text at multiple preset semantic levels based on semantic role parsing, the server will first split the action description text into multiple different sentence components, and identify verbs from the action description text. Then, based on the semantic association between multiple different sentence components and verbs, it will determine the roles played by different sentence components and obtain action description information at multiple semantic levels.

[0058] In a specific application, the server can implement semantic analysis through a pre-trained natural language model for semantic parsing, and parse the action description text at multiple preset semantic levels. By inputting the action description text into the pre-trained natural language model for semantic parsing, action description information at multiple semantic levels can be obtained. Among them, the pre-trained natural language model for semantic parsing can be trained according to the actual application scenario. For example, the pre-trained natural language model for semantic parsing can specifically be a BERT (Bidirectional Encoder Representations from Transformers) model for relationship extraction and semantic role labeling.

[0059] In a specific application, the server can also use a semantic role parsing tool to implement semantic analysis, parse the action description text at multiple preset semantic levels, and input the action description text into the semantic role parsing tool to obtain action description information at multiple semantic levels. Among them, the semantic role parsing tool can be selected according to the actual application scenario. For example, the semantic role parsing tool can be AllenNLP (an NLP (Natural Language Processing) research library based on PyTorch (an open source Python machine learning library based on Torch, used for applications such as natural language processing) to provide the industry's best and most advanced deep learning models in various language tasks).

[0060] Step 206 : Encode the action description information of each of the multiple semantic levels respectively to obtain the action description representations of each of the multiple semantic levels.

[0061] The action description representation refers to a feature that can represent the action description information in the semantic level. For example, the action description representation refers to a feature vector that can represent the action description information in the semantic level.

[0062] Specifically, the server encodes each action description at each semantic level to obtain a first feature vector for each action description. Based on the first feature vectors, the server then obtains action description representations for each semantic level. The first feature vector represents the content of the action description and distinguishes the action description from other information.

[0063] In a specific application, the server can encode the action description information of each semantic level in multiple semantic levels through a pre-trained natural language model for text feature extraction to obtain the first feature vector of each action description information. Among them, the pre-trained natural language model for text feature extraction can be trained according to the actual application scenario. For example, the pre-trained natural language model for text feature extraction is specifically a CLIP (Contrastive Language-Image Pre-Training) model. The CLIP model is a pre-trained model that can be trained using unlabeled data. The trained CLIP model can input a piece of text (or an image) and output a vector representation of the text (image). In this embodiment, the action description information is input and the vector representation of the action description information, i.e., the first feature vector, is output. Unlike other single text modal and single image modal models, CLIP is multimodal and includes two aspects: image processing and text processing.

[0064] In a specific application, the pre-training task of the CLIP model is to predict whether a given image and text are a pair, using the loss of contrastive learning. In this embodiment, the contrastive learning method is used to pre-train the CLIP model, directly treating the image and the corresponding text as a whole to determine whether the text and image are a pair. The main structure of the CLIP model includes a text encoder and an image encoder. During training, the CLIP model inputs the image and text used for training into the image encoder and the text encoder respectively to obtain vector representations of the image and text. The vector representations of the image and text are then mapped to a common multimodal space to obtain new vector representations of the image and text that can be directly compared. Finally, the similarity between the vector representations of the image and text is calculated. The objective function of contrastive learning is to make the similarity of positive sample pairs higher and the similarity of negative sample pairs lower.

[0065] In a specific application, after obtaining the first feature vector of each action description information, for each semantic level, the server can fuse the first feature vectors of at least two action description information belonging to the semantic level, and use the fused feature vector as the action description representation of the semantic level, to obtain the action description representations of multiple semantic levels. In a specific application, for each semantic level, the server can fuse the first feature vectors of at least two action description information belonging to the semantic level by splicing, superposition, etc.

[0066] Furthermore, before fusing the first feature vectors of the action description information belonging to the semantic level, the server may also update the first feature vectors of each action description information based on the semantic association relationship between at least one pair of action description information between different semantic levels, so as to achieve accurate representation of each action description information in combination with the context content.

[0067] Step 208 : Based on the action description representation of the first semantic level among the multiple semantic levels, perform noise reduction processing of the first semantic level on the sampled noise signal to obtain an action feature vector outputted at the first semantic level.

[0068] The noise reduction process refers to removing noise from the sampled noise signal. The action feature vector outputted at the first semantic level refers to a vector that can represent the features of the action of the virtual object at the first semantic level.

[0069] Specifically, in the noise reduction processing of the first semantic level among multiple semantic levels, the server, under the guidance of the action description representation of the first semantic level, performs the noise reduction processing of the first semantic level on the sampled noise signal, and reconstructs the action feature vector output by the first semantic level. In a specific application, the server will treat the sampled noise signal as a noise signal that has undergone multi-step noise addition, and then predict the noise signal added in each step of the multi-step noise addition based on the action description representation of the first semantic level. Based on the noise signal added in each step, the server will gradually perform noise reduction processing on the sampled noise signal, and gradually subtract the noise signal added in each step from the sampled noise signal to obtain the action feature vector output by the first semantic level.

[0070] It should be noted that the action description representation at the first semantic level exists as a condition for generating the action feature vector, and is used to guide the generation of the action feature vector, so that the generated action feature vector can be more relevant to the action description representation at the first semantic level.

[0071] In a specific application, the noise reduction processing of the first semantic level can be shown in Figure 3. The sampled noise signal n is used as the noise signal after multi-step noise addition (T-step noise addition is shown in Figure 3). The noise signal added in each step of the multi-step noise addition is predicted based on the action description representation of the first semantic level. Based on the noise signal added in each step, the sampled noise signal n is gradually subjected to noise reduction processing. The noise signal added in each step is gradually subtracted from the sampled noise signal to obtain the action feature vector outputted at the first semantic level. As shown in Figure 3, the server starts from the last step of the multi-step noise addition (number of noise addition steps T), and performs reverse noise reduction processing on the input noise signal based on the action description representation of the first semantic level. At the last step of the multi-step noise addition, the noise signal obtained after noise reduction is In the penultimate step of multi-step denoising (denoising step number T-1), the input noise signal is the noise signal obtained after denoising output in the last step (denoising step number T). For the noise signal input in the first step (as shown in Figure 3 ) is the noise reduction signal obtained by noise reduction processing, which is the action feature vector output by the first semantic level (as shown in Figure 3 as z′ i ).

[0072] Step 210: At each semantic level after the first semantic level in the multiple semantic levels, based on the action feature vector output by the previous semantic level and the action description representations of at least two semantic levels from the first semantic level to the current semantic level, the sampled noise signal is subjected to denoising processing to obtain the action feature vector after cascade denoising of multiple semantic levels; wherein, the action granularity level represented by the action feature vectors output by the denoising processing of multiple semantic levels decreases in order from the highest semantic level to the lowest semantic level.

[0073] Granularity refers to the degree of detail of data statistics within a given dimension. In this embodiment, the given dimension refers to the dimension of describing virtual object actions. Therefore, the action granularity level refers to the degree of detail of the description of virtual object actions, i.e., the level of refinement or comprehensiveness of the description of virtual object actions. It is understood that the more refined the description of virtual object actions, the smaller the action granularity represented by the action feature vector. Conversely, the less refined the description of virtual object actions, the larger the action granularity represented by the action feature vector.

[0074] In this embodiment, the action granularity level represented by the action feature vectors output by the noise reduction processing of multiple semantic levels decreases in order from the highest semantic level to the lowest semantic level. That is, the action granularity level represented by the action feature vector output by the noise reduction processing of the first semantic level, which is the highest semantic level, is the largest, and the action granularity level represented by the action feature vector output by each semantic level after the first semantic level decreases from semantic level to semantic level, that is, the action granularity level becomes smaller from semantic level to semantic level. It can be understood that the smaller the action granularity level, the higher the degree of refinement of the action feature vector in describing the action of the virtual object, that is, the finer the action granularity, and the richer and more fine-grained motion details that can be included.

[0075] Specifically, at each semantic level after the first semantic level in the multiple semantic levels, the server will perform noise reduction processing on the sampled noise signal based on the action feature vector output by the previous semantic level and the action description representations of at least two semantic levels from the first semantic level to the current semantic level, so as to obtain the action feature vector after cascade noise reduction of multiple semantic levels. In a specific application, in the noise reduction processing of each semantic level after the first semantic level, the server will use the sampled noise signal as the noise signal that has undergone multi-step noise addition, and predict the noise signal added in each step of the multi-step noise addition based on the action feature vector output by the previous semantic level and the action description representations of at least two semantic levels from the first semantic level to the current semantic level, and based on the noise signal added in each step, gradually perform noise reduction processing on the sampled noise signal to obtain the action feature vector output by the semantic level.

[0076] In a specific application, in the noise reduction processing of each semantic level after the first semantic level, the server will start from the last step of multi-step noise addition, and perform reverse noise reduction processing on the noise signal input at each step based on the action feature vector output by the previous semantic level and the action description representations of at least two semantic levels from the first semantic level to the current semantic level. The noise reduction signal obtained by performing noise reduction processing on the noise signal input at the first step in the multi-step noise addition is used as the action feature vector output by the semantic level.

[0077] In a specific application, in the noise reduction processing of each semantic level after the first semantic level, for each step in the multi-step noise addition, the server will encode the number of steps of the targeted noise addition step to obtain the noise addition step feature, and then fuse the noise addition step feature, the action feature vector output of the previous semantic level, and the action description representations of at least two semantic levels from the first semantic level to the current semantic level to obtain the noise reduction condition feature. Then, based on the noise reduction condition feature and the noise signal input to the targeted noise addition step, the noise added to the targeted noise addition step is predicted. Based on the predicted added noise, the noise signal input to the targeted noise addition step is subjected to noise reduction processing to obtain a noise reduction signal.

[0078] In a specific application, the noise reduction processing of each semantic level can be implemented by using a noise reducer, and the cascade noise reduction can specifically refer to the noise reduction processing of the sampled noise signal by using multiple noise reducers connected in series. For example, as shown in Figure 4, the server can connect three noise reducers R1, R2, and R3 in series to gradually represent the sampled noise signal n and the action descriptions of multiple semantic levels (as shown in Figure 4, the action description representation of the first semantic level is C1, and the two action description representations of the second semantic level are and The three action descriptions at the third semantic level are represented as and ) to obtain the action feature vector after cascade denoising. The denoising processing of each semantic level is implemented by iterative denoising processing using a denoiser. The denoising processing of the first semantic level is implemented using denoiser R1. At each semantic level after the first semantic level, the server will perform denoising on the sampled noise signal n based on the action description representations of at least two semantic levels from the first semantic level to the current semantic level and the action feature vector output by the previous semantic level (as shown in FIG4 , the output of denoiser R1 is Z1, and the output of denoiser R2 is Z2) to obtain the action feature vector after cascade denoising (as shown in FIG4 , the output of denoiser R3 is Z3).

[0079] In a specific application, the denoiser R2 will represent the action description C1 of the first semantic level and the action description representation of the current semantic level. and The noise reduction device R3 uses the output Z1 of the noise reduction device R1 as a joint condition to reduce the noise of the sampled noise signal n and reconstruct the action feature vector Z2 from the noise of the sampled noise signal n. The noise reduction device R3 will use the action description representation C1 of the first semantic level and the action description representation C2 of the previous semantic level to reduce the noise of the sampled noise signal n. and Action description representation at this semantic level and And Z2 output by the denoiser R2 are used as joint conditions to perform denoising on the sampled noise signal n, and reconstruct the action feature vector Z3 from the sampled noise signal n, that is, the action feature vector after cascade denoising.

[0080] Step 212: Decode the action feature vector after cascade noise reduction to obtain the virtual object action.

[0081] Specifically, the server can obtain the virtual object action by decoding the action feature vector after cascade noise reduction. In a specific application, decoding the action feature vector after cascade noise reduction, that is, converting the action feature vector after cascade noise reduction back to the virtual object's posture space through mapping, the obtained virtual object action can be a virtual object action sequence. That is, through the virtual object action generation method involved in this application, it is possible to generate a corresponding virtual object action sequence from given action description information.

[0082] In a specific application, the given action description information can be in Chinese or in text in other languages. Taking the action description information in Chinese as an example, as shown in FIG5 , 10 examples of corresponding virtual object action sequences generated from the given action description information are given. It can be seen from the example in FIG5 that high-quality virtual object action sequences can be generated through the virtual object action generation method involved in this application.

[0083] The above-mentioned virtual object action generation method obtains an action description text for describing the virtual object action, parses the action description text at multiple preset semantic levels through semantic analysis, obtains action description information for each of the multiple semantic levels, obtains a sampled noise signal for generating the virtual object action, encodes the action description information for each of the multiple semantic levels, and obtains action description representations for each of the multiple semantic levels. Based on the action description representation of the first semantic level among the multiple semantic levels, the sampled noise signal is denoised at the first semantic level to obtain an action feature vector output by the first semantic level. At each semantic level after the first semantic level among the multiple semantic levels, the sampled noise signal is denoised using the action feature vector output by the previous semantic level and the action description representations of at least two semantic levels from the first semantic level to the current semantic level as joint conditions. The action description representations of each of the multiple semantic levels can be used to gradually enrich fine-grained motion details, and a more fine-grained action feature vector that has undergone cascade denoising of multiple semantic levels is obtained, which accurately represents the virtual object action. The virtual object action can then be obtained by decoding the cascade denoised action feature vector. The entire process can use the action description information of multiple semantic levels as fine-grained control signals, and refine the generated virtual object actions by capturing the action features of multiple semantic levels, thereby improving the accuracy of the generated virtual object actions.

[0084] In one embodiment, multiple semantic levels include an overall motion level, a local action level, and an action detail level; through semantic analysis, the action description text is parsed at multiple preset semantic levels to obtain action description information for each of the multiple semantic levels, including: taking the action description text as the action description information of the overall motion level, and extracting at least one verb and a phrase corresponding to the at least one verb for modifying the verb from the action description text; taking at least one verb as the action description information of the local action level, and taking the phrase corresponding to the at least one verb for modifying the verb as the action description information of the action detail level.

[0085] Among them, the overall motion level is mainly used to describe the virtual object action as a whole. Describing the virtual object action as a whole means describing the virtual object action from the perspectives of the object that performs the virtual object action, the local action performed, and the details of the local action. The local action level is mainly used to describe the virtual object action through several local actions included in the virtual object action, and the action detail level is mainly used to describe the virtual object action through the details of several local actions. For example, the details of the local action can be the direction of the execution of the virtual object action, the state of the virtual object during execution, etc. The corresponding phrases used to modify the verb refer to the phrases used to modify the verb in the sentence. For example, the verb-related phrases used to modify the verb can specifically refer to adjectives, adverbs, prepositions, etc. that modify the verb.

[0086] Specifically, the multiple semantic levels include an overall motion level, a local action level, and an action detail level. Through semantic analysis, the action description text is parsed at multiple preset semantic levels. That is, considering multiple semantic levels, the action description information of each semantic level is extracted from the action description text. When extracting the action description information of multiple semantic levels, the server will use the action description text as the action description information of the overall motion level, and extract at least one verb and a phrase corresponding to the at least one verb for modifying the verb from the action description text, use the at least one verb as the action description information of the local action level, and use the phrase corresponding to the at least one verb for modifying the verb as the action description information of the action detail level.

[0087] In a specific application, the server can determine the part of speech of each word by performing part-of-speech analysis on each word in the action description text to determine at least one verb, and then determine the phrases corresponding to at least one verb for modifying the verb by analyzing the relationship between the at least one verb and each word.

[0088] In a specific application, taking the action description text "a person walks forward, then turns left, and then continues walking to the right" as an example, at least one verb that can be extracted from it includes: walk, turn, and continue walking. The phrases corresponding to "walk" used to modify the verb are "a person", "forward", the phrases corresponding to "turn" used to modify the verb are "a person", "then", and "left", and the phrases corresponding to "continue walking" used to modify the verb are "a person", "afterwards", and "right". After semantic hierarchical analysis, the action description information of multiple semantic levels obtained can be shown in Figure 6. The action description information at the overall movement level is "a person walks forward, then turns left, and then continues walking to the right", the action description information at the local action level is "walk", "turn", and "continue walking", and the action description information at the action detail level is "a person", "forward", "then", "left", "afterwards", and "right".

[0089] In this embodiment, in this way, it is possible to obtain action description information at the overall motion level, local action level, and action detail level, and then use action description information at multiple semantic levels as fine-grained control signals. By capturing action features at multiple semantic levels, the generated virtual object action can be refined, thereby improving the accuracy of the generated virtual object action.

[0090] In one embodiment, the action description information of each of the multiple semantic levels is encoded respectively to obtain the action description representation of each of the multiple semantic levels, including: encoding each action description information of each semantic level in the multiple semantic levels to obtain a first feature vector of each action description information; based on the semantic association relationship between the action description information of at least one pair of different semantic levels, updating the first feature vector of each action description information based on the attention mechanism to obtain a second feature vector of each action description information; for each semantic level, splicing the second feature vectors of at least two action description information belonging to the semantic level, and using the spliced ​​second feature vector as the action description representation of the semantic level to obtain the action description representation of each of the multiple semantic levels.

[0091] The first feature vector refers to the vector used to represent the action description information after it is encoded. A semantic association relationship refers to the semantic relationship between two action descriptions as a pair. For example, when two action descriptions are verbs and their modifiers, such as adverbs, adjectives, and prepositions, the two action descriptions can be considered to have a semantic association relationship. The second feature vector refers to the vector used to represent the action description information after the first feature vector is updated.

[0092] Specifically, the server will encode each action description information of each semantic level in multiple semantic levels respectively to obtain the first feature vector of each action description information. Based on the semantic association relationship between the action description information of at least one pair of different semantic levels, the server will update the first feature vector of the action description information with the semantic association relationship based on the attention mechanism to obtain the updated second feature vector of each action description information. For each semantic level, the second feature vectors of at least two action description information belonging to the semantic level will be spliced, and the spliced ​​second feature vector will be used as the action description representation of the semantic level to obtain the action description features of multiple semantic levels.

[0093] In a specific application, the server can encode each action description information of each semantic level in multiple semantic levels through a pre-trained natural language model for text feature extraction to obtain the first feature vector of each action description information. When performing an update process based on the attention mechanism, for each action description information, the server will perform an interactive process based on the attention mechanism on the first feature vector of the action description information and the action description information that has a semantic association relationship with the action description information, determine the attention weight coefficient of the action description information and the action description information that has a semantic association relationship with the action description information, and then perform a weighted sum of the first feature vectors of the action description information and the action description information that has a semantic association relationship with the action description information according to the attention weight coefficient to obtain the second feature vector of the action description information.

[0094] In this embodiment, the first feature vector of each action description information can be obtained by encoding. By utilizing the semantic association relationship, the first feature vector is updated based on the attention mechanism. On the basis of fully considering the action description information with semantic association relationship, the second feature vector that accurately expresses each action description information can be obtained. Then, by splicing the second feature vectors of the action description information belonging to the same semantic level, the action description representations of multiple semantic levels can be obtained.

[0095] In one embodiment, based on the semantic association relationship between action description information between at least one pair of different semantic levels, the first feature vector of each action description information is updated based on the attention mechanism to obtain the second feature vector of each action description information, including: taking each action description information as a semantic node, and based on the semantic association relationship between action description information between at least one pair of different semantic levels, connecting the two semantic nodes representing a pair of action description information with a semantic association relationship to determine the connecting edge connecting each semantic node; taking the first feature vector of each action description information as the node representation of each semantic node; constructing a hierarchical semantic graph based on each semantic node, the connecting edge connecting each semantic node, and the node representation of each semantic node; using the graph attention mechanism to update the node representation of each semantic node in the hierarchical semantic graph, and obtaining the second feature vector of each action description information based on the updated node representation of each semantic node.

[0096] The graph attention mechanism is used to introduce an attention mechanism to achieve better neighbor aggregation. By learning the weights of neighbors, weighted aggregation of neighbors can be achieved. Therefore, using graph attention to remember noisy neighbors is more robust, and the attention mechanism also gives the model a certain degree of interpretability. It is important to note that the graph attention mechanism further enhances graph-based reasoning compared to simple graph convolution by dynamically focusing on neighborhood features.

[0097] Specifically, the server will treat each action description information as a semantic node, and based on the semantic association relationship between at least one pair of action description information between different semantic levels, connect the two semantic nodes representing a pair of action description information with a semantic association relationship (action description information between different semantic levels) to obtain a connection edge connecting each semantic node. On this basis, the server will also use the first feature vector of each action description information as the node representation of each semantic node, and then construct a hierarchical semantic graph based on each semantic node, the connection edge connecting each semantic node, and the node representation of each semantic node. After constructing the hierarchical semantic graph, the server will use the graph attention mechanism to update the node representation of each semantic node in the hierarchical semantic graph, and use the updated node representation of each semantic node as the second feature vector of the action description information corresponding to the semantic node.

[0098] In specific applications, when using the graph attention mechanism to update the node representation of each semantic node in the hierarchical semantic graph, for each semantic node in the hierarchical semantic graph, the server will determine at least one adjacent node of the targeted semantic node, and then use the node representation of at least one adjacent node and the node representation of the targeted semantic node to update the node representation of the targeted semantic node.

[0099] In a specific application, taking the action description text as "a person walks forward, then turns left, and then continues to walk to the right", and multiple semantic levels including the overall motion level, the local action level and the action detail level as an example, the constructed hierarchical semantic graph can be shown in Figure 7, wherein the semantic node "a person walks forward, then turns left, and then continues to walk to the right" of the overall motion level is connected with the semantic nodes "walk", "turn" and "continue walking" of the local action level, the semantic node "walk" of the local action level is connected with the semantic nodes "a person" and "forward" of the action detail level, the semantic node "turn" of the local action level is connected with the semantic nodes "a person", "then" and "left" of the action detail level, and the semantic node "continue walking" of the local action level is connected with the semantic nodes "a person", "afterwards" and "right" of the action detail level. In a specific application, the constructed hierarchical semantic graph can be simplified as shown in Figure 8, where the overall motion level includes one semantic node (also called the overall motion node), the local action level includes three semantic nodes (also called the local action nodes), and the action detail level includes six semantic nodes (also called the action detail nodes).

[0100] In this embodiment, on the basis of determining the semantic nodes, determining the connecting edges connecting each semantic node based on the semantic association relationship, and determining the node representation of each semantic node, each semantic node, the connecting edges connecting each semantic node, and the node representation of each semantic node can be used to realize the construction of a hierarchical semantic graph that represents the semantic hierarchical relationship of the action description text. Then, the graph attention mechanism can be used to update the node representation of each semantic node in the hierarchical semantic graph, so that the node representation of each semantic node fully interacts to obtain the updated node representation of each semantic node. Then, the updated node representation of each semantic node can be used to obtain a second feature vector that accurately expresses the description information of each action.

[0101] In one embodiment, the graph attention mechanism is used to update the node representation of each semantic node in the hierarchical semantic graph, including: for each semantic node in the hierarchical semantic graph, determining at least one adjacent node of the targeted semantic node; performing interactive processing based on the graph attention mechanism on the node representation of at least one adjacent node and the node representation of the targeted semantic node to determine the attention weight coefficient of at least one adjacent node and the targeted semantic node; and performing weighted summation on the node representation of at least one adjacent node and the node representation of the targeted semantic node according to the attention weight coefficient to obtain an updated node representation of the targeted semantic node.

[0102] Among them, adjacent nodes refer to semantic nodes that are connected to the targeted semantic nodes through connecting edges in the hierarchical semantic graph. For example, in the hierarchical semantic graph shown in Figure 7, at least one adjacent node of the semantic node "a person walks forward, then turns left, and then continues to walk to the right" at the overall movement level is the semantic node "walk", "turn" and "continue to walk" at the local action level. At least one adjacent node of the semantic node "walk" at the local action level is the semantic node "a person walks forward, then turns left, and then continues to walk to the right" at the overall movement level and the semantic nodes "a person" and "forward" at the action detail level. At least one adjacent node of the semantic node "turn" at the local action level is the semantic node "a person walks forward, then turns left, and then continues to walk to the right" at the overall movement level and the semantic nodes "a person", "then" and "left" at the action detail level. At least one adjacent node of the semantic node "continue walking" at the local action level is the semantic node "a person walks forward, then turns left, and then continues walking to the right" at the overall movement level and the semantic nodes "a person", "afterwards" and "right" at the action detail level.

[0103] Specifically, for each semantic node in the hierarchical semantic graph, the server will determine at least one adjacent node of the targeted semantic node based on the connection relationship between the semantic nodes in the hierarchical semantic graph, and perform interactive processing based on the graph attention mechanism on the node representation of at least one adjacent node and the node representation of the targeted semantic node, and determine the attention weight coefficient of at least one adjacent node and the targeted semantic node. For each adjacent node in at least one adjacent node, the attention weight coefficient of the adjacent node represents the importance of the node representation of the adjacent node to the targeted semantic node. Based on this, the terminal can perform a weighted summation of the node representation of at least one adjacent node and the node representation of the targeted semantic node according to the attention weight coefficient to obtain an updated node representation of the targeted semantic node.

[0104] In a specific application, when performing interactive processing based on the graph attention mechanism on the node representation of at least one adjacent node and the node representation of the targeted semantic node, the server can determine the attention weight coefficient of at least one adjacent node and the targeted semantic node through similarity calculation, that is, for each adjacent node in at least one adjacent node, the server can calculate the node representation similarity between the node representation of the targeted adjacent node and the node representation of the targeted semantic node, and use the node representation similarity as the attention weight coefficient of the targeted adjacent node.

[0105] In specific applications, when performing interactive processing based on the graph attention mechanism on the node representation of at least one adjacent node and the node representation of the targeted semantic node, the server can also determine the attention weight coefficient of at least one adjacent node and the targeted semantic node by first linearly transforming and then mapping. That is, for each adjacent node in at least one adjacent node, the server will first use the pre-trained linear change layer to perform a linear transformation on the node representation of the targeted adjacent node and the node representation of the targeted semantic node, that is, map them to high-dimensional features to obtain sufficient expression ability, and then splice the two node representations after the linear transformation, and then map the two spliced ​​node representations to a real number, and use the real number as the attention weight coefficient of the targeted adjacent node.

[0106] In specific applications, the graph attention mechanism of this embodiment only allows adjacent nodes to participate in the attention mechanism of the targeted semantic node, thereby introducing graph structural information. That is, when performing interaction processing based on the graph attention mechanism, only one-hop adjacent nodes are considered. It should be noted that the one-hop adjacent nodes of the targeted semantic node include the targeted semantic node itself, which can be understood as self-loop edges.

[0107] In a specific application, after determining the attention weight coefficient of at least one adjacent node and the targeted semantic node, in order to make the attention weight coefficients between different semantic nodes easy to compare, the server can normalize the attention weight coefficient of at least one adjacent node and the targeted semantic node, and then use the normalized attention weight coefficient to perform weighted summation on the node representation of at least one adjacent node and the node representation of the targeted semantic node to obtain an updated node representation of the targeted semantic node.

[0108] In one embodiment, the server can also use the graph attention mechanism through a pre-trained graph attention network to update the node representation of each semantic node in the hierarchical semantic graph. By inputting the node representation of each semantic node in the hierarchical semantic graph into the pre-trained graph attention network, the pre-trained graph attention network can output the updated node representation of each semantic node.

[0109] It should be noted that when the pre-trained graph attention network uses the graph attention mechanism to update the node representation of each semantic node in the hierarchical semantic graph, the processing principle adopted is basically the same as the above-mentioned embodiment. For each semantic node in the hierarchical semantic graph, at least one adjacent node of the targeted semantic node is first determined, and then the node representation of at least one adjacent node and the node representation of the targeted semantic node are interactively processed based on the graph attention mechanism to determine the attention weight coefficient of at least one adjacent node and the targeted semantic node. Finally, according to the attention weight coefficient, the node representation of at least one adjacent node and the node representation of the targeted semantic node are weighted and summed to obtain the updated node representation of the targeted semantic node.

[0110] In this embodiment, for each semantic node in the hierarchical semantic graph, by first determining at least one adjacent node of the targeted semantic node, and then performing interactive processing on the node representation based on the graph attention mechanism, the attention weight coefficient of at least one adjacent node and the targeted semantic node can be obtained. Then, the node representation of the targeted semantic node can be updated by performing weighted summation on the node representation of at least one adjacent node and the node representation of the targeted semantic node according to the attention weight coefficient, so that the node representation of the targeted semantic node can fully integrate the node representation of the adjacent nodes and more accurately express the action description information.

[0111] In one embodiment, the virtual object action generation method also includes: when a virtual object action is obtained, in response to an edge weight adjustment event of the connecting edge connecting each semantic node in the hierarchical semantic graph, adjusting the edge weight of the connecting edge indicated by the edge weight adjustment event to obtain an updated hierarchical semantic graph; using the graph attention mechanism, updating the node representation of each semantic node in the updated hierarchical semantic graph, and obtaining the third eigenvector of each action description information based on the updated node representation of each semantic node; for each semantic level, splicing the third eigenvectors of at least two action description information belonging to the semantic level, and using the spliced ​​third eigenvector as the updated action description representation of the semantic level to obtain the updated action description representation of each of the multiple semantic levels; generating an adjusted virtual object action based on the updated action description representation of each of the multiple semantic levels.

[0112] An edge weight adjustment event adjusts the weights of the edges connecting semantic nodes in a hierarchical semantic graph. For example, if the initial weights of the edges connecting semantic nodes in a hierarchical semantic graph are all the same, an edge weight adjustment event can be used to adjust the weight of at least one of these edges, enabling more fine-grained control over virtual object motion generation.

[0113] Specifically, when a virtual object action is obtained, if finer-grained control of virtual object action generation is required, the interactive object can trigger an edge weight adjustment event for the connecting edges connecting each semantic node in the hierarchical semantic graph. In response to the edge weight adjustment event, the server adjusts the edge weight of the connecting edge indicated by the edge weight adjustment event to obtain an updated hierarchical semantic graph. After obtaining the updated hierarchical semantic graph, the server uses the graph attention mechanism to update the node representation of each semantic node in the updated hierarchical semantic graph, and uses the updated node representation of each semantic node as the third feature vector of the corresponding action description information. For each semantic level, the third feature vectors of at least two action description information belonging to the semantic level are spliced, and the spliced ​​third feature vector is used as the updated action description representation of the semantic level to obtain the updated action description representations of multiple semantic levels. The updated action description representations of multiple semantic levels are used to generate adjusted virtual object actions to achieve finer-grained control of virtual object action generation.

[0114] In a specific application, after obtaining the updated action description representations of multiple semantic levels, the server will perform noise reduction processing on the sampled noise signal at the first semantic level based on the updated action description representation of the first semantic level, and obtain the adjusted action feature vector output by the first semantic level. At each semantic level after the first semantic level, based on the adjusted action feature vector output by the previous semantic level and the updated action description representations of at least two semantic levels from the first semantic level to the current semantic level, the sampled noise signal is subjected to noise reduction processing to obtain the adjusted action feature vector after cascade noise reduction. The adjusted action feature vector after cascade noise reduction is decoded to obtain the adjusted virtual object action.

[0115] In specific applications, the interactive object can trigger an edge weight adjustment event for the connecting edges connecting each semantic node in the hierarchical semantic graph through voice or text. After receiving the voice or text of the interactive object, the server will recognize the voice or text, identify the adjustment method used to adjust the edge weight, and then adjust the edge weight indicated by the adjustment method. For example, taking the action description text as "a person walks forward, then turns right, and then continues to walk right", the voice or text for adjusting the edge weight can be "turn a little more to the left". After receiving the voice or text, the server will determine that the adjustment method is "turn a little more to the left" and adjust the edge weight indicated by the adjustment method (i.e., the weight of the connecting edge connecting the two semantic nodes "turn" and "left") to increase the edge weight to achieve "turn a little more to the left".

[0116] In a specific application, as shown in Figure 9, taking the action description text of "a person walks forward, then turns right, and then continues to walk to the right" as an example, the generated baseline virtual object action is shown in Figure 9. If the weight of the edge connecting the semantic node "turn" and the semantic node "left" (as shown in Figure 9, the connecting edge connecting semantic node 3 and semantic node 8) is increased (i.e., enhanced), it can be seen from the comparison between the fine-tuning result of Figure 9 (adjusted virtual object action) and the baseline virtual object action that the amplitude of turning left will become larger. If the weight of the edge connecting the semantic node "turn" and the semantic node "left" is reduced (i.e., weakened), it can be seen from the comparison between the fine-tuning result of Figure 9 (adjusted virtual object action) and the baseline virtual object action that the amplitude of turning left will become smaller.

[0117] In a specific application, if the weight of the edge connecting the semantic node "a person walks forward, then turns right, and then continues walking right" and the semantic node "continue walking" (as shown in Figure 9, the edge connecting semantic node 1 and semantic node 4) is increased (i.e., enhanced), the action of "continue walking" will be more obvious, as can be seen from the comparison of the fine-tuning result (adjusted virtual object action) in Figure 9 with the baseline virtual object action. If the weight of the edge connecting the semantic node "a person walks forward, then turns right, and then continues walking right" and the semantic node "continue walking" is reduced (i.e., weakened), the action of "continue walking" will become less obvious, as can be seen from the comparison of the fine-tuning result (adjusted virtual object action) in Figure 9 with the baseline virtual object action.

[0118] In this embodiment, the adjusted virtual object can be generated by adjusting the edge weights of the connecting edges in the hierarchical semantic graph. The edge weight adjustment is used to achieve fine-grained control over the generation of virtual objects, which can make the generated adjusted virtual object more in line with the requirements.

[0119] In one embodiment, based on the action description representation of the first semantic level among multiple semantic levels, the sampled noise signal is subjected to first semantic level noise reduction processing to obtain the action feature vector outputted at the first semantic level, including: treating the sampled noise signal as a noise signal subjected to multi-step noise addition, starting from the last step of the multi-step noise addition, based on the action description representation of the first semantic level among multiple semantic levels, performing reverse noise reduction processing on the noise signal inputted at each step, and using the noise reduction signal obtained by performing noise reduction processing on the noise signal inputted at the first step as the action feature vector outputted at the first semantic level.

[0120] Specifically, when performing noise reduction processing at the first semantic level, the server will treat the sampled noise signal as a noise signal that has undergone multi-step noise addition. Starting from the last step of the multi-step noise addition, guided by the action description representation of the first semantic level, the server will perform reverse noise reduction processing on the noise signal input at each step, and the noise reduction signal obtained by performing noise reduction processing on the noise signal input at the first step will be used as the action feature vector output at the first semantic level.

[0121] In a specific application, the noise signal input to the final step of multi-step noise addition is the sampled noise signal. Starting from the penultimate step of the multi-step noise addition, the noise signal input to each step is the noise-reduced signal output after noise reduction processing in the subsequent step. Furthermore, for each step in the multi-step noise addition, the action description representation at the first semantic level is used as a guide. Based on the action description representation at the first semantic level and the noise addition step, the noise added to the specific noise addition step is predicted. Noise reduction processing is then performed on the noise signal input to the specific noise addition step based on the predicted added noise.

[0122] In a specific application, assuming that the multi-step noise addition is T-step noise addition, when performing noise reduction processing on the sampled noise signal, T-step noise reduction processing is required. The server will treat the sampled noise signal as the noise signal that has undergone T-step noise addition. Starting from the noise reduction step number T, guided by the action description representation of the first semantic level, the noise signal input at each step is reversely noise reduced. The noise reduction signal obtained by noise reduction processing the noise signal input at the first step (noise reduction step number 1) is used as the action feature vector output at the first semantic level. When the noise reduction step number is T, the input noise signal is the sampled noise signal. Starting from the noise reduction step number T-1, the noise signal input at each step is the noise reduction signal output after the noise reduction processing in the next step.

[0123] In a specific application, the noise reduction process performed at each step can be implemented based on a pre-trained denoiser, which can be configured and trained according to the actual application scenario. The noise reduction process for obtaining the action feature vector output at the first semantic level can be shown in Figure 10. For each of the T steps of noise addition, the pre-trained denoiser can be used to predict noise based on the action description representation at the first semantic level, the input noise signal, and the noise addition step. The noise predicted by the pre-trained denoiser can then be used to denoise the input noise signal to obtain a denoised signal. After denoising the noise signal input at the first step (denoising step number 1), the denoised signal obtained from the denoising process is used as the action feature vector output at the first semantic level. During this process, the pre-trained denoiser is used T times.

[0124] As shown in Figure 10, the server starts from the last step of multi-step denoising (noise step number T) and performs reverse denoising on the input noise signal based on the action description representation of the first semantic level. At the last step of multi-step denoising, the noise signal obtained after denoising is In the penultimate step of multi-step denoising (denoising step number T-1), the input noise signal is the noise signal obtained after denoising output in the last step (denoising step number T). The noise signal of the first step input (as shown in Figure 10) ) is the noise reduction signal obtained by noise reduction processing, which is the action feature vector outputted at the first semantic level (as shown in Figure 10 as z′ i ).

[0125] In this embodiment, by taking the sampled noise signal as the noise signal that has undergone multi-step noise addition, starting from the last step of the multi-step noise addition, based on the action description representation of the first semantic level, the noise signal input at each step is subjected to reverse noise reduction processing. It is possible to achieve gradual and accurate noise reduction guided by the action description representation of the first semantic level, and obtain the action feature vector outputted at the first semantic level.

[0126] In one embodiment, for each step in multi-step noisy processing, the step of performing denoising processing on the noise signal inputted at the noisy step includes: encoding the number of steps of the noisy step to obtain a noisy step feature; fusing the action description representation at the first semantic level and the noisy step feature to obtain a denoising condition feature; and performing denoising processing on the noise signal inputted at the noisy step according to the denoising condition feature to obtain a denoised signal.

[0127] The "noise step feature" refers to a feature that identifies the targeted noise step and distinguishes it from other noise steps. The "noise reduction condition feature" refers to a feature that serves as a guide for the noise reduction process. The noise reduction process performed for different noise reduction condition features varies. For example, different noise reduction condition features may result in different noise additions to the targeted noise step during the noise reduction process.

[0128] Specifically, for each step in the multi-step noisy process, when performing noise reduction processing on the noise signal inputted at the noisy step, the server will encode the number of steps of the noisy step to obtain the noise step feature, and then fuse the action description representation at the first semantic level and the noise step feature to obtain the noise reduction condition feature. Finally, guided by the noise reduction condition feature, the noise signal inputted at the noisy step is subjected to noise reduction processing to obtain the noise reduction signal.

[0129] In specific applications, the server can encode the number of noise steps targeted using a pre-trained encoding network. The pre-trained encoding network can be configured based on the actual application scenario. For example, the pre-trained encoding network can be a pre-trained MLP (Multi-Layer Perceptron). The server can then fuse the action description representation at the first semantic level with the noise step features through concatenation to obtain the noise reduction conditional features.

[0130] In this embodiment, by encoding the number of steps of the targeted noisy step to obtain the noisy step feature, the action description representation of the first semantic level and the noisy step feature are fused to obtain the noise reduction condition feature, and then the noise reduction condition feature can be used as a guide to perform noise reduction processing on the noise signal input of the targeted noisy step to obtain a noise reduction signal and achieve noise reduction.

[0131] In one embodiment, according to the noise reduction condition characteristics, the noise signal input to the targeted noise adding step is subjected to noise reduction processing to obtain the noise reduction signal, which includes: according to the noise reduction condition characteristics and the noise signal input to the targeted noise adding step, predicting the added noise corresponding to the targeted noise adding step to obtain a first predicted added noise corresponding to the targeted noise adding step; and subtracting the first predicted added noise from the noise signal input to the targeted noise adding step to perform noise reduction processing to obtain the noise reduction signal.

[0132] Specifically, the server will encode the noise reduction condition features and the noise signal of the noise addition step input based on the attention mechanism, obtain the attention encoding vectors corresponding to the noise reduction condition features and the noise signal of the noise addition step input, and then decode the attention encoding vectors to obtain the first predicted added noise corresponding to the noise addition step. Finally, the first predicted added noise is subtracted from the noise signal of the noise addition step input to perform noise reduction processing to obtain the noise reduction signal.

[0133] Among them, the attention mechanism is a resource allocation scheme that allocates computing resources to more important tasks when computing power is limited, while solving the problem of information overload. In neural network learning, generally speaking, the more parameters a model has, the stronger its expressive power is, and the more information the model stores, but this will bring about the problem of information overload. Then, by introducing the attention mechanism, focusing on the information that is more critical to the current task among the numerous input information, reducing the attention to other information, and even filtering out irrelevant information, the problem of information overload can be solved and the efficiency and accuracy of task processing can be improved. In this embodiment, the focus is on the noise reduction condition characteristics and the information that is more critical to the prediction of the first prediction of adding noise in the noise signal of the noise adding step input, so as to improve the efficiency and accuracy of the prediction of the first prediction of adding noise.

[0134] In a specific application, the attention mechanism can be a multi-head attention mechanism, and the server can obtain the first predicted added noise corresponding to the targeted noisy step through a multi-level encoding and decoding process. In a specific application, the server can use a pre-trained denoiser to predict the added noise corresponding to the targeted noisy step. The pre-trained denoiser takes the noise reduction condition characteristics and the noise signal inputted by the targeted noisy step as input, and outputs the first predicted added noise corresponding to the targeted noisy step. The pre-trained denoiser can be configured and trained according to the actual application scenario. In a specific application, the pre-trained denoiser can be a network based on an N1-layer Transformer (transformation network) and N2 attention heads, where N1 and N2 are positive integers and can be configured according to the actual application scenario.

[0135] In a specific application, a schematic diagram of predicting the added noise corresponding to the targeted noisy step can be shown in Figure 11. The server will use MLP (multi-layer perceptron) to encode the number of steps t of the targeted noisy step to obtain the noisy step feature, and splice the action description representation c of the first semantic level and the noisy step feature (indicated by ⊕ in Figure 11) to obtain the noise reduction condition feature. The noise reduction condition feature and the noise signal input to the targeted noisy step are input into the pre-trained denoiser, so that the pre-trained denoiser predicts the added noise corresponding to the targeted noisy step (i.e., the noise added in the tth step) based on the noise reduction condition feature and the noise signal input to the targeted noisy step, and obtains the first predicted added noise corresponding to the targeted noisy step.

[0136] In this embodiment, by predicting the added noise corresponding to the targeted noise adding step based on the noise reduction condition characteristics and the noise signal input to the targeted noise adding step, a first predicted added noise corresponding to the targeted noise adding step can be obtained, and then the noise signal input to the targeted noise adding step can be directly added based on the first predicted noise to perform noise reduction processing to obtain a noise reduction signal, thereby achieving noise reduction through noise prediction.

[0137] In one embodiment, the action of a virtual object is determined by a pre-trained action sequence generation model, which includes a cascaded denoising network and a decoder; the cascaded denoising network is used to perform denoising processing on each semantic level in multiple semantic levels to obtain an action feature vector after cascade denoising of multiple semantic levels; the decoder is used to decode the action feature vector after cascade denoising to obtain the action of the virtual object.

[0138] Specifically, the pre-trained action sequence generation model refers to a model used to generate virtual object actions. The action sequence generation model includes a cascade denoising network and a decoder, wherein the cascade denoising network is used to perform denoising processing on each semantic level in multiple semantic levels to obtain an action feature vector after cascade denoising of multiple semantic levels, and the decoder is used to decode the action feature vector after cascade denoising to obtain the virtual object action.

[0139] In a specific application, taking a cascade denoising network including three denoisers as an example, the structure of a pre-trained action sequence generation model can be shown in FIG12. In the cascade denoising network, the input of the denoiser of the first semantic level is the sampled noise signal n and the action description representation C1 of the first semantic level. After the sampled noise signal n is subjected to multi-step inverse denoising processing (i.e., the iteration shown in FIG12) as a noise signal that has undergone multi-step denoising, the denoiser of the first semantic level outputs an action feature vector Z1. At each semantic level after the first semantic level, the input of the denoiser of the semantic level includes the sampled noise signal n, the action description representations from the first semantic level to the current semantic level, and the action feature vector outputted by the previous semantic level (as shown in FIG12, the action description representation inputted by the second semantic level includes the action description representation C1 of the first semantic level and the two action description representations of the current semantic level). and The action description representation of the third semantic level input includes the action description representation C1 of the first semantic level and the action description representation C2 of the second semantic level. and And the action description representation of this semantic level and The action feature vector output by the denoiser at the second semantic level is Z2), and the denoiser at this semantic level is also used to perform multi-step reverse denoising (i.e., the iteration shown in Figure 12). The action feature vector Z3 output by the denoiser at the third semantic level is the action feature vector after cascade denoising. The cascade denoised action feature vector output by the cascade denoising network is the input of decoder D3, which decodes the cascade denoised action feature vector to obtain the virtual object action.

[0140] In this embodiment, an action sequence generation model including a cascaded noise reduction network and a decoder can be used to accurately infer the actions of virtual objects, thereby improving the accuracy of the generated virtual object actions.

[0141] In one embodiment, the cascaded denoising network is obtained through a training step, which includes: obtaining multiple training samples; for each of the multiple training samples, training the initial denoising network according to the sample description text and action sequence in the training sample to obtain the cascaded denoising network.

[0142] Among them, the training sample refers to the sample used to train the cascade denoising network. Each training sample includes a sample description text and an action sequence. The sample description text in the training sample is used to describe the action sequence in the training sample, that is, the sample description text in the training sample corresponds to the action sequence. Similar to the action description text, the sample description text can also include information such as action category, motion path, and action style. The action sequence is a sequence composed of multiple actions, and the multiple actions correspond to the virtual object actions described by the sample description text. For example, the multiple actions can specifically be at least two actions in the process of the virtual object walking forward and turning right. It should be noted that the number of actions in the action sequence can be configured according to the actual application scenario. The initial denoising network refers to a denoising network that has not undergone parameter training. By training the initial denoising network, the cascade denoising network can be obtained.

[0143] Specifically, the server obtains multiple training samples and, for each of the multiple training samples, trains an initial denoising network based on the sample description text and action sequence in the training sample to obtain a cascaded denoising network. In a specific application, the initial denoising network includes multiple cascaded initial denoisers. Training the initial denoising network, i.e., training the cascaded initial denoisers, enables the multiple initial denoisers to have the ability to predict noise after training. The pre-trained cascaded denoising network can then be used to denoise the sampled noise signal to generate a cascaded denoised action feature vector.

[0144] In this embodiment, by obtaining multiple training samples, the sample description text and action sequence in each training sample can be used to train the initial denoising network, thereby obtaining the cascade denoising network, and then the cascade denoising network can be used for denoising processing to achieve accurate reasoning of virtual object actions and improve the accuracy of the generated virtual object actions.

[0145] In one embodiment, the initial denoising network is trained based on the sample description text and action sequence in the targeted training sample to obtain a cascade denoising network, including: parsing the sample description text in the targeted training sample at multiple semantic levels through semantic analysis to obtain sample description information for each of the multiple semantic levels; encoding the sample description information for each of the multiple semantic levels to obtain sample description representations for each of the multiple semantic levels; and training the initial denoising network based on the sample description representations for each of the multiple semantic levels and the action sequence in the targeted training sample to obtain a cascade denoising network.

[0146] Specifically, the semantic analysis method can be based on semantic role parsing. The server will parse the sample description text in the training sample at multiple preset semantic levels based on the semantic role parsing to obtain sample description information for each of the multiple semantic levels. The sample description information of each semantic level in the multiple semantic levels is encoded separately to obtain the fourth eigenvector of each sample description information. Based on the fourth eigenvector of each sample description information, the sample description representation of each of the multiple semantic levels is obtained. Based on the sample description representation of each of the multiple semantic levels and the action sequence in the training sample, the initial denoising network is trained to obtain a cascade denoising network. In order to facilitate training and processing, the action sequence in the training sample can be serialized data.

[0147] In a specific application, the server can use a pre-trained natural language model for text feature extraction to encode the sample description information at each of the multiple semantic levels, obtain the fourth feature vector of each sample description information, and then concatenate the fourth feature vectors of the sample description information at the same semantic level to obtain the sample description representations for each of the multiple semantic levels. The pre-trained natural language model for text feature extraction can be trained according to the actual application scenario.

[0148] In this embodiment, on the basis of obtaining sample description representations of multiple semantic levels by performing semantic hierarchical analysis and encoding, the initial denoising network can be trained using the sample description representations and the action sequences in the targeted training samples to achieve the acquisition of the cascade denoising network, so that the cascade denoising network can be used for denoising processing to achieve accurate reasoning of virtual object actions and improve the accuracy of the generated virtual object actions.

[0149] In one embodiment, the initial denoising network is trained based on the sample description representations of each of the multiple semantic levels and the action sequences in the targeted training samples to obtain a cascaded denoising network, including: performing action encoding of multiple encoding levels on the action sequences in the targeted training samples to obtain implicit action representations corresponding to each of the multiple semantic levels; and the initial denoising network is trained based on the sample description representations of each of the multiple semantic levels and the implicit action representations corresponding to each of the multiple semantic levels to obtain a cascaded denoising network.

[0150] Each of the multiple coding levels is used to encode the action sequence in the training sample. Different coding levels use different encoding dimensions when encoding the action sequence. This approach enables the generation of multi-dimensional implicit action representations. Implicit action representations can be understood as implicit action encoding vectors that represent the action sequence.

[0151] Specifically, at each of the multiple coding levels, the server learns action representations by encoding and decoding the action sequences in the targeted training samples to obtain implicit action representations for that coding level. When implicit action representations for multiple coding levels are obtained, the implicit action representations for the multiple coding levels are used as implicit action representations corresponding to the multiple semantic levels. After obtaining the implicit action representations corresponding to the multiple semantic levels, the server uses the sample description representations for the multiple semantic levels and the implicit action representations corresponding to the multiple semantic levels to train the initial denoising network, resulting in a cascaded denoising network.

[0152] In this embodiment, by performing action encoding on the action sequence at multiple encoding levels, the corresponding implicit action representations of multiple semantic levels can be obtained. Then, the corresponding implicit action representations and sample description representations of multiple semantic levels can be used to train the initial denoising network from the perspective of multiple semantic levels, and a cascade denoising network that can achieve fine-grained denoising can be obtained.

[0153] In one embodiment, multiple coding levels correspond one-to-one to multiple semantic levels; the coding dimensions of each coding level in the multiple coding levels are in order from the first coding level to the last coding level; the action sequences in the targeted training samples are respectively subjected to action coding at multiple coding levels to obtain implicit action representations corresponding to the multiple semantic levels, including: performing action coding at multiple coding levels on the action sequences in the targeted training samples to obtain motion latent space features of the multiple coding levels; decoding the motion latent space features of the multiple coding levels to obtain implicit action representations corresponding to the multiple semantic levels.

[0154] Among them, the motion latent space feature refers to the feature obtained by mapping the action sequence in the training sample to the latent space. The latent space is a representation of compressed data. Its function is to learn data features and simplify data representation in order to find patterns. By mapping data to the latent space, the dimension of the data can be reduced. The encoding dimension refers to the number of dimensions used by the encoding level to encode the action sequence. The encoding dimension of each encoding level in the multiple encoding levels increases in order from the first encoding level to the last encoding level. That is, the encoding dimension of each encoding level in the multiple encoding levels increases from the first encoding level to the last encoding level. In other words, the encoding dimension of the first encoding level is the smallest, and the encoding dimension of the last encoding level is the largest.

[0155] Specifically, multiple coding levels correspond one-to-one to multiple semantic levels, and the coding dimensions of each coding level in the multiple coding levels are in order from the first coding level to the last coding level, that is, the feature dimensions of the obtained motion latent space features also increase from coding level to coding level. For each coding level in the multiple coding levels, the server will perform motion encoding on the action sequence in the training sample with the coding dimension of the targeted coding level to obtain the motion latent space features of the targeted coding level, and then decode the motion latent space features of the targeted coding level to obtain the implicit action representation corresponding to the targeted coding level. The implicit action representation corresponding to the targeted coding level is used as the implicit action representation corresponding to the semantic level corresponding to the targeted coding level.

[0156] In specific applications, the action sequence can be serialized data, and the motion latent space feature can be the action feature data distribution corresponding to the action sequence. By sampling the action feature data distribution, the action feature sampling points corresponding to the action sequence can be obtained, and then the action feature sampling points can be decoded to obtain the implicit action representation corresponding to the targeted coding level.

[0157] In a specific application, the obtained motion feature data distribution includes a mean and variance. On this basis, the server can randomly sample sample points from a standard normal distribution and then use the reparameterization technique to obtain the motion feature sampling points corresponding to the motion sequence based on the mean and variance and the randomly sampled sample points. The principle of the reparameterization technique is that if z is a random variable that follows a Gaussian distribution with a mean g(x) and a covariance h(x), then z can be expressed as z = h(x)ζ + g(x), where ζ ~ N(0, I) is a standard normal distribution. Therefore, after obtaining the mean and variance and randomly sampling a sampling point, the motion feature sampling point z corresponding to the motion sequence can be directly obtained.

[0158] In a specific application, a pre-trained variational autoencoder can be used to implement the steps of first encoding, then sampling, and finally decoding in this embodiment. For each coding level in multiple coding levels, the action sequence in the targeted training sample can be input into the pre-trained variational autoencoder to obtain the implicit action representation corresponding to the targeted coding level.

[0159] In a specific application, a variational autoencoder can be defined as an autoencoder whose training is regularized to avoid overfitting and ensure that the latent space has good properties for data generation. Like a standard autoencoder, a variational autoencoder is a structure consisting of an encoder and a decoder, trained to minimize the reconstruction error between the encoded and decoded data and the original data. However, to introduce some regularization of the latent space, the encoder-decoder process is modified in a variational autoencoder: instead of encoding the input as a single point in the latent space, it is encoded as a probability distribution over the latent space. The variational autoencoder is trained as follows: first, the input is encoded as a distribution over the latent space; second, a point in the latent space is sampled from this distribution; third, the sampled point is decoded and the reconstruction error is calculated; finally, the reconstruction error is backpropagated through the network.

[0160] In this embodiment, by performing action encoding on the action sequence at multiple encoding levels respectively, implicit action representations corresponding to multiple semantic levels can be obtained, thereby realizing implicit representation of the action sequence. Then, the implicit action representations and sample description representations corresponding to multiple semantic levels can be used to train the initial denoising network from the perspective of multiple semantic levels, and a cascade denoising network that can realize fine-grained denoising can be obtained.

[0161] In one embodiment, the initial denoising network includes a plurality of cascaded initial denoisers; and each initial denoiser corresponds to a semantic level respectively; the initial denoising network is trained based on the sample description representations of the plurality of semantic levels and the implicit action representations corresponding to the plurality of semantic levels, and the cascaded denoising network includes: for each of the plurality of initial denoisers, the targeted initial denoiser is trained based on the sample description representations of at least two semantic levels from the first semantic level to the target semantic level corresponding to the targeted initial denoiser, and the implicit action representation corresponding to the target semantic level, to obtain a trained denoiser; the cascaded denoising network is obtained according to the trained denoisers corresponding to the plurality of initial denoisers.

[0162] Specifically, the initial denoising network includes a plurality of cascaded initial denoisers, and each initial denoiser corresponds to a semantic level. When training the initial denoising network, for each of the multiple initial denoisers, the server will train the targeted initial denoiser based on the sample description representations of at least two semantic levels from the first semantic level to the target semantic level corresponding to the targeted initial denoiser, and the implicit action representation corresponding to the target semantic level, to obtain a trained denoiser. A cascaded denoising network is obtained based on the trained denoisers corresponding to each of the multiple initial denoisers.

[0163] In a specific application, when training the targeted initial denoiser, the server will first perform noise processing on the implicit action representation corresponding to the target semantic level corresponding to the targeted initial denoiser, and then use the sample description representations of at least two semantic levels from the first semantic level to the target semantic level corresponding to the targeted initial denoiser as conditions to predict the noise added during the noise addition process using the targeted initial denoiser, so as to adjust the parameters of the targeted initial denoiser by comparing the noise actually added during the noise addition process with the noise added during the noise addition process predicted by the targeted initial denoiser, so that the targeted initial denoiser can achieve accurate noise prediction, so that the targeted initial denoiser can be used to achieve accurate noise prediction in the inference stage, and thus the predicted noise can be used for noise reduction.

[0164] In this embodiment, for each of the multiple initial denoisers, the initial denoiser is trained based on the sample description representations of at least two semantic levels from the first semantic level to the target semantic level corresponding to the initial denoiser, and the implicit action representation corresponding to the target semantic level, so as to obtain a trained denoiser, and then a cascade denoising network can be obtained based on the trained denoisers corresponding to each of the multiple initial denoisers.

[0165] In one embodiment, for each of a plurality of initial denoisers, the initial denoiser is trained based on the sample description representations of at least two semantic levels from the first semantic level to the target semantic level corresponding to the initial denoiser, and the implicit action representation corresponding to the target semantic level, to obtain a trained denoiser, including: obtaining the number of noise addition steps for adding noise and sampling a random noise signal; adding the random noise signal to the implicit action representation corresponding to the target semantic level according to the number of noise addition steps to obtain a noise action representation; inputting the noise action representation, the number of noise addition steps and the sample description representations of at least two semantic levels from the first semantic level to the target semantic level corresponding to the initial denoiser into the initial denoiser, predicting the added noise by the initial denoiser to obtain a second predicted added noise; adjusting the parameters of the initial denoiser according to the second predicted added noise to obtain a trained denoiser.

[0166] Specifically, for each of the multiple initial denoisers, when training the targeted initial denoiser, the server will first obtain the number of noise adding steps for adding noise, and sample the random noise signal, and according to the number of noise adding steps, gradually add the random noise signal to the implicit action representation corresponding to the target semantic level to obtain the noise action representation, and input the noise action representation, the number of noise adding steps, and the sample description representations of at least two semantic levels from the first semantic level to the target semantic level corresponding to the targeted initial denoiser into the targeted initial denoiser, so as to predict the added noise through the targeted initial denoiser to obtain a second predicted added noise, and finally adjust the parameters of the targeted initial denoiser according to the second predicted added noise to obtain a trained denoiser.

[0167] In specific applications, the number of noise adding steps used to add noise can be configured according to the actual application scenario, and this embodiment does not limit this here. It should be noted that the larger the number of noise adding steps, the closer the noise action representation obtained is to the Gaussian distribution. Therefore, the noise action representation obtained after adding the random noise signal can be regarded as Gaussian noise. In this embodiment, it is equivalent to gradually applying the sampled random noise signal according to the number of noise adding steps to the implicit action representation corresponding to the target semantic level corresponding to the initial noise reducer, so that the implicit action representation is destroyed and turned into complete Gaussian noise, and then in the reverse stage, the initial noise reducer is used to learn to restore the Gaussian noise to the implicit action representation corresponding to the target semantic level corresponding to the initial noise reducer.

[0168] In a specific application, the training of the initial denoiser in this embodiment to obtain the trained denoiser is based on a diffusion model. Diffusion models are a type of generative model that learns noise prediction through a Markov noise-adding process, ultimately transforming a Gaussian noise distribution into a target data distribution. Unlike other generative networks, diffusion models gradually apply noise to samples in the forward phase until the samples are corrupted and transformed into pure Gaussian noise. In the reverse phase, they then learn to restore the Gaussian noise to the original sample.

[0169] In this embodiment, the sample refers to the implicit action representation corresponding to the target semantic level corresponding to the initial denoiser, the gradual application of noise refers to the gradual application of the sampled random noise signal according to the number of noise addition steps, Gaussian noise refers to the noise action representation, and the reverse stage learning refers to the input of the noise action representation, the number of noise addition steps, and the sample description representation of at least two semantic levels from the first semantic level to the target semantic level corresponding to the initial denoiser into the initial denoiser, and the added noise is predicted by the initial denoiser to obtain a second predicted added noise.

[0170] In a specific application, the server compares the second prediction with the added noise and random noise signals to obtain a predicted noise error. If the predicted noise error exceeds an error threshold, the server adjusts the parameters of the initial denoiser based on the predicted noise error. The server then continues training the initial denoiser with adjusted parameters until the calculated predicted noise error is less than or equal to the error threshold, resulting in a trained denoiser. The error threshold can be configured based on the actual application scenario.

[0171] In this embodiment, by obtaining the number of noise adding steps for adding noise and sampling the random noise signal, the random noise signal can be added to the implicit action representation corresponding to the target semantic level using the number of noise adding steps to realize the noise adding process and obtain the noise action representation. Then, the noise action representation, the number of noise adding steps and the sample description representations of at least two semantic levels from the first semantic level to the target semantic level corresponding to the initial denoiser can be input into the initial denoiser. The added noise is predicted by the initial denoiser to learn the noise prediction and obtain the second predicted added noise. Thus, the parameters of the initial denoiser can be adjusted according to the second predicted added noise to obtain the trained denoiser and realize the training of the initial denoiser.

[0172] In one embodiment, the noise action representation, the number of noise addition steps, and the sample description representations of at least two semantic levels from the first semantic level to the target semantic level corresponding to the initial denoiser are input into the initial denoiser, and the added noise is predicted by the initial denoiser to obtain the second predicted added noise, which includes: when the initial denoiser has a previous level denoiser connected in series, the noise action representation, the number of noise addition steps, the sample description representations of at least two semantic levels from the first semantic level to the target semantic level corresponding to the initial denoiser, and the reconstructed action representation output by the previous level denoiser are input into the initial denoiser, and the added noise is predicted by the initial denoiser to obtain the second predicted added noise.

[0173] Specifically, in the process of training the targeted initial denoiser, when the targeted initial denoiser has a previous-level denoiser connected in series, the server will input the noise action representation, the number of noise addition steps, the sample description representations of at least two semantic levels from the first semantic level to the target semantic level corresponding to the targeted initial denoiser, and the reconstructed action representation output by the previous-level denoiser into the targeted initial denoiser, and predict the added noise through the targeted initial denoiser to obtain a second predicted added noise. In a specific application, the reconstructed action representation output by the previous-level denoiser refers to the representation corresponding to the implicit action representation before noise addition, which is reconstructed by the previous-level denoiser after predicting the added noise based on the data input to it and performing noise reduction processing on the input noise action representation based on the predicted noise, that is, the action representation restored from the noise action representation by learning the noise prediction.

[0174] In this embodiment, when the targeted initial denoiser has an upper-level denoiser connected in series, the noise action representation, the number of noise addition steps, the sample description representations of at least two semantic levels from the first semantic level to the target semantic level corresponding to the targeted initial denoiser, and the reconstructed action representation output by the upper-level denoiser are input into the targeted initial denoiser, and the added noise is predicted by the targeted initial denoiser. The noise prediction can be learned in combination with the reconstructed action representation output by the upper-level denoiser, which can improve the accuracy of the noise prediction and obtain a second predicted added noise.

[0175] The present application proposes a method for generating actions of virtual objects (specifically, virtual characters) driven by refined controllable text based on hierarchical semantics. The method receives a text describing the action of a virtual object as input, and synthesizes the corresponding virtual object action according to the action category, motion path, action style and other information specified in the action description text. The virtual object action can specifically be a 3D virtual object skeleton or mesh sequence. Compared with traditional methods, the inventor believes that the solution proposed in the present application parses the input text into a new control signal, namely, the action description information of each of the multiple semantic levels, and uses the action description information of each of the multiple semantic levels as a fine-grained control signal. By capturing the action features of each of the multiple semantic levels to refine and generate the virtual object action, the accuracy of the generated virtual object action is improved.

[0176] Specifically, the multiple semantic levels in this application include the overall motion level, the local action level, and the action detail level. Correspondingly, the text-to-motion generation process is also decomposed into three semantic levels, corresponding to capturing overall motion, local action, and action details. Compared with traditional methods, the method of this application has better controllability and can synthesize high-quality virtual object motion, which can specifically be an action sequence.

[0177] The inventors believe that the current text-driven human motion generation methods can be summarized into two major categories, one is the joint encoding-based method and the other is the diffusion model-based method. The joint encoding-based method usually learns a motion variational autoencoder and a text variational autoencoder. Then, this type of method uses KL divergence to constrain the text and motion encoders to a shared implicit space. The diffusion model-based method uses the conditional diffusion model for human motion generation to learn a robust probabilistic mapping from text descriptors to human motion sequences. Both of the above methods rely on the global representation of text and directly learn the mapping from the global text representation with a high-level language to the motion sequence.

[0178] However, traditional methods directly use neural networks to automatically and implicitly extract text features, which may overemphasize certain details in the text and ignore other important information. This makes the network insensitive to subtle changes in the input text and lacks fine-grained controllability. In addition, traditional methods cannot generate action details well. On the one hand, the text description of an action often involves multiple actions and attributes. However, the global text representations extracted by current methods usually cannot convey the clarity and details required to fully understand the text, resulting in an inability to effectively guide the synthesis of motion details. On the other hand, the direct mapping of existing methods from global text representations with high-level languages ​​to motion sequences further hinders the generation of action details.

[0179] Based on this, this application proposes a method for generating virtual object actions driven by refined controllable text based on hierarchical semantics. By utilizing the hierarchical structure of the action description text, the action description text is parsed at multiple preset semantic levels through semantic analysis to obtain the action description information of each of the multiple semantic levels. The action description information of each of the multiple semantic levels is used as a fine-grained signal to generate controllable motion. Specifically, the action description text describes an overall motion that includes multiple actions, and the overall motion is composed of several local actions, and each local action is composed of different action details that serve as its attributes, such as the direction and speed of movement of the action. This global-to-local structure helps to reliably and comprehensively understand the action description to achieve fine-grained control of the virtual object action.

[0180] In one embodiment, the virtual object action generation method of the present application is explained by taking the fine-grained control of virtual object action generation based on a hierarchical semantic graph constructed through action description information of multiple semantic levels, and the multiple semantic levels include the overall motion level, the local action level and the action detail level as an example.

[0181] Specifically, the overall framework of the virtual object action generation method of the present application is shown in Figure 13, which mainly includes two core components: a graph reasoning module and a coarse-to-fine action sequence generation module. For the action description text used to describe the virtual object action, the server will extract at least one verb and the phrase used to modify the verb corresponding to each of the at least one verb appearing in the action description text based on the semantic role parsing tool, and determine the semantic role of each phrase used to modify the verb to obtain action description information for each of the multiple semantic levels. After obtaining the action description information for each of the multiple semantic levels, the server will treat the action description text as the overall motion node of the overall motion level in the hierarchical semantic graph, and treat at least one verb as a local action node of the local action level in the hierarchical semantic graph, and connect it to the overall motion node with a direct edge. At the same time, the server will also treat the phrase used to modify the verb corresponding to each of the at least one verb as an action detail node of the action detail level connected to the corresponding local action node. Subsequently, the server will use a pre-trained text encoder to encode the action description text, at least one verb, and the phrase used to modify the verb corresponding to each of the at least one verb into node representations of the corresponding semantic nodes.

[0182] In the graph reasoning module, the server uses a pre-trained graph attention network to construct interactions at different levels in the hierarchical semantic graph, with the goal of reducing ambiguity at each semantic node. For example, the verb "pick up" can represent different actions without context, but the phrase "using both hands" used to modify the verb eliminates possible ambiguity of this verb, so the action should be "pick up with both hands" rather than "pick up with one hand." Therefore, using the pre-trained graph attention network to reason about interactions in the hierarchical semantic graph can obtain three levels of textual representations: action description representations at multiple semantic levels, which are responsible for capturing the control information of the overall movement, the control information of the local action, and the control information of the action details.

[0183] In specific applications, the graph attention mechanism can be used through the graph attention network to update the node representation of each semantic node in the hierarchical semantic graph. After obtaining the updated node representation of each semantic node, the server will use the updated node representation of each semantic node as the second feature vector of each action description information. For each semantic level, the second feature vectors of at least two action description information belonging to the semantic level are spliced, and the spliced ​​second feature vector is used as the action description representation of the semantic level to obtain the action description representations of multiple semantic levels.

[0184] In the coarse-to-fine action sequence generation module, we decompose the text-to-action generation process from coarse to fine into three semantic levels, which are responsible for capturing the overall motion, local action, and action details respectively.

[0185] First, during the training phase, the server constructs a three-level motion encoder. Specifically, the server trains an action autoencoder at each of the three semantic levels, learning the action representation through an encoding-decoding process to obtain an implicit action representation z at each semantic level. Taking the learning of action representations for overall motion as an example, the action autoencoder consists of an encoder E1 and a decoder D1. It learns an effective motion representation z1 by minimizing the reconstruction error of D1 (E1(S1)), where S1 refers to the motion sequence used during training. After performing end-to-end optimization on all motion autoencoder components (i.e., E1, ..., D3), the server freezes all parameters. This allows us to obtain implicit action representations z1, z2, and z3 at three different semantic levels for a given action sequence (specifically, 3D human motion) in an input training sample.

[0186] Subsequently, a hierarchical motion generation module is designed during the training phase, which generates action sequences based on the diffusion model. Compared to other generative frameworks, the diffusion model is a generative model based on a thermodynamic random diffusion process. This process consists of a forward process of gradually adding noise to the samples from the data distribution, and a backward process of training the neural network to reverse the forward process by gradually removing the noise. In the forward process, the noise addition process in the implicit space is defined as in represents the implicit action representation of the i-th semantic level in the t-th noise step, represents the implicit action representation of the i-th semantic level in the t-1-th noise step, α t is a preconfigured hyperparameter related to the noise addition step t, which can be obtained based on the noise addition step t.

[0187] In this embodiment, denoisers R1, R2 and R3 at three semantic levels are connected in series during the training phase. After the training is completed, the trained denoisers R1, R2 and R3 at the three semantic levels can be used to obtain the finest-grained action implicit encoding from coarse to fine from the sampling noise signal used to generate the action of the virtual object and the action description text used to describe the action of the virtual object, that is, the action feature vector after cascade denoising of multiple semantic levels.

[0188] In specific applications, at the application stage, at the overall motion level, we only use the features of the overall motion nodes (i.e., the action description representation C1 at the overall motion level) as the conditional encoding of the diffusion model to generate the coarse-grained action feature vector Z 11 At the local action level, we use the features of the overall motion node (i.e., the action description representation C1 at the overall motion level), the features of the local action node (i.e., the action description representation C1 at the local action level), and the features of the local action node (i.e., the action description representation C1 at the local action level). and ) and Z 11 The conditional encoding together as a diffusion model further generates the implicit action encoding Z 22 At the action detail level, we use the features of all nodes in the hierarchical semantic graph (as shown in Figure 14, the action description of the overall motion level is represented as C1, and the action description of the local action level is represented as and The action description at the action detail level is represented as and ) and Z 22 The conditional encoding together as a diffusion model generates a fine-grained implicit action code Z 33 Finally, decoder D3 converts Z 33The implicit feature space is converted back to the original three-dimensional virtual object posture space, thereby generating a corresponding virtual object motion sequence from a given text description (i.e., action description text), and the virtual object action can specifically be a 3D human action sequence.

[0189] In one embodiment, Table 1 and Table 2 respectively give the quantitative experimental results of the present application on the HumanML3D and KIT-ML datasets, among which the best results are all the methods of the present application. The methods compared with the method of the present application in Table 1 and Table 2 include: Real motion (real motion), Seq2Seq (sequence to sequence), Language2Pose (joint language pose), Text2Gesture (text-gesture), Hier (multi-layer attention model), MoCoGAN (model for video generation), Dance2Music (dance-music model), TM2T (a model for generating human motion), T2M (text generation animation), MDM (human motion diffusion model), MLD (motion latent diffusion), etc.

[0190] Currently, five evaluation indicators are widely used in cross-modal generation tasks: R-Precision (reflecting the text-motion matching accuracy in retrieval), FID (Frechet Inception Distance, a measure used to calculate the distance between the feature vectors of the real image and the generated image), MM Dist (Multi-Modal Distance), Diversity (diversity, defined as the variance of the motion feature vectors of the generated motion in all text descriptions, reflecting the diversity of motion synthesized from a set of different descriptions) and MModality (Multi-modality, multimodal attitude measures the diversity of motion generated in each text description, reflecting the diversity of motion synthesized from a specific description).

[0191] Of these five quantitative metrics, R-Precision, FID, and MM Dist primarily reflect the fidelity of the generated 3D human motions compared to real motions; Diversity and MModality primarily reflect the diversity of the generated 3D human motions. The results in Tables 1 and 2 demonstrate that our method surpasses existing methods in both fidelity and diversity of generated results on two major datasets, achieving state-of-the-art performance.

[0192] Table 1 Quantitative comparison of different methods on the HumanML3D dataset

[0193] Table 2 Quantitative comparison of different methods on the KIT-ML dataset

[0194] The inventors believe that the solution of this application has two significant advantages over traditional methods. First, the explicit decomposition and representation of the semantic space enables the solution of this application to establish a fine-grained correspondence between text data and motion sequences, thereby avoiding unbalanced learning of different text components and coarse-grained control signal representation. Second, the hierarchical refinement of motion sequence generation gradually enhances the generated results from coarse to fine, avoiding excessively coarse granularity of the generated results. This ensures the quality of model generation while also improving the diversity of the results.

[0195] In one embodiment, in order to further fine-tune the generated virtual object actions to achieve finer-grained control, the solution of the present application can also continuously improve the generated virtual object actions by modifying the edge weights of the hierarchical semantic graph to generate virtual object actions that better meet the needs.

[0196] Specifically, when the virtual object action is obtained, the server will respond to the edge weight adjustment event of the connecting edge connecting each semantic node in the hierarchical semantic graph, adjust the edge weight of the connecting edge indicated by the edge weight adjustment event, and obtain an updated hierarchical semantic graph. Utilize the graph attention mechanism to update the node representation of each semantic node in the updated hierarchical semantic graph, and obtain the third feature vector of each action description information based on the updated node representation of each semantic node. For each semantic level, the third feature vectors of at least two action description information belonging to the semantic level are spliced, and the spliced ​​third feature vector is used as the updated action description representation of the semantic level to obtain the updated action description representation of each of the multiple semantic levels, and based on the updated action description representation of each of the multiple semantic levels, the adjusted virtual object action is generated.

[0197] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0198] Based on the same inventive concept, embodiments of the present application also provide a virtual object action generation device for implementing the aforementioned virtual object action generation method. The implementation solution provided by this device is similar to the implementation solution described in the aforementioned method. Therefore, the specific limitations of one or more virtual object action generation device embodiments provided below can be found in the above-mentioned limitations of the virtual object action generation method and will not be repeated here.

[0199] In one embodiment, as shown in FIG14 , a virtual object action generation device is provided, comprising: an acquisition module 1402 , a semantic parsing module 1404 , an encoding module 1406 , a first noise reduction processing module 1408 , a second noise reduction processing module 1410 , and a decoding module 1412 , wherein:

[0200] An acquisition module 1402 is used to acquire an action description text for describing an action of a virtual object;

[0201] Semantic parsing module 1404 is used to parse the action description text at multiple preset semantic levels through semantic analysis to obtain action description information for each of the multiple semantic levels and obtain a sampled noise signal for generating virtual object actions;

[0202] An encoding module 1406 is configured to encode the action description information of each of the multiple semantic levels to obtain the action description representations of each of the multiple semantic levels;

[0203] A first noise reduction processing module 1408 is configured to perform noise reduction processing at the first semantic level on the sampled noise signal based on the action description representation at the first semantic level among the multiple semantic levels, to obtain an action feature vector outputted at the first semantic level;

[0204] The second noise reduction processing module 1410 is configured to perform noise reduction processing on the sampled noise signal at each semantic level after the first semantic level in the multiple semantic levels based on the action feature vector output by the previous semantic level and the action description representations of at least two semantic levels from the first semantic level to the current semantic level, thereby obtaining an action feature vector after cascade noise reduction of the multiple semantic levels; wherein the action granularity represented by the action feature vectors output by the noise reduction processing of the multiple semantic levels decreases in order from the highest semantic level to the lowest semantic level;

[0205] The decoding module 1412 is used to decode the action feature vector after cascade noise reduction to obtain the virtual object action.

[0206] The above-mentioned virtual object action generation device obtains an action description text for describing the virtual object action, parses the action description text at multiple preset semantic levels through semantic analysis, obtains action description information for each of the multiple semantic levels, and obtains a sampled noise signal for generating the virtual object action. The action description information for each of the multiple semantic levels is encoded respectively to obtain action description representations for each of the multiple semantic levels. Based on the action description representation of the first semantic level among the multiple semantic levels, the sampled noise signal is subjected to noise reduction processing at the first semantic level to obtain an action feature vector output by the first semantic level. At each semantic level after the first semantic level among the multiple semantic levels, the sampled noise signal is subjected to noise reduction processing using the action feature vector output by the previous semantic level and the action description representations of at least two semantic levels from the first semantic level to the current semantic level as joint conditions. The action description representations of each of the multiple semantic levels can be used to gradually enrich fine-grained motion details, thereby obtaining a more fine-grained action feature vector after cascade noise reduction at multiple semantic levels that accurately represents the virtual object action. The virtual object action can then be obtained by decoding the cascaded noise reduction action feature vector. The entire process can use the action description information of multiple semantic levels as fine-grained control signals, and refine the generated virtual object actions by capturing the action features of multiple semantic levels, thereby improving the accuracy of the generated virtual object actions.

[0207] In one embodiment, the multiple semantic levels include an overall motion level, a local action level, and an action detail level; the semantic parsing module is also used to use the action description text as action description information at the overall motion level, and extract at least one verb and a phrase corresponding to at least one verb for modifying the verb from the action description text, use at least one verb as action description information at the local action level, and use the phrase corresponding to at least one verb for modifying the verb as action description information at the action detail level.

[0208] In one embodiment, the encoding module is also used to encode each action description information of each semantic level in multiple semantic levels respectively to obtain a first feature vector of each action description information; based on the semantic association relationship between the action description information of at least one pair of different semantic levels, the first feature vector of each action description information is updated based on the attention mechanism to obtain a second feature vector of each action description information; for each semantic level, the second feature vectors of at least two action description information belonging to the semantic level are spliced, and the spliced ​​second feature vectors are used as the action description representation of the semantic level to obtain the action description representations of multiple semantic levels.

[0209] In one embodiment, the encoding module is also used to treat each action description information as a semantic node, and based on the semantic association relationship between at least one pair of action description information between different semantic levels, connect the two semantic nodes representing a pair of action description information with a semantic association relationship to determine the connecting edge connecting each semantic node, and use the first eigenvector of each action description information as the node representation of each semantic node, respectively. According to each semantic node, the connecting edge connecting each semantic node and the node representation of each semantic node, a hierarchical semantic graph is constructed, and the node representation of each semantic node in the hierarchical semantic graph is updated by using the graph attention mechanism. According to the updated node representation of each semantic node, the second eigenvector of each action description information is obtained.

[0210] In one embodiment, the encoding module is also used to determine, for each semantic node in the hierarchical semantic graph, at least one adjacent node of the targeted semantic node, perform interactive processing on the node representation of at least one adjacent node and the node representation of the targeted semantic node based on the graph attention mechanism, determine the attention weight coefficient of at least one adjacent node and the targeted semantic node, and perform weighted summation on the node representation of at least one adjacent node and the node representation of the targeted semantic node according to the attention weight coefficient to obtain an updated node representation of the targeted semantic node.

[0211] In one embodiment, the virtual object action generation device also includes an adjustment module, which is used to, when a virtual object action is obtained, respond to an edge weight adjustment event of the connecting edge connecting each semantic node in the hierarchical semantic graph, adjust the edge weight of the connecting edge indicated by the edge weight adjustment event to obtain an updated hierarchical semantic graph, and use the graph attention mechanism to update the node representation of each semantic node in the updated hierarchical semantic graph. According to the updated node representation of each semantic node, the third eigenvector of each action description information is obtained. For each semantic level, the third eigenvectors of at least two action description information belonging to the semantic level are spliced, and the spliced ​​third eigenvector is used as the updated action description representation of the semantic level to obtain the updated action description representation of each of the multiple semantic levels, and the adjusted virtual object action is generated based on the updated action description representation of each of the multiple semantic levels.

[0212] In one embodiment, the first noise reduction processing module is also used to treat the sampled noise signal as a noise signal that has undergone multi-step noise addition, starting from the last step of the multi-step noise addition, based on the action description representation of the first semantic level among multiple semantic levels, perform reverse noise reduction processing on the noise signal input at each step, and use the noise reduction signal obtained by performing noise reduction processing on the noise signal input at the first step as the action feature vector outputted at the first semantic level.

[0213] In one embodiment, the first noise reduction processing module is used to encode the targeted noise step to obtain the noise step feature, fuse the action description representation of the first semantic level and the noise step feature to obtain the noise reduction condition feature, and perform noise reduction processing on the noise signal inputted by the targeted noise step according to the noise reduction condition feature to obtain the noise reduction signal.

[0214] In one embodiment, the first noise reduction processing module is used to predict the added noise corresponding to the noise addition step according to the noise reduction condition characteristics and the noise signal input to the noise addition step, obtain the first predicted added noise corresponding to the noise addition step, subtract the first predicted added noise from the noise signal input to the noise addition step to perform noise reduction processing and obtain the noise reduction signal.

[0215] In one embodiment, the action of a virtual object is determined by a pre-trained action sequence generation model, which includes a cascaded denoising network and a decoder; the cascaded denoising network is used to perform denoising processing on each semantic level in multiple semantic levels to obtain an action feature vector after cascade denoising of multiple semantic levels; the decoder is used to decode the action feature vector after cascade denoising to obtain the action of the virtual object.

[0216] In one embodiment, the virtual object action generation device also includes a training module, which is used to obtain multiple training samples. For each of the multiple training samples, the initial denoising network is trained according to the sample description text and action sequence in the training sample to obtain a cascade denoising network.

[0217] In one embodiment, the training module is also used to parse the sample description text in the targeted training sample at multiple semantic levels through semantic analysis to obtain sample description information of each of the multiple semantic levels, encode the sample description information of each of the multiple semantic levels respectively to obtain sample description representations of each of the multiple semantic levels, and train the initial denoising network based on the sample description representations of each of the multiple semantic levels and the action sequence in the targeted training sample to obtain a cascade denoising network.

[0218] In one embodiment, the training module is also used to perform action encoding at multiple encoding levels on the action sequences in the targeted training samples, obtain implicit action representations corresponding to each of the multiple semantic levels, and train the initial denoising network based on the sample description representations of each of the multiple semantic levels and the implicit action representations corresponding to each of the multiple semantic levels to obtain a cascaded denoising network.

[0219] In one embodiment, multiple coding levels correspond one-to-one to multiple semantic levels; the coding dimensions of each coding level in the multiple coding levels are in order from the first coding level to the last coding level; the training module is also used to perform motion encoding of the action sequences in the training samples at multiple coding levels respectively, obtain the motion latent space features of the multiple coding levels respectively, and decode the motion latent space features of the multiple coding levels respectively to obtain implicit action representations corresponding to the multiple semantic levels respectively.

[0220] In one embodiment, the initial denoising network includes a plurality of cascaded initial denoisers; and each initial denoiser corresponds to a semantic level respectively; the training module is also used to train each of the multiple initial denoisers based on the sample description representations of at least two semantic levels from the first semantic level to the target semantic level corresponding to the targeted initial denoiser, and the implicit action representation corresponding to the target semantic level, to obtain a trained denoiser, and obtain a cascaded denoising network based on the trained denoisers corresponding to each of the multiple initial denoisers.

[0221] In one embodiment, the training module is also used to obtain the number of noise adding steps for adding noise, and to sample a random noise signal. According to the number of noise adding steps, the random noise signal is added to the implicit action representation corresponding to the target semantic level to obtain a noise action representation. The noise action representation, the number of noise adding steps, and the sample description representations of at least two semantic levels from the first semantic level to the target semantic level corresponding to the initial denoiser are input into the initial denoiser. The added noise is predicted by the initial denoiser to obtain a second predicted added noise. The parameters of the initial denoiser are adjusted according to the second predicted added noise to obtain a trained denoiser.

[0222] In one embodiment, the training module is also used to input the noise action representation, the number of noise addition steps, the sample description representations of at least two semantic levels from the first semantic level to the target semantic level corresponding to the initial denoiser, and the reconstructed action representation output by the previous denoiser into the initial denoiser when the initial denoiser is connected in series with an upper denoiser, and predict the added noise through the initial denoiser to obtain a second predicted added noise.

[0223] Each module in the aforementioned virtual object motion generation device may be implemented in whole or in part through software, hardware, or a combination thereof. Each module may be embedded in or independent of a processor within a computer device in the form of hardware, or may be stored in a computer device memory in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0224] In one embodiment, a computer device is provided. The computer device can be a server or a terminal. Taking the server as an example, its internal structure diagram can be shown in Figure 15. The computer device includes a processor, a memory, an input / output interface (I / O), and a communication interface. The processor, memory, and I / O interface are connected via a system bus, and the communication interface is connected to the system bus via the I / O interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store data such as training samples. The I / O interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a method for generating virtual object actions.

[0225] Those skilled in the art will understand that the structure shown in FIG15 is merely a block diagram of a portion of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0226] In one embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0227] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0228] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0229] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0230] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0231] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A method for generating a virtual object action, characterized in that: Executed by a computer device, the method includes: Obtaining an action description text for describing an action of a virtual object; Parsing the action description text at a plurality of preset semantic levels through semantic analysis to obtain action description information of each of the plurality of semantic levels, and acquiring a sampled noise signal for generating the action of the virtual object; Encoding the action description information of each of the multiple semantic levels respectively to obtain the action description representations of each of the multiple semantic levels; Based on the action description representation of a first semantic level among the multiple semantic levels, performing noise reduction processing of the first semantic level on the sampled noise signal to obtain an action feature vector output by the first semantic level; At each semantic level after the first semantic level in the multiple semantic levels, based on the action feature vector output by the previous semantic level and the action description representations of at least two semantic levels from the first semantic level to the current semantic level, the sampled noise signal is subjected to denoising to obtain the action feature vector after cascade denoising of the multiple semantic levels; wherein the action granularity level represented by the action feature vector output by the denoising process of the multiple semantic levels decreases in order from the highest semantic level to the lowest semantic level; and The action feature vector after the cascade noise reduction is decoded to obtain the virtual object action.

2. The method according to claim 1, characterized in that The multiple semantic levels include an overall motion level, a local action level, and an action detail level; the action description text is parsed at multiple preset semantic levels through semantic analysis to obtain the action description information of each of the multiple semantic levels, including: Using the action description text as the action description information of the overall motion level, and extracting at least one verb and a phrase corresponding to the at least one verb and used to modify the verb from the action description text; and The at least one verb is used as the action description information at the local action level, and the phrases corresponding to the at least one verb and used to modify the verb are used as the action description information at the action detail level.

3. The method according to claim 1 or 2, characterized in that: The encoding of the action description information of each of the multiple semantic levels to obtain the action description representations of each of the multiple semantic levels includes: Encoding each action description information of each semantic level in the multiple semantic levels respectively to obtain a first feature vector of each action description information; Based on the semantic association relationship between at least one pair of action description information at different semantic levels, updating the first feature vector of each action description information based on the attention mechanism to obtain a second feature vector of each action description information; and For each of the semantic levels, the second feature vectors of at least two action description information belonging to the semantic level are concatenated, and the concatenated second feature vectors are used as the action description representation of the semantic level to obtain the action description representations of the multiple semantic levels.

4. The method according to claim 3, characterized in that The updating process based on the attention mechanism of the first feature vector of each action description information based on the semantic association relationship between at least one pair of action description information at different semantic levels to obtain the second feature vector of each action description information includes: Taking each of the action description information as a semantic node, and based on the semantic association relationship between at least one pair of action description information at different semantic levels, connecting two semantic nodes representing a pair of action description information having a semantic association relationship, so as to determine a connection edge connecting each of the semantic nodes; Using the first feature vector of each action description information as the node representation of each semantic node; Constructing a hierarchical semantic graph according to the semantic nodes, the edges connecting the semantic nodes, and the node representations of the semantic nodes; and The graph attention mechanism is used to update the node representation of each semantic node in the hierarchical semantic graph, and the second feature vector of each action description information is obtained according to the updated node representation of each semantic node.

5. The method according to claim 4, characterized in that The updating of the node representation of each semantic node in the hierarchical semantic graph by using the graph attention mechanism includes: For each semantic node in the hierarchical semantic graph, determining at least one adjacent node of the semantic node; Performing interactive processing on the node representation of the at least one adjacent node and the node representation of the targeted semantic node based on a graph attention mechanism to determine attention weight coefficients of the at least one adjacent node and the targeted semantic node; and According to the attention weight coefficient, a weighted sum is performed on the node representation of the at least one adjacent node and the node representation of the targeted semantic node to obtain an updated node representation of the targeted semantic node.

6. The method according to claim 4, characterized in that The method further comprises: In the case where the virtual object action is obtained, in response to an edge weight adjustment event of a connection edge connecting each of the semantic nodes in the hierarchical semantic graph, the edge weight of the connection edge indicated by the edge weight adjustment event is adjusted to obtain an updated hierarchical semantic graph; Using a graph attention mechanism, updating the node representation of each of the semantic nodes in the updated hierarchical semantic graph, and obtaining a third feature vector of each of the action description information according to the updated node representation of each of the semantic nodes; For each of the semantic levels, concatenate the third feature vectors of at least two pieces of action description information belonging to the semantic level, and use the concatenated third feature vectors as updated action description representations of the semantic level to obtain updated action description representations of the multiple semantic levels; and Based on the updated action description representations of each of the multiple semantic levels, an adjusted virtual object action is generated.

7. The method according to any one of claims 1 to 6, characterized in that: The step of performing noise reduction processing of the first semantic level on the sampled noise signal based on the action description representation of the first semantic level among the multiple semantic levels to obtain the action feature vector outputted by the first semantic level includes: The sampled noise signal is taken as a noise signal that has undergone multi-step noise addition. Starting from the last step of the multi-step noise addition, based on the action description representation of the first semantic level among the multiple semantic levels, a reverse noise reduction process is performed on the noise signal input at each step, and a noise reduction signal obtained by performing noise reduction process on the noise signal input at the first step is used as the action feature vector outputted at the first semantic level.

8. The method according to claim 7, characterized in that For each step of the multi-step noise addition, the step of performing noise reduction processing on the noise signal input for the noise addition step includes: Encoding the number of the noise adding step to obtain a noise adding step feature; The action description representation at the first semantic level is fused with the noise-added step feature to obtain a noise reduction condition feature; and According to the noise reduction condition characteristics, the noise signal input in the noise adding step is subjected to noise reduction processing to obtain a noise reduction signal.

9. The method according to claim 8, characterized in that The step of performing noise reduction processing on the noise signal inputted in the noise adding step according to the noise reduction condition characteristics to obtain the noise reduction signal comprises: According to the noise reduction condition characteristics and the noise signal inputted in the noise adding step, the added noise corresponding to the noise adding step is predicted to obtain a first predicted added noise corresponding to the noise adding step; and The first predicted added noise is subtracted from the noise signal inputted in the noise adding step to perform noise reduction processing to obtain a noise reduction signal.

10. The method according to any one of claims 1 to 9, characterized in that: The action of the virtual object is determined by a pre-trained action sequence generation model, which includes a cascade denoising network and a decoder; the cascade denoising network is used to perform denoising processing on each of the multiple semantic levels to obtain the action feature vector after cascade denoising of the multiple semantic levels; the decoder is used to decode the action feature vector after cascade denoising to obtain the virtual object action.

11. The method according to claim 10, characterized in that The cascade denoising network is obtained through a training step, and the training step includes: Obtaining multiple training samples; and For each training sample in the multiple training samples, the initial denoising network is trained according to the sample description text and the action sequence in the training sample to obtain a cascade denoising network.

12. The method according to claim 11, characterized in that The step of training the initial denoising network according to the sample description text and the action sequence in the training sample to obtain the cascade denoising network includes: By semantic analysis, the sample description text in the training sample is parsed at the multiple semantic levels to obtain sample description information of the multiple semantic levels; Encoding the sample description information of each of the multiple semantic levels respectively to obtain the sample description representations of each of the multiple semantic levels; and Based on the sample description representations of each of the multiple semantic levels and the action sequences in the targeted training samples, the initial denoising network is trained to obtain a cascade denoising network.

13. The method according to claim 12, characterized in that The step of training the initial denoising network based on the sample description representations of the respective multiple semantic levels and the action sequences in the targeted training samples to obtain a cascade denoising network comprises: Performing action encoding at multiple encoding levels on the action sequences in the training samples to obtain implicit action representations corresponding to the multiple semantic levels; and Based on the sample description representations of the multiple semantic levels and the sample description representations corresponding to the multiple semantic levels Implicit action representation, the initial denoising network is trained to obtain a cascade denoising network.

14. The method according to claim 13, characterized in that The multiple coding levels correspond to the multiple semantic levels one by one; the coding dimension of each coding level in the multiple coding levels increases in order from the first coding level to the last coding level; the action sequences in the training samples are respectively subjected to action coding at multiple coding levels to obtain implicit action representations corresponding to the multiple semantic levels, including: Performing motion coding at multiple coding levels on the motion sequences in the training samples to obtain motion latent space features of the multiple coding levels; and The motion latent space features of each of the multiple coding levels are decoded respectively to obtain implicit action representations corresponding to each of the multiple semantic levels.

15. The method according to claim 13, characterized in that The initial denoising network includes a plurality of cascaded initial denoisers; and each initial denoiser corresponds to a semantic level respectively; The training of the initial denoising network based on the sample description representations of the multiple semantic levels and the implicit action representations corresponding to the multiple semantic levels to obtain the cascade denoising network includes: For each of the multiple initial denoisers, based on sample description representations of at least two semantic levels from the first semantic level to a target semantic level corresponding to the targeted initial denoiser, and an implicit action representation corresponding to the target semantic level, the targeted initial denoiser is trained to obtain a trained denoiser; and A cascade denoising network is obtained according to the trained denoisers corresponding to each of the multiple initial denoisers.

16. The method according to claim 15, characterized in that The step of training each of the multiple initial denoisers based on sample description representations of at least two semantic levels from the first semantic level to a target semantic level corresponding to the initial denoiser, and an implicit action representation corresponding to the target semantic level, to obtain a trained denoiser includes: Get the number of noise adding steps for adding noise, and sample the random noise signal; According to the noise adding step number, the random noise signal is added to the implicit action representation corresponding to the target semantic level to obtain the noise action representation; Inputting the noise action representation, the number of noise adding steps, and the sample description representations of at least two semantic levels from the first semantic level to the target semantic level corresponding to the targeted initial denoiser into the targeted initial denoiser, and predicting the added noise by the targeted initial denoiser to obtain a second predicted added noise; and Noise is added according to the second prediction to adjust parameters of the initial denoiser to obtain a trained denoiser.

17. The method according to claim 16, characterized in that The step of inputting the noise action representation, the number of noise adding steps, and the sample description representations of at least two semantic levels from the first semantic level to the target semantic level corresponding to the targeted initial denoiser into the targeted initial denoiser, and predicting the added noise by the targeted initial denoiser to obtain the second predicted added noise comprises: When the targeted initial denoiser has an upper-level denoiser connected in series, the noise action representation, the number of noise addition steps, the sample description representations of at least two semantic levels from the first semantic level to the target semantic level corresponding to the targeted initial denoiser, and the reconstructed action representation output by the upper-level denoiser are input into the targeted initial denoiser, and the added noise is predicted by the targeted initial denoiser to obtain a second predicted added noise.

18. A virtual object action generation device, characterized in that: The device comprises: An acquisition module, used for acquiring an action description text for describing an action of a virtual object; A semantic parsing module, used to parse the action description text at a plurality of preset semantic levels through semantic analysis, obtain action description information of each of the plurality of semantic levels, and obtain a sampled noise signal for generating the action of the virtual object; An encoding module, used for encoding the action description information of each of the multiple semantic levels respectively, to obtain the action description representations of each of the multiple semantic levels; A first noise reduction processing module is used to perform noise reduction processing of a first semantic level among the multiple semantic levels on the sampled noise signal based on the action description representation of the first semantic level, so as to obtain an action feature vector outputted by the first semantic level; A second noise reduction processing module is used for each semantic level after the first semantic level in the multiple semantic levels to perform noise reduction processing on the sampled noise signal based on the action feature vector output by the previous semantic level and the action description representations of at least two semantic levels from the first semantic level to the current semantic level, so as to obtain the action feature vector after cascade noise reduction of the multiple semantic levels; wherein the action granularity level represented by the action feature vector output by the noise reduction processing of the multiple semantic levels decreases in order from the highest semantic level to the lowest semantic level; and A decoding module is used to decode the action feature vector after the cascade noise reduction to obtain the virtual object action.

19. The device according to claim 18, characterized in that The multiple semantic levels include an overall motion level, a local action level and an action detail level; the semantic parsing module is also used to use the action description text as the action description information of the overall motion level, and extract at least one verb and a phrase corresponding to the at least one verb for modifying the verb from the action description text, use the at least one verb as the action description information of the local action level, and use the phrase corresponding to the at least one verb for modifying the verb as the action description information of the action detail level.

20. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 17 are implemented.

Images