Method for generating virtual object behavior, device for generating virtual object behavior, computer program, and computer equipment

Abstract

This application relates to a method, apparatus, and computer equipment for generating virtual object motion. The method includes the steps of: acquiring motion description text (202); analyzing the motion description text at a predetermined number of semantic levels by semantic analysis to acquire motion description information for each of the multiple semantic levels and acquiring a sampling noise signal for generating virtual object motion (204); encoding the motion description information for each of the multiple semantic levels to acquire a motion description representation for each of the multiple semantic levels (206); and performing noise reduction processing on the sampling noise signal based on the motion description representation of the first semantic level to acquire a motion feature vector output from the first semantic level (208); and multiple semantic levels Step (210) is to obtain a cascaded noise-reduced operation feature vector that has gone through multiple semantic levels by performing noise reduction processing on the sampling noise signal based on the operation feature vector output from the previous semantic level and the operation description representations of at least two semantic levels from the first semantic level to the current semantic level, wherein the operation granularity level represented by the operation feature vector output from the noise reduction processing of multiple semantic levels decreases in order from the highest semantic level to the lowest semantic level, and Step (212) is to obtain virtual object operation by decoding the cascaded noise-reduced operation feature vector.

Description

Technical Field

[0001] This application claims priority based on a Chinese patent application filed with the China National Intellectual Property Administration on August 3, 2023, with an application number of 202310970212.1 and an invention title of "Virtual Object Action Generation Method, Device, Computer Equipment, and Storage Medium", and all of its contents are incorporated herein by reference.

[0002] This application relates to the technical field of computers, and particularly to a virtual object action generation method, device, computer equipment, computer-readable storage medium, and computer program product.

Background Art

[0003] With the development of computer technology, text-driven virtual object action generation technology has emerged. This technology can generate virtual object actions by using action description texts that describe virtual objects.

[0004] In the prior art, as a commonly adopted virtual object action generation method, an action description text is input into a generation model (such as an adversarial generation network, variational autoencoder, diffusion model, etc.) as a control signal, and the generation model directly maps the action description text to a virtual object action.

[0005] However, in the conventional method, since the action description text is directly mapped to the virtual object action, usually only virtual object actions with a coarse granularity can be generated, and there is a problem that the generated virtual object actions are not accurate.

Summary of the Invention

Problems to be Solved by the Invention

[0006] Embodiments of this application provide a virtual object action generation method, a virtual object action generation device, a computer program, and computer equipment. [Means for solving the problem]

[0007] A method for generating virtual object behaviors executed by a computer device, Steps include obtaining behavioral description text to describe the behavior of a virtual object, The steps include: obtaining action description information for each of the multiple semantic levels by analyzing the action description text at predetermined multiple semantic levels through semantic analysis, and obtaining a sampling noise signal for generating the virtual object action; The steps include: obtaining an action description representation for each of the multiple semantic levels by encoding the action description information for each of the multiple semantic levels; The steps include: obtaining an operation feature vector output from the first semantic level by performing noise reduction processing on the sampling noise signal based on the operation description representation of the first semantic level among the plurality of semantic levels; A step to obtain a cascaded noise-reduced operation feature vector that has passed through the multiple semantic levels by performing noise reduction processing on the sampling noise signal based on the operation feature vector output from the previous semantic level and the operation description representations of at least two semantic levels from the first semantic level to the current semantic level, wherein the operation granularity level represented by the operation feature vector output from the noise reduction processing of the multiple semantic levels decreases in order from the highest semantic level to the lowest semantic level. The process includes the step of obtaining the virtual object behavior by decoding the operational feature vector after cascading noise reduction.

[0008] A virtual object behavior generation device, A module for obtaining behavioral description text to describe the behavior of virtual objects, A semantic analysis module that analyzes the operation description text at predetermined multiple semantic levels through semantic analysis, obtains operation description information for each of the multiple semantic levels, and obtains a sampling noise signal for generating the virtual object operation, An encoding module that obtains an action description representation for each of the multiple semantic levels by encoding the action description information for each of the multiple semantic levels, A first noise reduction processing module obtains an operation feature vector output from the first semantic level by performing noise reduction processing on the sampling noise signal based on the operation description representation of the first semantic level among the plurality of semantic levels, A second noise reduction processing module obtains an operation feature vector after cascaded noise reduction through the multiple semantic levels by performing noise reduction processing on the sampling noise signal based on the operation feature vector output from the previous semantic level and the operation description representations of at least two semantic levels from the first semantic level to the current semantic level, wherein the operation granularity level represented by the operation feature vector output in the noise reduction processing of the multiple semantic levels decreases in order from the highest semantic level to the lowest semantic level, The system includes a decoding module that obtains the virtual object behavior by decoding the operational feature vector after cascaded noise reduction.

[0009] A computer device comprising memory and one or more processors, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the one or more processors, they cause the one or more processors to perform the steps of the virtual object operation generation method described above.

[0010] One or more non-volatile computer-readable storage media storing computer-readable instructions, wherein, when the computer-readable instructions are executed by one or more processors, the one or more processors cause the one or more processors to perform the steps of the virtual object behavior generation method described above.

[0011] A computer program product or computer program including computer-readable instructions, wherein the computer-readable instructions are stored in a computer-readable storage medium, and one or more processors of a computer device read the computer-readable instructions from the computer-readable storage medium, and the one or more processors execute the computer-readable instructions, thereby causing the computer device to perform the steps of the virtual object operation generation method described above.

[0012] Details of one or more embodiments of the present application are described in the following drawings and description. Other features, purposes, and advantages of the present application will become apparent from the specification, drawings, and claims. [Brief explanation of the drawing]

[0013] To more clearly describe the embodiments of the present application or the configuration of the prior art, the following is a brief introduction of the drawings necessary for describing the embodiments or the prior art. Clearly, the drawings in the following description only show some embodiments of the present application, and those skilled in the art can obtain other drawings from these without any creative effort. [Figure 1] This is a diagram illustrating the application environment of a virtual object behavior generation method in one embodiment. [Figure 2] This is a schematic diagram of the procedure for generating virtual object behavior in one embodiment. [Figure 3] This is a schematic diagram of the initial semantic level noise reduction process in one embodiment. [Figure 4] This is a schematic diagram illustrating the acquisition of the operational feature vector after cascade noise reduction in one embodiment. [Figure 5]It is a schematic diagram of a virtual object operation sequence in an embodiment. [Figure 6] It is a schematic diagram of operation description information at multiple semantic levels in an embodiment. [Figure 7] It is a schematic diagram of a hierarchical semantic diagram in an embodiment. [Figure 8] It is a schematic diagram of a hierarchical semantic diagram in other embodiments. [Figure 9] It is a schematic diagram of the generation of adjusted virtual object operations by edge weight adjustment in an embodiment. [Figure 10] It is a schematic diagram of a noise reduction processing process for obtaining an operation feature vector output from the first semantic level in an embodiment. [Figure 11] It is a schematic diagram of the prediction of additional noise corresponding to the target noise addition step in an embodiment. [Figure 12] It is a schematic diagram of the structure of a pre-trained operation sequence generation module in an embodiment. [Figure 13] It is an overall framework diagram of a virtual object operation generation method in an embodiment. [Figure 14] It is a configuration block diagram of a virtual object operation generation device in an embodiment. [Figure 15] It is an internal configuration diagram of a computer device in an embodiment.

Embodiments for Carrying Out the Invention

[0014] To make the object, configuration, and advantages of the present application clearer, the present application will be described in more detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described here are only for interpreting the present application and do not limit the present application.

[0015] The virtual object behavior generation method provided in the embodiment of the present invention is applicable to the application environment shown in Figure 1. Here, terminal 102 communicates with server 104 via a network. The data storage system can store data that server 104 needs to process. The data storage system may be integrated into server 104, or it may be located in the cloud or on another server. Server 104 obtains an action description text for describing the virtual object's behavior, analyzes the action description text at a predetermined number of semantic levels through semantic analysis to obtain action description information for each of the multiple semantic levels, obtains a sampling noise signal for generating the virtual object's behavior, encodes the action description information for each of the multiple semantic levels to obtain an action description representation for each of the multiple semantic levels, performs noise reduction processing on the sampling noise signal based on the action description representation of the first semantic level among the multiple semantic levels to obtain an action feature vector output from the first semantic level, performs noise reduction processing on the sampling noise signal for each semantic level after the first semantic level, based on the action feature vector output from the previous semantic level and the action description representations of at least two semantic levels from the first semantic level to the current semantic level to obtain an action feature vector after cascading noise reduction through multiple semantic levels, decodes the action feature vector after cascading noise reduction to obtain the virtual object's behavior, and pushes the virtual object's behavior to terminal 102 for display. Here, the operational granularity level, represented by the operational feature vectors output by noise reduction processing at multiple semantic levels, decreases progressively from the highest-level semantic level to the lowest-level semantic level.

[0016] Here, terminal 102 may be, but is not limited to, various desktop computers, laptop computers, smartphones, tablet computers, Internet of Things devices, and portable wearable devices. Internet of Things devices may include smart speakers, smart TVs, smart air conditioners, smart in-car devices, etc. Portable wearable devices may include smartwatches, smart bracelets, head-mounted devices, etc. Server 104 may be implemented as an independent server or as a server cluster consisting of multiple servers.

[0017] In one embodiment, a method for generating virtual object behavior is provided, as shown in Figure 2. This method may be executed by a terminal or a server alone, or it may be implemented jointly by a terminal and a server. In this embodiment, the case in which this method is applied to a server will be described as an example. This method includes the following steps.

[0018] Step 202 involves obtaining the behavior description text to describe the behavior of the virtual object.

[0019] Here, a virtual object refers to an object that can be active in a virtual environment. This active object may be a virtual person, a virtual animal, etc. For example, if the virtual environment is a 3D virtual environment, a virtual object refers to a virtual person, a virtual animal, etc., displayed within the 3D virtual environment. A virtual object has its own shape and volume within the 3D virtual environment and occupies a part of the space within the 3D virtual environment. The virtual environment is the environment provided when a client runs on a terminal. This virtual environment may be a simulation environment of the real world, a semi-simulation / semi-virtual environment, or a purely virtual environment. For example, the virtual environment may specifically be a 3D virtual environment.

[0020] Here, virtual object behavior refers to the actions a virtual object takes when it is active within the virtual environment. For example, a virtual object action may specifically be walking forward, standing up and then walking forward, walking to the right, or jumping forward. Action description text refers to the text that describes the virtual object behavior. Action description text may include information such as action type, movement path, and action style. Here, action type refers to the type to which the virtual object behavior belongs. For example, a specific action type may be walking, running, or jumping. The movement path indicates the direction of the virtual object's movement. For example, a specific movement path may be forward, left, or right. Action style indicates the state of the virtual object while it is moving. For example, a specific action style may be happy or sad. To illustrate with an example, the action description text may specifically say that a person walks forward, then turns left, and then continues walking to the right. Here, "a person" refers to a virtual object.

[0021] Specifically, when virtual object motion generation is required, the server obtains motion description text to describe the virtual object motion, and generates the virtual object motion based on information such as motion type, motion path, and motion style contained in the motion description text. In specific applications, the virtual object motion generation in this application can be widely applied to scenarios such as augmented reality (AR) / virtual reality (VR) content creation, game content creation, and 3D animation design in order to efficiently produce realistic and diverse virtual object motions.

[0022] In step 204, semantic analysis is performed to analyze the action description text at a predetermined number of semantic levels, thereby obtaining action description information for each of the multiple semantic levels and obtaining a sampling noise signal for generating virtual object behavior.

[0023] Here, semantic analysis refers to analyzing the meaning of each word in the action description text to determine the structure of the action description text and the part of speech of each word in the action description text. For example, the structure of the action description text may take the form of (attributive modifier) ​​subject + (adverbial modifier) ​​predicate + (complement or attributive modifier) ​​+ object. Also, for example, the part of speech of a word in the action description text may be a noun, verb, adverb, adjective, preposition, etc.

[0024] Here, the semantic level refers to the angle from which the virtual object's behavior is described. Multiple semantic levels are used to describe the virtual object's behavior from multiple different angles. Different semantic levels focus on different angles. By using multiple semantic levels to describe the virtual object's behavior from multiple different angles, a comprehensive description of the virtual object's behavior can be achieved. In this embodiment, multiple semantic levels may be predefined according to the actual application scenario. For example, multiple semantic levels may specifically include a global motion level, a local motion level, and a motion detail level. The global motion level is primarily used to describe the virtual object's behavior as a whole. The local motion level is primarily used to describe the virtual object's behavior by some local motions included in the virtual object's behavior. The motion detail level is primarily used to describe the virtual object's behavior by some details of local motions.

[0025] Here, semantic-level action description information refers to information for describing the action of a virtual object at the semantic level. For example, if the semantic level is the overall motion level, the semantic-level action description information may specifically be information that describes the action of the virtual object as a whole. Also, for example, if the semantic level is the local action level, the semantic-level action description information may specifically be verbs that express some local actions included in the action of the virtual object. Also, for example, if the semantic level is the action detail level, the semantic-level action description information may specifically be modifiers that modify verbs that express some local actions included in the action of the virtual object.

[0026] Here, the sampling noise signal refers to the noise signal acquired using a random sampling method when attempting to generate virtual object behavior. For example, the sampling noise signal may specifically refer to the Gaussian noise signal acquired using a random sampling method when attempting to generate virtual object behavior.

[0027] Specifically, the semantic analysis method may be based on semantic role analysis. In this case, the server obtains action description information at multiple semantic levels by analyzing the action description text at predetermined multiple semantic levels based on semantic role analysis, and obtains a sampling noise signal for generating virtual object actions using a random sampling method. Here, semantic role refers to the different roles played by different sentence components (e.g., subject, object, time, place, etc.) within an action event described in a sentence. The names of these roles are usually nouns or verbal parts within a verb phrase. In this embodiment, semantic role refers to the different roles played by different sentence components (e.g., subject, object, time, place, etc.) in the action description text. Note that the semantic role that a sentence component plays in a sentence depends on the predicate verb.

[0028] In a specific application, when analyzing an action description text at a predetermined number of semantic levels based on semantic role analysis, the server first divides the action description text into multiple different sentence components, recognizes verbs from the action description text, and then determines the role played by each different sentence component based on the semantic relationship between the multiple different sentence components and the verb, thereby obtaining action description information at multiple semantic levels.

[0029] In one specific application, the server may use a pre-trained natural language model for semantic analysis to analyze behavioral description text at predetermined semantic levels. By inputting behavioral description text into the pre-trained natural language model for semantic analysis, behavioral description information at multiple semantic levels can be obtained. Here, this pre-trained natural language model for semantic analysis may be trained according to the actual application scenario. For example, this pre-trained natural language model for semantic analysis may specifically be a Bidirectional Encoder Representations from Transformers (BERT) model for relation extraction and semantic role annotation.

[0030] In one specific application, the server may use a semantic role analysis tool to analyze the behavior description text at a predetermined number of semantic levels through semantic analysis. By inputting the behavior description text into the semantic role analysis tool, behavior description information at multiple semantic levels can be obtained. Here, the semantic role analysis tool may be selected according to the actual application scenario. For example, this semantic role analysis tool may specifically be AllenNLP (a Natural Language Processing (NLP) research library based on PyTorch (an open-source Python machine learning library based on Torch used in applications such as natural language processing) to provide the industry's best state-of-the-art deep learning models for each language task).

[0031] In step 206, the operation description information for each of the multiple semantic levels is encoded to obtain the operation description representation for each of the multiple semantic levels.

[0032] Here, an action description representation refers to a feature that can represent action description information at the semantic level. For example, an action description representation refers to a feature vector that can represent action description information at the semantic level.

[0033] Specifically, the server obtains a first feature vector for each action description by encoding the action description information for each of the multiple semantic levels, and then obtains the action description representation for each of the multiple semantic levels based on the first feature vector of each action description. Here, the first feature vector refers to a feature vector that can represent the content of the action description information. The first feature vector allows the action description information to be distinguished from other information.

[0034] In a specific application, the server may obtain a first feature vector for each action description by encoding each action description at each of several semantic levels using a pre-trained natural language model for text feature extraction. Here, the pre-trained natural language model for text feature extraction may be trained according to the actual application scenario. For example, this pre-trained natural language model for text feature extraction may specifically be a Contrastive Language-Image Pre-Training (CLIP) model. A CLIP model is a pre-trained model that can be trained using unlabeled data. A trained CLIP model can take text (or an image) as input and output a vector representation of the text (or image). In this embodiment, when action description information is input, it outputs a first feature vector, which is a vector representation of the action description information. Unlike other models of single-text modality or single-image modality, CLIP is multimodal and includes content from both image processing and text processing.

[0035] In one specific application, the pre-training task of the CLIP model is to predict whether a given image and text are a pair, using contrast learning loss. In this embodiment, the CLIP model is pre-trained using a contrast learning method to determine whether a text and image are a pair by treating the image and its corresponding text directly as a single whole. The main structure of the CLIP model includes a text encoder and an image encoder. During training, the CLIP model obtains vector representations of the image and text by inputting the training image and text into the image encoder and text encoder, respectively, then obtains new directly comparable vector representations of the image and text by mapping the vector representations of the image and text into a common multimodal space, and finally calculates the similarity between the vector representations of the image and text. The objective function of contrast learning is to increase the similarity of positive sample pairs and decrease the similarity of negative sample pairs.

[0036] In a specific application, after obtaining the first feature vector of each action description information, the server may, for each semantic level, merge the first feature vectors of at least two action description information belonging to that semantic level, and use the merged feature vector as the action description representation of that semantic level, thereby obtaining action description representations for multiple semantic levels. In one specific application, for each semantic level, the server may merge the first feature vectors of at least two action description information belonging to that semantic level using methods such as concatenation or superposition.

[0037] Furthermore, before fusing the first feature vectors of the behavioral description information belonging to the semantic level, the server may first update the first feature vector of each behavioral description information based on semantic relationship relationships between behavioral description information at least one pair of different semantic levels, in order to achieve an accurate representation of each behavioral description information in conjunction with the contextual content.

[0038] In step 208, based on the behavioral description representation of the first of multiple semantic levels, the behavioral feature vector output from the first semantic level is obtained by performing noise reduction processing on the sampled noise signal based on the behavioral description representation of the first semantic level.

[0039] Here, noise reduction processing refers to the removal of noise from the sampled noise signal. The behavioral feature vector output from the initial semantic level refers to a vector that can represent the characteristics of the virtual object's behavior at the initial semantic level.

[0040] Specifically, in the noise reduction process for the first semantic level among multiple semantic levels, the server performs noise reduction processing on the sampled noise signal at the guidance of the operation description representation of the first semantic level, thereby reconstructing the operation feature vector output from the first semantic level. In a specific application, the server treats the sampled noise signal as a noise signal that has undergone multi-step noise addition, predicts the noise signal added at each step of the multi-step noise addition based on the operation description representation of the first semantic level, performs step-by-step noise reduction processing on the sampled noise signal based on the noise signal added at each step, and obtains the operation feature vector output from the first semantic level by subtracting the noise signal added at each step from the sampled noise signal.

[0041] Furthermore, the initial semantic-level action description exists as a condition for generating the action feature vector, guiding its generation and enabling the generated action feature vector to correlate more closely with the initial semantic-level action description.

[0042] In one specific application, the initial semantic level noise reduction process may involve, as shown in Figure 3, treating the sampled noise signal n as a noise signal after multi-step noise addition (shown in Figure 3 as T-step noise addition), predicting the noise signals added at each step of the multi-step noise addition based on the initial semantic level operation description, performing step-by-step noise reduction on the sampled noise signal n based on the noise signals added at each step, and obtaining the operation feature vector output from the initial semantic level by subtracting the noise signals added at each step from the sampled noise signal. As shown in Figure 3, the server performs reverse noise reduction on the input noise signal from the last step of the multi-step noise addition (noise addition step number T) based on the initial semantic level operation description. The noise signal that has been noise-reduced at the last step of the multi-step noise addition is

[0043]

number

[0044]

number

[0045]

number

[0046]

number

[0047] In step 210, for each semantic level after the first of the multiple semantic levels, noise reduction processing is performed on the sampling noise signal based on the operation feature vector output from the previous semantic level and the operation description representations of at least two semantic levels from the first semantic level to the current semantic level. This process obtains an operation feature vector after cascading noise reduction through multiple semantic levels, and the operation granularity level represented by the operation feature vector output from the noise reduction processing of multiple semantic levels is reduced in order from the highest semantic level to the lowest semantic level.

[0048] Here, granularity refers to the level of detail in data statistics within the same dimension. In this embodiment, the same dimension refers to the dimension of the description of virtual object behavior. Therefore, the behavior granularity level refers to the level of detail in the description of virtual object behavior, that is, the degree of subdivision or synthesis of the description of virtual object behavior. To make it easier to understand, the higher the degree of subdivision of the description of virtual object behavior, the smaller the behavior granularity level represented by the behavior feature vector. Conversely, the lower the degree of subdivision of the description of virtual object behavior, the larger the behavior granularity level represented by the behavior feature vector.

[0049] In this embodiment, the motion granularity level, represented by the motion feature vectors output by noise reduction processing at multiple semantic levels, decreases progressively from the highest semantic level to the lowest semantic level. That is, the motion granularity level represented by the motion feature vector output by the noise reduction processing at the first semantic level, which is the highest semantic level, is the largest, and the motion granularity level represented by the motion feature vectors output from each subsequent semantic level decreases with each semantic level; in other words, the motion granularity level decreases with each semantic level. To understand this, a smaller motion granularity level indicates a higher degree of subdivision of the virtual object's motion description by the motion feature vectors, meaning that the motion granularity becomes finer, allowing for the inclusion of richer, finer-grained motion details.

[0050] Specifically, for each semantic level after the first of multiple semantic levels, the server performs noise reduction processing on the sampling noise signal based on the operation feature vector output from the previous semantic level and the operation description representations of at least two semantic levels from the first semantic level to the current semantic level, thereby obtaining a cascaded noise-reduced operation feature vector that has gone through multiple semantic levels. In a specific application, in the noise reduction processing for each semantic level after the first semantic level, the server treats the sampling noise signal as a noise signal that has undergone multi-step noise addition, predicts the noise signal added at each step of the multi-step noise addition based on the operation feature vector output from the previous semantic level and the operation description representations of at least two semantic levels from the first semantic level to the current semantic level, and performs step-by-step noise reduction processing on the sampling noise signal based on the noise signal added at each step, thereby obtaining an operation feature vector output from that semantic level.

[0051] In one specific application, in the noise reduction processing for each semantic level after the initial semantic level, the server performs reverse noise reduction processing on the input noise signal at each step, starting from the last step of multi-step noise addition, based on the operation feature vector output from the previous semantic level and the operation description representations for at least two semantic levels from the initial semantic level to the current semantic level. The noise reduction signal obtained by processing the input noise signal at the first step of multi-step noise addition is then used as the operation feature vector output from that semantic level.

[0052] In one specific application, in the noise reduction process for each semantic level after the initial semantic level, for each multi-step noise addition step, the server obtains noise addition step features by encoding the step number of the target noise addition step, obtains noise reduction condition features by fusing the noise addition step features, the operation feature vector output from the previous semantic level, and the operation description representations of at least two semantic levels from the initial semantic level to the current semantic level, predicts the noise added in the target noise addition step based on the noise reduction condition features and the input noise signal of the target noise addition step, and obtains a noise reduction signal by performing noise reduction processing on the input noise signal of the target noise addition step based on the predicted added noise.

[0053] In one specific application, noise reduction processing at each semantic level may be implemented using a single noise reducer. In this case, cascaded noise reduction may specifically refer to applying noise reduction processing to the sampled noise signal using multiple series-connected noise reducers. For example, as shown in Figure 4, the server has three noise reducers.

[0054]

number

[0055]

number

[0056]

number

[0057]

number

[0058]

number

[0059]

number

[0060]

number

[0061]

number

[0062]

number

[0063] In one specific application, a noise reducer

[0064]

number

[0065]

number

[0066]

number

[0067]

number

[0068]

number

[0069]

number

[0070]

number

[0071]

number

[0072]

number

[0073] In step 212, the virtual object behavior is obtained by decoding the behavior feature vector after cascaded noise reduction.

[0074] Specifically, the server can obtain virtual object motion by decoding the motion feature vector after cascading noise reduction. In a specific application, decoding the motion feature vector after cascading noise reduction, that is, converting the motion feature vector after cascading noise reduction into the attitude space of the virtual object using a mapping method, may result in the obtained virtual object motion being a virtual object motion sequence. In other words, the virtual object motion generation method according to the present invention makes it possible to generate a corresponding virtual object motion sequence from given motion description information.

[0075] In one specific application, the given behavior description information may be in Chinese or in text in another language. Taking the case where the behavior description information is in Chinese as an example, as shown in Figure 5 (the behavior description information in Figure 5 is translated into Japanese), 10 examples of corresponding virtual object behavior sequences generated from the given behavior description information are shown. As can be seen from the example in Figure 5, the virtual object behavior generation method according to the present invention can generate high-quality virtual object behavior sequences.

[0076] In the above virtual object motion generation method, motion description text for describing virtual object motion is obtained, and by semantic analysis, the motion description text is analyzed at a predetermined number of semantic levels to obtain motion description information for each of the multiple semantic levels, a sampling noise signal for generating virtual object motion is obtained, and by encoding the motion description information for each of the multiple semantic levels, motion description representations for each of the multiple semantic levels can be obtained, and by performing noise reduction processing on the sampling noise signal based on the motion description representation of the first semantic level among the multiple semantic levels, a motion feature vector output from the first semantic level can be obtained, and for each semantic level after the first semantic level among the multiple semantic levels, noise reduction processing is performed on the sampling noise signal using the motion feature vector output from the previous semantic level and the motion description representations for at least two semantic levels from the first semantic level to the current semantic level as common conditions, thereby using the motion description representations for each of the multiple semantic levels to successively enrich the finer granularity of motion details and obtain a motion feature vector after cascade noise reduction that has passed through multiple semantic levels and accurately represents the virtual object motion at a finer granularity, and furthermore, by decoding the motion feature vector after cascade noise reduction, the virtual object motion can be obtained. The overall process involves using the behavioral description information from multiple semantic levels as fine-grained control signals, capturing the behavioral characteristics of multiple semantic levels to subdivide and generate virtual object behavior, thereby improving the accuracy of the generated virtual object behavior.

[0077] In one embodiment, the multiple semantic levels include a global motion level, a local motion level, and a motion detail level, and the step of obtaining motion description information for each of the multiple semantic levels by analyzing the motion description text at predetermined multiple semantic levels through semantic analysis includes the steps of: using the motion description text as motion description information at the global motion level, and extracting from the motion description text at least one verb and a phrase for modifying the verb corresponding to each of the at least one verb; and using the at least one verb as motion description information at the local motion level, and using the phrase for modifying the verb corresponding to each of the at least one verb as motion description information at the motion detail level.

[0078] Here, the overall motion level is primarily used to describe the virtual object's motion as a whole. Describing the virtual object's motion as a whole means describing it from the perspective of the object performing the motion, the local actions performed, and the details of the local actions. The local action level is primarily used to describe the virtual object's motion by a few local actions included in the virtual object's motion. The motion detail level is primarily used to describe the virtual object's motion by a few details of local actions. For example, the details of a local action may be the direction of execution of the virtual object's motion, the state of the virtual object during execution, etc. A phrase corresponding to a verb and modifying a verb refers to a phrase that modifies a verb in a sentence. For example, a phrase corresponding to a verb and modifying a verb may specifically refer to an adjective, adverb, preposition, etc. that modifies the verb.

[0079] Specifically, the multiple semantic levels include the overall motion level, the local motion level, and the motion detail level. Analyzing the motion description text at a predetermined multiple semantic levels through semantic analysis means extracting motion description information for each semantic level from the motion description text, considering the multiple semantic levels. When extracting motion description information at multiple semantic levels, the server uses the motion description text as motion description information at the overall motion level, extracts at least one verb and at least one phrase to modify the verb corresponding to each verb, uses at least one verb as motion description information at the local motion level, and uses the phrases to modify the verb corresponding to each of the at least one verb as motion description information at the motion detail level.

[0080] In a specific application, the server may determine the part of speech of each word in the action description text by performing part-of-speech analysis on each word, thereby determining at least one verb. Furthermore, by analyzing the relationship between at least one verb and each word, the server may determine a phrase to modify the verb that corresponds to each of the at least one verb.

[0081] In one specific application, if the action description text is "Someone walks forward, then turns left, and then continues walking to the right," then at least one verb that can be extracted from it is "walk," "turn," and "continue walking." The phrases modifying the verb corresponding to "walk" are "Someone," "forward," the phrases modifying the verb corresponding to "turn" are "Someone," "next," "left," and the phrases modifying the verb corresponding to "continue walking" are "Someone," "then," "right." After performing a hierarchical semantic analysis, the action description information obtained at multiple semantic levels may be as shown in Figure 6. The action description information at the overall motion level is "Someone walks forward, then turns left, and then continues walking to the right," the action description information at the local motion level is "walk," "turn," and "continue walking," and the action detail level is "Someone," "forward," "next," "left," "then," and "right."

[0082] In this embodiment, this method makes it possible to acquire motion description information at the overall motion level, local motion level, and motion detail level. Furthermore, by using motion description information at multiple semantic levels as fine-grained control signals and capturing motion characteristics at multiple semantic levels, it is possible to subdivide and generate virtual object motion, thereby improving the accuracy of the generated virtual object motion.

[0083] In one embodiment, the step of obtaining an action description representation for each of the multiple semantic levels by encoding the respective action description information for each of the multiple semantic levels includes: obtaining a first feature vector for each action description information by encoding the respective action description information for each of the multiple semantic levels; obtaining a second feature vector for each action description information by performing an update process based on an attention mechanism on the first feature vector of each action description information based on the semantic relationship between action description information for at least one pair of different semantic levels; and obtaining an action description representation for each of the multiple semantic levels by concatenating the second feature vectors of at least two action description information belonging to each semantic level, and using the concatenated second feature vector as the action description representation for the semantic level.

[0084] Here, the first feature vector refers to a vector used to represent the action description information after encoding it. A semantic relationship refers to a relationship in which two pairs of action description information are semantically related. For example, if the two action description information consists of a verb and modifiers such as adverbs, adjectives, and prepositions that modify the verb, then a semantic relationship is considered to exist between the two action description information. The second feature vector refers to a vector used to represent the action description information after updating the first feature vector.

[0085] Specifically, the server obtains a first feature vector for each action description by encoding each action description from each of the multiple semantic levels, and obtains a second feature vector for each action description after updating by performing an update process based on an attention mechanism on the first feature vectors of action description information that have semantic relationships based on semantic relationship relationships between action description information from at least one pair of different semantic levels, and obtains an action description representation for each of the multiple semantic levels by concatenating the second feature vectors of at least two action description information belonging to each semantic level and making the concatenated second feature vector the action description representation of the semantic level.

[0086] In a specific application, the server may obtain a first feature vector for each action description by encoding each action description at each of the multiple semantic levels using a pre-trained natural language model for text feature extraction. When performing update processing based on the attention mechanism, for each action description, the server performs interaction processing based on the attention mechanism on the first feature vector of the target action description and the action description that has a semantic relationship with the target action description, determines the attention weight coefficients for the target action description and the action description that has a semantic relationship with the target action description, and obtains a second feature vector for the target action description by weighting and adding the first feature vectors of the target action description and the action description that has a semantic relationship with the target action description based on the attention weight coefficients.

[0087] In this embodiment, a first feature vector of each action description information can be obtained by encoding, and by performing an update process based on an attention mechanism on the first feature vector using semantic relationship relationships, a second feature vector that accurately represents each action description information can be obtained after carefully considering the action description information with semantic relationship relationships. Furthermore, by concatenating the second feature vectors of action description information belonging to the same semantic level, it is possible to obtain action description representations for multiple semantic levels.

[0088] In one embodiment, the step of obtaining a second feature vector for each action description by performing an update process based on an attention mechanism on the first feature vector of each action description based on the semantic relationship between action description information at least one pair of different semantic levels includes: setting each action description as a semantic node, connecting two semantic nodes that represent a pair of action description information with a semantic relationship based on the semantic relationship between action description information at least one pair of different semantic levels, and determining the connecting edge that connects each semantic node; setting the first feature vector of each action description as the node representation of each semantic node; constructing a hierarchical semantic graph based on each semantic node, the connecting edge that connects each semantic node, and the node representation of each semantic node; and updating the node representation of each semantic node in the hierarchical semantic graph using a graph attention mechanism, and obtaining a second feature vector for each action description based on the updated node representation of each semantic node.

[0089] Here, the graph attention mechanism is used to achieve better neighbor aggregation by introducing an attention mechanism, and by learning neighbor weights, weighted neighbor aggregation can be achieved. For this reason, learning noisy neighbors using graph attention is more robust, and the attention mechanism gives the model a certain degree of interpretability. Furthermore, by dynamically paying attention to the features of neighbors, the graph attention mechanism further enhances graph-based inference compared to simple graph convolution.

[0090] Specifically, the server uses each action description as a semantic node and, based on the semantic relationship between at least one pair of action description information at different semantic levels, obtains connection edges connecting each semantic node by connecting two semantic nodes that represent a pair of action description information (action description information at different semantic levels) that have a semantic relationship. Based on this, the server uses the first feature vector of each action description as the node representation of each semantic node, and further constructs a hierarchical semantic graph based on each semantic node, the connection edges connecting each semantic node, and the node representation of each semantic node. After constructing the hierarchical semantic graph, the server uses a graph attention mechanism to update the node representation of each semantic node in the hierarchical semantic graph, and uses the updated node representation of each semantic node as the second feature vector of the action description information corresponding to each semantic node.

[0091] In a specific application, when using the graph attention mechanism to update the node representation of each semantic node in a hierarchical semantic graph, for each semantic node in the hierarchical semantic graph, the server determines at least one neighboring node of the target semantic node and updates the node representation of the target semantic node using the node representation of at least one neighboring node and the node representation of the target semantic node.

[0092] In one specific application, if the action description text is "A person walks forward, then turns left, and then continues walking to the right," and there are multiple semantic levels including the overall motion level, local motion level, and motion detail level, the constructed hierarchical semantic graph may be as shown in Figure 7. Here, the semantic node at the overall motion level, "A person walks forward, then turns left, and then continues walking to the right," is connected to the semantic nodes at the local motion level, "walk," "turn," and "continue walking." The semantic node at the local motion level, "walk," is connected to the semantic nodes at the motion detail level, "A person," and "forward." The semantic node at the local motion level, "turn," is connected to the semantic nodes at the motion detail level, "A person," "next," and "forward." The semantic node at the local motion level, "continue walking," is connected to the semantic nodes at the motion detail level, "A person," "then," and "to the right." In one specific application, the constructed hierarchical semantic graph may be simplified to one as shown in Figure 8. Here, the global motion level includes one semantic node (also called the global motion node), the local motion level includes three semantic nodes (also called local motion nodes), and the motion detail level includes six semantic nodes (also called motion detail nodes).

[0093] In this embodiment, semantic nodes are determined, connecting edges connecting each semantic node are determined based on semantic relationships, and the node representation of each semantic node is determined. Then, using each semantic node, the connecting edges connecting each semantic node, and the node representation of each semantic node, a hierarchical semantic graph representing the semantic hierarchy of the action description text can be constructed. Furthermore, by using a graph attention mechanism, the node representation of each semantic node in the hierarchical semantic graph can be updated, and by allowing the node representations of each semantic node to interact sufficiently, the updated node representation of each semantic node can be obtained. Furthermore, using the updated node representation of each semantic node, a second feature vector that accurately represents each action description information can be obtained.

[0094] In one embodiment, the step of updating the node representation of each semantic node in a hierarchical semantic graph using a graph attention mechanism includes the steps of: determining at least one adjacent node of the target semantic node for each semantic node in the hierarchical semantic graph; performing interaction processing based on the graph attention mechanism on the node representation of at least one adjacent node and the node representation of the target semantic node to determine attention weight coefficients for at least one adjacent node and the target semantic node; and obtaining the updated node representation of the target semantic node by weighted addition of the node representation of at least one adjacent node and the node representation of the target semantic node based on the attention weight coefficients.

[0095] Here, an adjacent node refers to a semantic node that is connected to the target semantic node via a connecting edge in a hierarchical semantic graph. For example, in a hierarchical semantic graph as shown in Figure 7, at least one adjacent node of the overall motion level semantic node "Someone walks forward, then turns left, and then continues walking to the right" is the local action level semantic node "walk," "turn," and "continue walking." At least one adjacent node of the local action level semantic node "walk" is the overall motion level semantic node "Someone walks forward, then turns left, and then continues walking to the right," as well as the action detail level semantic nodes "Someone" and "forward." At least one adjacent node of the local action level semantic node "turn" is the overall motion level semantic node "Someone walks forward, then turns left, and then continues walking to the right," as well as the action detail level semantic nodes "Someone," "next," and "left." At least one adjacent node to the local action level semantic node "continue walking" is the global motion level semantic node "someone walks forward, then turns left, then continues walking to the right," as well as the action detail level semantic nodes "someone," "then," and "to the right."

[0096] Specifically, for each semantic node in the hierarchical semantic graph, the server determines at least one neighboring node of the target semantic node based on the connection relationships between each semantic node in the hierarchical semantic graph. It then performs interaction processing based on the graph attention mechanism on the node representation of at least one neighboring node and the node representation of the target semantic node, determining attention weight coefficients for at least one neighboring node and the target semantic node. For each of the at least one neighboring node, this attention weight coefficient represents the importance of this neighboring node's node representation to the target semantic node. Based on this, the server can obtain the updated node representation of the target semantic node by weighting and adding the node representation of at least one neighboring node and the node representation of the target semantic node based on the attention weight coefficients.

[0097] In a specific application, when performing interaction processing based on a graph attention mechanism on the node representation of at least one adjacent node and the node representation of the target semantic node, the server may determine the attention weight coefficients of at least one adjacent node and the target semantic node by calculating similarity. That is, for each adjacent node among the at least one adjacent node, the server may calculate the node representation similarity between the node representation of the target adjacent node and the node representation of the target semantic node, and use the node representation similarity as the attention weight coefficient of the target adjacent node.

[0098] In a specific application, when performing interaction processing based on a graph attention mechanism on the node representation of at least one adjacent node and the node representation of the target semantic node, the server may determine the attention weight coefficients of at least one adjacent node and the target semantic node by first performing a linear transformation and then mapping. That is, for each adjacent node among the at least one adjacent node, the server may first use a pre-trained linear transformation layer to perform one linear transformation (i.e., mapping to high-dimensional features to obtain sufficient representational power) on the node representation of the target adjacent node and the node representation of the target semantic node, then concatenate the two node representations after the linear transformation, map the two concatenated node representations to a single real number, and use this real number as the attention weight coefficient of the target adjacent node.

[0099] In a specific application, the graph attention mechanism of this embodiment allows only adjacent nodes to participate in the attention mechanism of the target semantic node, and furthermore, graph structure information is introduced, meaning that when performing interaction processing based on the graph attention mechanism, only one-hop adjacent nodes are considered. Note that the one-hop adjacent nodes of the target semantic node include the target semantic node itself, which can be understood as a self-loop edge.

[0100] In one specific application, after determining the attention weight coefficients of at least one adjacent node and the target semantic node, the server may normalize the attention weight coefficients of at least one adjacent node and the target semantic node to facilitate comparison of attention weight coefficients between different semantic nodes, and then obtain the updated node representation of the target semantic node by weighted addition of the node representation of at least one adjacent node and the node representation of the target semantic node using the normalized attention weight coefficients.

[0101] In one embodiment, the server may use a graph attention mechanism with a pre-trained graph attention network to update the node representation of each semantic node in a hierarchical semantic graph. By inputting the node representation of each semantic node in the hierarchical semantic graph into the pre-trained graph attention network, the updated node representation of each semantic node can be output from this pre-trained graph attention network.

[0102] Furthermore, the pre-trained graph attention network, when updating the node representation of each semantic node in a hierarchical semantic graph using the graph attention mechanism, employs essentially the same processing principle as in the above embodiment. In both cases, for each semantic node in the hierarchical semantic graph, first, at least one neighboring node of the target semantic node is determined. Next, interaction processing based on the graph attention mechanism is performed on the node representation of at least one neighboring node and the node representation of the target semantic node to determine the attention weight coefficients of at least one neighboring node and the target semantic node. Finally, the updated node representation of the target semantic node is obtained by weighting and adding the node representation of at least one neighboring node and the node representation of the target semantic node based on the attention weight coefficients.

[0103] In this embodiment, for each semantic node in the hierarchical semantic graph, first, at least one adjacent node of the target semantic node is determined, and then interaction processing based on the graph attention mechanism is performed on the node representation to determine the attention weight coefficients of at least one adjacent node and the target semantic node. Furthermore, by weighting and adding the node representation of at least one adjacent node and the node representation of the target semantic node based on the attention weight coefficients, the node representation of the target semantic node can be updated. As a result, the node representation of the target semantic node can be sufficiently merged with the node representation of the adjacent node, and the behavioral description information can be represented more accurately.

[0104] In one embodiment, the virtual object behavior generation method further includes the steps of: obtaining an updated hierarchical semantic graph by adjusting the edge weights of the connecting edges indicated by edge weight adjustment events in response to edge weight adjustment events of the connecting edges that connect each semantic node in the hierarchical semantic graph when a virtual object behavior is obtained; updating the node representation of each semantic node in the updated hierarchical semantic graph using a graph attention mechanism and obtaining a third feature vector of each behavior description information based on the updated node representation of each semantic node; concatenating the third feature vectors of at least two behavior description information belonging to each semantic level for each semantic level, and obtaining updated behavior description expressions for multiple semantic levels using the concatenated third feature vector as the updated behavior description expression for the semantic level; and generating an adjusted virtual object behavior based on the updated behavior description expressions for multiple semantic levels.

[0105] Here, an edge weight adjustment event refers to an event that adjusts the weights of the connecting edges that connect each semantic node in a hierarchical semantic graph. For example, if the initial weights of the connecting edges that connect each semantic node in a hierarchical semantic graph are the same, then by adjusting the weight of at least one of those connecting edges through an edge weight adjustment event, finer-grained control over the generation of virtual object behavior may be achieved.

[0106] Specifically, when virtual object behavior is acquired, if it is necessary to achieve finer-grained control over the generation of virtual object behavior, the interacting object can trigger edge weight adjustment events on the connecting edges that connect each semantic node in the hierarchical semantic graph. The server responds to these edge weight adjustment events by adjusting the edge weights of the connecting edges indicated by the events, thereby acquiring an updated hierarchical semantic graph. After acquiring the updated hierarchical semantic graph, the server uses a graph attention mechanism to update the node representation of each semantic node in the updated hierarchical semantic graph. The updated node representation of each semantic node is used as the third feature vector of the corresponding behavior description information. For each semantic level, the third feature vectors of at least two behavior description information belonging to that semantic level are concatenated, and the concatenated third feature vector is used as the updated behavior description representation of the semantic level. The updated behavior description representations for multiple semantic levels are then acquired, and the adjusted virtual object behavior is generated using the updated behavior description representations for multiple semantic levels. This enables finer-grained control over the generation of virtual object behavior.

[0107] In a specific application, after obtaining updated behavioral description representations for each of multiple semantic levels, the server obtains an adjusted behavioral feature vector output from the first semantic level by performing noise reduction processing on the sampling noise signal based on the updated behavioral description representation of the first semantic level. For each semantic level after the first semantic level, the server obtains an adjusted behavioral feature vector after cascaded noise reduction by performing noise reduction processing on the sampling noise signal based on the adjusted behavioral feature vector output from the previous semantic level and the updated behavioral description representations for at least two semantic levels from the first semantic level to the current semantic level. Finally, the server obtains the adjusted virtual object behavior by decoding the adjusted behavioral feature vector after cascaded noise reduction.

[0108] In a specific application, the interacting entity can trigger edge weight adjustment events for the connecting edges that link each semantic node in a hierarchical semantic graph via speech or text. After the server receives the speech or text from the interacting entity, it recognizes the speech or text to identify the adjustment scheme for adjusting the edge weights and adjusts the edge weights indicated by the scheme. For example, if the action description text is "A person walks forward, then turns left, and then continues walking to the right," the speech or text to adjust the edge weights might be "Turn a little further to the left." After receiving this speech or text, the server determines that the adjustment scheme is "Turn a little further to the left," adjusts the edge weights indicated by the scheme (i.e., the weights of the connecting edges that link the two semantic nodes "turn" and "left"), and increases these edge weights to achieve "Turn a little further to the left."

[0109] In one specific application, as shown in Figure 9, if the action description text is "A person walks forward, then turns left, and then continues walking to the right," the generated baseline virtual object action is as shown in Figure 9. If the weight of the edge connecting the semantic node "turn" and the semantic node "left" (the edge connecting semantic node 3 and semantic node 8, as shown in Figure 9) is increased (i.e., strengthened), the amount of the left turn increases, as can be seen from the comparison between the fine-tuning result (adjusted virtual object action) in Figure 9 and the baseline virtual object action. If the weight of the edge connecting the semantic node "turn" and the semantic node "left" is decreased (i.e., weakened), the amount of the left turn decreases, as can be seen from the comparison between the fine-tuning result (adjusted virtual object action) in Figure 9 and the baseline virtual object action.

[0110] In one specific application, increasing the weight of the edge connecting the semantic node "Someone walks forward, then turns left, then continues walking to the right" and the semantic node "continues walking" (the edge connecting semantic node 1 and semantic node 4, as shown in Figure 9) makes the "continues walking" action more apparent, as can be seen from comparing the fine-tuning result (adjusted virtual object behavior) in Figure 9 with the reference virtual object behavior. Decreasing the weight of the edge connecting the semantic node "Someone walks forward, then turns left, then continues walking to the right" and the semantic node "continues walking" makes the "continues walking" action less apparent, as can be seen from comparing the fine-tuning result (adjusted virtual object behavior) in Figure 9 with the reference virtual object behavior.

[0111] In this embodiment, by adjusting the edge weights of connected edges in the hierarchical semantic graph, it is possible to generate adjusted virtual objects. By using edge weight adjustment, finer granularity control of virtual object generation can be achieved, and the generated adjusted virtual objects can be made to better suit the requirements.

[0112] In one embodiment, the step of obtaining an operation feature vector output from the first semantic level by performing noise reduction processing on the sampled noise signal based on the operation description expression of the first semantic level among multiple semantic levels includes the step of making the sampled noise signal a noise signal after multi-step noise addition, performing reverse noise reduction processing on the input noise signal of each step based on the operation description expression of the first semantic level among multiple semantic levels from the last step of multi-step noise addition, and making the noise-reduced signal obtained by performing noise reduction processing on the input noise signal of the first step an operation feature vector output from the first semantic level.

[0113] Specifically, when performing noise reduction processing at the initial semantic level, the server processes the sampled noise signal into a noise signal that has undergone multi-step noise addition. Starting from the last step of the multi-step noise addition, the server uses the operation description representation at the initial semantic level as a guide to perform reverse noise reduction processing on the input noise signal at each step. The noise-reduced signal obtained by processing the input noise signal at the first step is then used as the operation feature vector output from the initial semantic level.

[0114] In a specific application, the input noise signal for the last step of multi-step noise addition is the sampled noise signal, and from the second to last step of multi-step noise addition onward, the input noise signal for each step is the noise-reduced signal output after noise reduction processing in the next step. In each step of multi-step noise addition, it is necessary to use the initial semantic-level operation description as a guide, predict the noise added in the target noise addition step based on the initial semantic-level operation description and the target noise addition step, and then perform noise reduction processing on the input noise signal of the target noise addition step based on the predicted added noise.

[0115] In one specific application, assuming that multi-step noise addition is T-step noise addition, when performing noise reduction processing on a sampled noise signal, it is necessary to perform T-step noise reduction processing. The server takes the sampled noise signal as a noise signal that has undergone T-step noise addition, and from noise reduction step number T, using the operation description expression at the first semantic level as a guide, performs reverse noise reduction processing on the input noise signal at each step, and the noise-reduced signal that has undergone noise reduction processing on the input noise signal at the first step (noise reduction step number 1) is taken as the operation feature vector output from the first semantic level. When the noise reduction step number is T, the input noise signal is the sampled noise signal, and from noise reduction step number T-1, the input noise signal at each step is the noise-reduced signal output after noise reduction processing in the next step.

[0116] In one specific application, the noise reduction process at each step may be implemented based on a pre-trained noise reducer. This pre-trained noise reducer may be configured and trained according to the actual application scenario. In this case, the noise reduction process to obtain the operation feature vector output from the initial semantic level may be as shown in Figure 10. For each T-step noise addition step, the pre-trained noise reducer may perform noise prediction based on the operation description representation at the initial semantic level, the input noise signal, and the target noise addition step. Furthermore, by using the noise predicted by the pre-trained noise reducer to perform noise reduction on the input noise signal, a noise-reduced signal is obtained, and after completing the noise reduction process on the input noise signal of the first step (noise reduction step number 1), the noise-reduced signal obtained by performing noise reduction on the input noise signal of the first step (noise reduction step number 1) may be used as the operation feature vector output from the initial semantic level. In this process, the pre-trained noise reducer is used T times.

[0117] Here, as shown in Figure 10, the server performs reverse noise reduction processing on the input noise signal based on the initial semantic level operation description representation, starting from the last step of multi-step noise addition (noise addition step number T). The noise signal that has been noise-reduced in the last step of multi-step noise addition is

[0118]

number

[0119]

number

[0120]

number

[0121]

number

[0122] In this embodiment, the sampling noise signal is treated as a noise signal after multi-step noise addition, and from the last step of the multi-step noise addition, reverse noise reduction processing is performed on the input noise signal of each step based on the operation description expression of the first semantic level. By using the operation description expression of the first semantic level as a guide, accurate noise reduction can be achieved step by step, and the operation feature vector output from the first semantic level can be obtained.

[0123] In one embodiment, the step of performing noise reduction processing on the input noise signal of a target noise addition step for each step of adding noise in multiple steps includes: obtaining noise addition step features by encoding the step number of the target noise addition step; obtaining noise reduction condition features by fusing the initial semantic level operation description expression with the noise addition step features; and obtaining a noise reduction signal by performing noise reduction processing on the input noise signal of the target noise addition step based on the noise reduction condition features.

[0124] Here, a noise addition step feature refers to a feature that represents the target noise addition step, allowing it to be distinguished from other noise addition steps. A noise reduction condition feature refers to a feature that serves as a guide condition for the noise reduction process. Noise reduction processes performed for different noise reduction condition features are not exactly the same. For example, the additional noise corresponding to the predicted target noise addition step will differ when performing noise reduction processing for different noise reduction condition features.

[0125] Specifically, for each step of multi-step noise addition, when performing noise reduction processing on the input noise signal of the target noise addition step, the server obtains noise addition step features by encoding the step number of the target noise addition step, then obtains noise reduction condition features by fusing the initial semantic-level operation description representation with the noise addition step features, and finally obtains a noise-reduced signal by performing noise reduction processing on the input noise signal of the target noise addition step using the noise reduction condition features as a guide.

[0126] In a specific application, the server may encode the step number of the target noise addition step using a pre-trained coding network. Here, the pre-trained coding network may be configured according to the actual application scenario. For example, the pre-trained coding network may be a pre-trained multi-layer perceptron (MLP). The server may obtain noise reduction condition features by fusing the initial semantic-level behavioral description representation and the noise addition step features in a concatenated manner.

[0127] In this embodiment, noise addition step features can be obtained by encoding the step number of the target noise addition step, and noise reduction condition features can be obtained by fusing the initial semantic-level operation description expression with the noise addition step features. Furthermore, noise reduction processing can be performed on the input noise signal of the target noise addition step using the noise reduction condition features as a guide, thereby obtaining a noise reduction signal and achieving noise reduction.

[0128] In one embodiment, the step of obtaining a noise-reduced signal by performing noise reduction processing on the input noise signal of the target noise addition step based on noise reduction condition features includes the step of obtaining a first predicted additional noise corresponding to the target noise addition step by predicting the additional noise corresponding to the target noise addition step based on the noise reduction condition features and the input noise signal of the target noise addition step, and the step of obtaining a noise-reduced signal by subtracting the first predicted additional noise from the input noise signal of the target noise addition step and performing noise reduction processing.

[0129] Specifically, the server obtains attention-encoded vectors corresponding to the noise reduction condition features and the input noise signal of the target noise addition step by encoding them based on the attention mechanism. Next, it obtains the first predicted additional noise corresponding to the target noise addition step by decoding the attention-encoded vectors. Finally, it obtains the noise-reduced signal by subtracting the first predicted additional noise from the input noise signal of the target noise addition step and performing noise reduction processing.

[0130] Here, the attention mechanism is a resource allocation approach that, under circumstances where computational power is limited, allocates computational resources to more important tasks and solves the problem of information overload. In neural network learning, generally, the more parameters a model has, the stronger its expressive power becomes and the larger the amount of information it stores, but this leads to the problem of information overload. By introducing the attention mechanism, the information overload problem can be solved and the efficiency and accuracy of task processing can be improved by focusing on information that is more important to the current task among a large amount of input information, reducing attention to other information, and ultimately filtering out irrelevant information. In this embodiment, the information that is more important for predicting the first predicted additional noise is focused on among the input noise signals of the noise reduction condition features and the target noise addition step, thereby improving the efficiency and accuracy of predicting the first predicted additional noise.

[0131] In a specific application, the attention mechanism may be a multi-head attention mechanism. The server may obtain a first predicted additional noise corresponding to the target noise addition step by a multi-level encoding and decoding process. In one specific application, the server may achieve prediction of the additional noise corresponding to the target noise addition step by a pre-trained noise reducer. This pre-trained noise reducer takes noise reduction condition features and the input noise signal of the target noise addition step as input and outputs a first predicted additional noise corresponding to the target noise addition step. This pre-trained noise reducer may be configured and trained according to the actual application scenario. In one specific application, this pre-trained noise reducer may be a network based on an N1-layer Transformer (transformation network) and N2 attention headers, where N1 and N2 are positive integers and may be configured according to the actual application scenario.

[0132] In one specific application, a schematic diagram of the prediction of additional noise corresponding to the target noise addition step may be shown in Figure 11. The server uses an MLP (multilayer perceptron) to encode the step number t of the target noise addition step, thereby obtaining the noise addition step features, and concatenates the initial semantic-level operation description representation c with the noise addition step features (in Figure 11).

[0133]

number

[0134] In this embodiment, by predicting the additional noise corresponding to the target noise addition step based on the noise reduction condition characteristics and the input noise signal of the target noise addition step, a first predicted additional noise corresponding to the target noise addition step can be obtained. Furthermore, by directly performing noise reduction processing on the input noise signal of the target noise addition step based on the first predicted additional noise, a noise reduction signal can be obtained, and noise reduction can be achieved using the noise prediction method.

[0135] In one embodiment, the virtual object's behavior is determined by a pre-trained behavior sequence generation model, which includes a cascaded noise reduction network and a decoder. The cascaded noise reduction network obtains a behavior feature vector after cascaded noise reduction across multiple semantic levels by performing noise reduction processing on each of the multiple semantic levels. The decoder obtains the virtual object's behavior by decoding the behavior feature vector after cascaded noise reduction.

[0136] Specifically, a pre-trained motion sequence generation model refers to a model for generating virtual object motion. This motion sequence generation model includes a cascaded noise reduction network and a decoder. The cascaded noise reduction network performs noise reduction processing on each of the multiple semantic levels to obtain a motion feature vector after cascaded noise reduction across multiple semantic levels. The decoder obtains the virtual object motion by decoding the motion feature vector after cascaded noise reduction.

[0137] In a specific application, taking the case where the cascaded noise reduction network includes three noise reducers, the structure of the pre-trained operation sequence generation model may be as shown in Figure 12. In the cascaded noise reduction network, the inputs to the first semantic level noise reducer are the sampled noise signal n and the operation description representation of the first semantic level.

[0138]

number

[0139]

number

[0140]

number

[0141]

number

[0142]

number

[0143]

number

[0144] In this embodiment, an action sequence generation model including a cascaded noise reduction network and a decoder can be used to achieve accurate inference for virtual object behavior and improve the accuracy of the generated virtual object behavior.

[0145] In one embodiment, the cascaded noise reduction network is obtained by a training step, which includes the steps of acquiring a plurality of training samples and acquiring the cascaded noise reduction network by training an initial noise reduction network for each of the plurality of training samples based on the sample description text and operation sequence in the target training sample.

[0146] Here, a training sample refers to a sample used to train a cascaded noise reduction network. Each training sample includes a sample description text and an action sequence. The sample description text in a training sample describes the action sequence in that training sample. That is, the sample description text in a training sample corresponds to the action sequence. Similar to the action description text, the sample description text may also include information such as the action type, motion path, and action style. An action sequence is a sequence consisting of multiple actions, and these multiple actions correspond to the virtual object actions described in the sample description text. For example, these multiple actions may specifically be at least two actions in the process of a virtual object walking forward and making a right turn. The number of actions in an action sequence may be arranged according to the actual application scenario. An initial noise reduction network refers to a noise reduction network that has not undergone parameter training. By training the initial noise reduction network, a cascaded noise reduction network can be obtained.

[0147] Specifically, the server acquires multiple training samples and, for each of the training samples, trains an initial noise reduction network based on the sample description text and operation sequence in the target training sample, thereby acquiring a cascaded noise reduction network. In a specific application, the initial noise reduction network includes multiple cascaded initial noise reducers, and training the initial noise reduction network means training multiple cascaded initial noise reducers and giving the trained multiple initial noise reducers noise prediction capability. Furthermore, the pre-trained cascaded noise reduction network can be used to perform noise reduction processing on the sampled noise signal and generate an operation feature vector after cascaded noise reduction.

[0148] In this embodiment, by acquiring multiple training samples, an initial noise reduction network can be trained using the sample description text and operation sequence in each training sample, thereby enabling the acquisition of a cascaded noise reduction network. This allows for noise reduction processing using the cascaded noise reduction network, enabling accurate inference regarding virtual object behavior and improving the accuracy of the generated virtual object behavior.

[0149] In one embodiment, the step of obtaining a cascaded noise reduction network by training an initial noise reduction network based on the sample description text and operation sequence in the target training sample includes the steps of obtaining sample description information for each of the multiple semantic levels by analyzing the sample description text in the target training sample at multiple semantic levels by semantic analysis; obtaining sample description representations for each of the multiple semantic levels by encoding each of the sample description information for each of the multiple semantic levels; and obtaining a cascaded noise reduction network by training an initial noise reduction network based on each of the sample description representations for each of the multiple semantic levels and the operation sequence in the target training sample.

[0150] Specifically, the semantic analysis method may be based on semantic role analysis. In this case, the server obtains sample description information for each of the multiple semantic levels by analyzing the sample description text in the target training sample at multiple semantic levels based on semantic role analysis, obtains a fourth feature vector for each sample description by encoding each sample description information for each of the multiple semantic levels, obtains a sample description representation for each of the multiple semantic levels based on the fourth feature vector of each sample description, and obtains a cascaded noise reduction network by training an initial noise reduction network based on the sample description representations for each of the multiple semantic levels and the operation sequence in the target training sample. Here, in order to facilitate training and processing, the operation sequence in the target training sample may be sequenced data.

[0151] In a specific application, the server may obtain a fourth feature vector for each sample description by encoding each sample description for each of the multiple semantic levels using a pre-trained natural language model for text feature extraction, and then obtain a sample description representation for each of the multiple semantic levels by concatenating the fourth feature vectors of sample description information for the same semantic level. Here, the pre-trained natural language model for text feature extraction may be trained according to the actual application scenario.

[0152] In this embodiment, by performing hierarchical semantic analysis and encoding, sample description representations for each of multiple semantic levels can be obtained. Then, using the sample description representations and the operation sequences in the target training samples, an initial noise reduction network can be trained, and a cascaded noise reduction network can be obtained. This allows for noise reduction processing using the cascaded noise reduction network, enabling accurate inference for virtual object behavior and improving the accuracy of the generated virtual object behavior.

[0153] In one embodiment, the step of obtaining a cascaded noise reduction network by training an initial noise reduction network based on sample description representations of multiple semantic levels and the action sequences in the target training samples includes the step of obtaining latent action representations corresponding to each of the multiple semantic levels by performing action coding of multiple coding levels on the action sequences in the target training samples, and the step of obtaining a cascaded noise reduction network by training an initial noise reduction network based on sample description representations of multiple semantic levels and the latent action representations corresponding to each of the multiple semantic levels.

[0154] Here, any of the multiple coding levels are used to perform action coding on the action sequence in the target training sample, and different coding levels perform action coding on the action sequence in different coding dimensions. With this method, a multi-dimensional latent action representation can be obtained. The latent action representation can be understood as a latent action coding vector that represents the action sequence.

[0155] Specifically, for each of the multiple coding levels, the server learns an action representation by encoding and decoding the action sequence in the target training sample, and obtains a latent action representation for that coding level. If latent action representations for multiple coding levels are obtained, each of the latent action representations for multiple coding levels becomes the latent action representation corresponding to each of the multiple semantic levels. After obtaining the latent action representations corresponding to each of the multiple semantic levels, the server obtains a cascaded noise reduction network by training an initial noise reduction network using the sample description representations for each of the multiple semantic levels and the latent action representations corresponding to each of the multiple semantic levels.

[0156] In this embodiment, by performing action coding at multiple coding levels on an action sequence, latent action representations corresponding to each of multiple semantic levels can be obtained. Furthermore, by utilizing the latent action representations and sample description representations corresponding to each of the multiple semantic levels, an initial noise reduction network can be trained from the perspective of multiple semantic levels, thereby obtaining a cascaded noise reduction network that can achieve fine-grained noise reduction.

[0157] In one embodiment, there is a one-to-one correspondence between multiple coding levels and multiple semantic levels, and the coding dimension of each coding level among the multiple coding levels increases sequentially from the first coding level to the last coding level. The step of obtaining latent action representations corresponding to each of the multiple semantic levels by performing action coding of each of the multiple coding levels on the action sequence in the target training sample includes the step of obtaining the motion latent space features of each of the multiple coding levels by performing action coding of each of the multiple coding levels on the action sequence in the target training sample, and the step of obtaining latent action representations corresponding to each of the multiple semantic levels by decoding the motion latent space features of each of the multiple coding levels.

[0158] Here, motor latent space features refer to features obtained by mapping the motion sequences in the target training samples to latent space. Latent space is a compressed representation of data, and its role is to simplify the representation of data by learning the features of the data in order to find patterns. By mapping data to latent space, the dimensionality of the data can be reduced. Encoding dimension refers to the number of dimensions of the motion coding to the motion sequence at each encoding level. The encoding dimension of each encoding level among multiple encoding levels increases progressively from the first encoding level to the last encoding level. That is, the encoding dimension of each encoding level among multiple encoding levels increases progressively with each encoding level. In other words, the encoding dimension of the first encoding level is the smallest, and the encoding dimension of the last encoding level is the largest.

[0159] Specifically, there is a one-to-one correspondence between multiple coding levels and multiple semantic levels, and the coding dimension of each coding level increases progressively from the first coding level to the last coding level. That is, the feature dimension of the acquired motion latent space features also increases progressively with each coding level. For each coding level among the multiple coding levels, the server performs motion coding on the motion sequence in the target training sample using the coding dimension of the target coding level to acquire the motion latent space features of the target coding level, decodes the motion latent space features of the target coding level to acquire the latent motion representation corresponding to the target coding level, and sets the latent motion representation corresponding to the target coding level as the latent motion representation corresponding to the semantic level corresponding to the target coding level.

[0160] In specific applications, the motion sequence may be sequenced data, and the motion latent spatial features may be a motion feature data distribution corresponding to the motion sequence. By sampling the motion feature data distribution, motion feature sampling points corresponding to the motion sequence can be obtained, and furthermore, by decoding the motion feature sampling points, a latent motion representation corresponding to the target coding level can be obtained.

[0161] In one specific application, the acquired behavioral feature data distribution includes the mean and variance. Based on this, the server may randomly sample points from a standard normal distribution and, using a reparameterization technique, obtain behavioral feature sample points corresponding to the behavioral sequence based on the mean and variance, as well as the randomly sampled sample points. Here, as a principle of the reparameterization technique, if z is a random variable following a Gaussian distribution with mean g(x) and covariance h(x), then z is

[0162]

number

[0163]

number

[0164] In one specific application, a pre-trained variational autoencoder may be used to implement the steps of first encoding, then sampling, and finally decoding in this embodiment. For each of the multiple encoding levels, the latent action representation corresponding to the target encoding level may be obtained by inputting the action sequence in the target training sample into the pre-trained variational autoencoder.

[0165] In one specific application, a variational autoencoder can be defined as an autoencoder whose training is normalized to avoid overfitting, ensuring that the latent space has good attributes that allow it to perform the data generation process. Like standard autoencoders, a variational autoencoder is an encoder-decoder structure that is trained to minimize the reconstruction error between the encoded and decoded data and the initial data. However, to introduce some normalization of the latent space, variational autoencoders make some modifications to the encoding-decoding process. That is, instead of encoding the input as a single point in the latent space, it encodes it as a probability distribution in the latent space. The training process for a variational autoencoder is as follows: First, the input is encoded as a distribution in the latent space; second, a point in the latent space is sampled from this distribution; third, the sampled point is decoded to calculate the reconstruction error; and finally, the reconstruction error is backpropagated through the network.

[0166] In this embodiment, by performing action coding at multiple coding levels on an action sequence, latent action representations corresponding to each of the multiple semantic levels can be obtained, thereby realizing a latent representation of the action sequence. Furthermore, by utilizing the latent action representations and sample description representations corresponding to each of the multiple semantic levels, an initial noise reduction network can be trained from the perspective of multiple semantic levels, thereby obtaining a cascaded noise reduction network that can achieve fine-grained noise reduction.

[0167] In one embodiment, the initial noise reduction network includes a cascaded set of initial noise reducers, each initial noise reducer corresponding to one semantic level, and the step of obtaining a cascaded noise reduction network by training the initial noise reduction network based on the sample description representations for each of the multiple semantic levels and the latent operation representations corresponding to each of the multiple semantic levels includes the step of obtaining a trained noise reducer by training a target initial noise reducer for each of the multiple initial noise reducers based on the sample description representations for at least two semantic levels from the first semantic level to the target semantic level corresponding to the target initial noise reducer and the latent operation representation corresponding to the target semantic level, and the step of obtaining a cascaded noise reduction network based on the trained noise reducers corresponding to each of the multiple initial noise reducers.

[0168] Specifically, the initial noise reduction network includes multiple cascaded initial noise reducers, each corresponding to one semantic level. When training the initial noise reduction network, for each of the multiple initial noise reducers, the server obtains a trained noise reducer by training the target initial noise reducer based on sample description representations for at least two semantic levels from the initial semantic level to the target semantic level corresponding to the target initial noise reducer, and the latent operation representation corresponding to the target semantic level. Based on the trained noise reducers corresponding to each of the multiple initial noise reducers, the server obtains the cascaded noise reduction network.

[0169] In a specific application, when training the target initial noise reducer, the server first performs noise addition processing on the latent operating representation corresponding to the target semantic level corresponding to the target initial noise reducer. Next, using the sample description representations of at least two semantic levels from the initial semantic level to the target semantic level corresponding to the target initial noise reducer as conditions, the server uses the target initial noise reducer to predict the noise added during the noise addition process. By comparing the noise actually added during the noise addition process with the noise added during the noise addition process predicted by the target initial noise reducer, the server adjusts the parameters of the target initial noise reducer so that it can achieve accurate noise prediction. As a result, during the inference phase, accurate noise prediction can be achieved using the target initial noise reducer, and noise reduction processing can be performed using the predicted noise.

[0170] In this embodiment, a trained noise reducer can be obtained by training a target initial noise reducer for each of the multiple initial noise reducers based on sample description representations for at least two semantic levels, from the initial semantic level to the target semantic level corresponding to the target initial noise reducer, and a latent operation representation corresponding to the target semantic level. Furthermore, a cascaded noise reduction network can be obtained based on the trained noise reducers corresponding to each of the multiple initial noise reducers.

[0171] In one embodiment, the step of obtaining a trained noise reducer by training a target initial noise reducer for each of a plurality of initial noise reducers based on sample description representations for at least two semantic levels from the initial semantic level to the target semantic level corresponding to the target initial noise reducer, and a latent operation representation corresponding to the target semantic level, includes the steps of obtaining a noise addition step number for noise addition and sampling a random noise signal; obtaining a noise operation representation by adding the random noise signal to the latent operation representation corresponding to the target semantic level based on the noise addition step number; obtaining a second predicted additional noise by inputting the noise operation representation, the noise addition step number, and the sample description representations for at least two semantic levels from the initial semantic level to the target semantic level corresponding to the target initial noise reducer to the target initial noise reducer and predicting the added noise by the target initial noise reducer; and obtaining a trained noise reducer by performing parameter adjustments on the target initial noise reducer based on the second predicted additional noise.

[0172] Specifically, when training a target initial noise reducer for each of the multiple initial noise reducers, the server first obtains a noise addition step number for noise addition, samples a random noise signal, and, based on the noise addition step number, adds the random noise signal step by step to the latent operating representation corresponding to the target semantic level to obtain a noise operating representation. The server then inputs the noise operating representation, the noise addition step number, and the sample description representations for at least two semantic levels from the initial semantic level to the target semantic level corresponding to the target initial noise reducer into the target initial noise reducer. The target initial noise reducer then predicts the added noise to obtain a second predicted additional noise, and finally, based on the second predicted additional noise, the server adjusts the parameters of the target initial noise reducer to obtain a trained noise reducer.

[0173] In a specific application, the number of noise addition steps for noise addition may be set according to the actual application scenario. This embodiment is not limited thereto. Note that the larger the noise addition step number, the closer the acquired noise behavior representation will be to a Gaussian distribution. For this reason, a noise behavior representation with added random noise signals can be considered as Gaussian noise. In this embodiment, according to the noise addition step number, a sampled random noise signal is added step by step to the latent behavior representation corresponding to the target semantic level corresponding to the target initial noise reducer, thereby destroying this latent behavior representation to become complete Gaussian noise, and in the reverse step, the target initial noise reducer is used to learn the process of restoring from Gaussian noise to the latent behavior representation corresponding to the target semantic level corresponding to the target initial noise reducer.

[0174] In one specific application, the step of obtaining a trained noise reducer by training the initial noise reducer in this embodiment is implemented based on a diffusion model. The diffusion model is a generative model that learns noise prediction through a Markov noise addition process and ultimately achieves the transformation of a Gaussian noise distribution into a target data distribution. The diffusion model differs from other generative networks in that, in the forward stage, it adds noise to the sample step by step until the sample is destroyed into complete Gaussian noise, and then, in the reverse stage, it learns the process of reconstructing the original sample from the Gaussian noise.

[0175] In this embodiment, "sample" refers to the latent operating representation corresponding to the target semantic level corresponding to the target initial noise reducer; "adding noise step by step" refers to adding sampled random noise signals step by step based on the noise addition step number; "Gaussian noise" refers to the noise operating representation; and "learning in the reverse step" refers to inputting the noise operating representation, the noise addition step number, and the sample description representations for at least two semantic levels from the initial semantic level to the target semantic level corresponding to the target initial noise reducer into the target initial noise reducer, and obtaining a second predicted additional noise by predicting the added noise using the target initial noise reducer.

[0176] In a specific application, the server obtains the predicted noise error by comparing the second predicted additional noise with a random noise signal. If the predicted noise error is greater than the error threshold, it adjusts the parameters of the target initial noise reducer based on the predicted noise error. The server then continues to train the adjusted initial noise reducer until the calculated predicted noise error is less than or equal to the error threshold, thereby obtaining a trained noise reducer. Here, the error threshold may be set according to the actual application scenario.

[0177] In this embodiment, a noise addition step number for noise addition is obtained, and by sampling a random noise signal, the noise addition step number is used to add the random noise signal to the latent operation representation corresponding to the target semantic level, thereby realizing the noise addition process and obtaining the noise operation representation. Furthermore, the noise operation representation, the noise addition step number, and the sample description representations of at least two semantic levels from the initial semantic level to the target semantic level corresponding to the target initial noise reducer are input to the target initial noise reducer, which then predicts the added noise, learns the noise prediction, and obtains the second predicted added noise. Based on the second predicted added noise, the parameters of the target initial noise reducer are adjusted to obtain a trained noise reducer, thereby realizing the training of the initial noise reducer.

[0178] In one embodiment, the step of obtaining a second predicted additional noise by inputting a noise operation expression, a noise addition step number, and sample description expressions for at least two semantic levels from the initial semantic level to the target semantic level corresponding to the target initial noise reducer into the target initial noise reducer, and having the target initial noise reducer predict the added noise, includes, if a preceding noise reducer in series with the target initial noise reducer exists, the step of inputting a noise operation expression, a noise addition step number, sample description expressions for at least two semantic levels from the initial semantic level to the target semantic level corresponding to the target initial noise reducer, and a reconstruction operation expression output from the preceding noise reducer into the target initial noise reducer, and having the target initial noise reducer predict the added noise, thereby obtaining a second predicted additional noise.

[0179] Specifically, in the process of training the target initial noise reducer, if there is a previous noise reducer connected in series with the target initial noise reducer, the server inputs the noise operation representation, the noise addition step number, the sample description representations for at least two semantic levels from the initial semantic level to the target semantic level corresponding to the target initial noise reducer, and the reconstructed operation representation output from the previous noise reducer to the target initial noise reducer. The target initial noise reducer then predicts the added noise and obtains the second predicted additional noise. In a specific application, the reconstructed operation representation output from the previous noise reducer refers to the representation corresponding to the latent operation representation before noise addition, which is reconstructed after the previous noise reducer predicts the added noise based on the input data and performs noise reduction processing on the input noise operation representation based on the predicted noise; in other words, the operation representation restored from the noise operation representation by learning noise prediction.

[0180] In this embodiment, if there is a preceding noise reducer connected in series with the target initial noise reducer, the noise operation expression, the noise addition step number, the sample description expressions for at least two semantic levels from the initial semantic level to the target semantic level corresponding to the target initial noise reducer, and the reconstruction operation expression output from the preceding noise reducer are input to the target initial noise reducer. By predicting the added noise using the target initial noise reducer, the noise prediction can be learned by referring to the reconstruction operation expression output from the preceding noise reducer, thereby improving the accuracy of the noise prediction and obtaining a second predicted additional noise.

[0181] This invention proposes a text-driven method for generating highly controllable virtual object (specifically, virtual character) motion based on hierarchical meaning. This method receives motion description text for virtual object motion as input and synthesizes the corresponding virtual object motion based on information such as motion type, motion path, and motion style specified in the motion description text. Specifically, this virtual object motion may be the skeleton or grid sequence of a 3D virtual object. Compared to conventional methods, the inventors believe that the invention analyzes the input text as motion description information for multiple semantic levels, which are new control signals, and uses this motion description information for each semantic level as a fine-grained control signal. By capturing the motion characteristics of each semantic level, the invention subdivides and generates virtual object motion, thereby improving the accuracy of the generated virtual object motion.

[0182] Specifically, the multiple semantic levels in this application include the overall motion level, the local motion level, and the motion detail level. Correspondingly, the text-to-motion generation process is also decomposed into three semantic levels corresponding to the capture of the overall motion, local motion, and motion details, respectively. Compared to conventional methods, the method of this application has better controllability and can synthesize high-quality virtual object motion. This virtual object motion may specifically be a motion sequence.

[0183] The inventors believe that current text-driven human motion generation methods can be categorized into two types: one based on joint coding and the other based on a diffusion model. The joint coding method typically learns one motion variational autoencoder and one text variational autoencoder. This method then uses KL divergence to constrain the text and motion encoders to a shared latent space. The diffusion model method uses a conditional diffusion model for human motion generation to learn a robust probabilistic mapping from text descriptors to human motion sequences. Both of these methods rely on a global text representation and directly learn the mapping from the global text representation, which has a high-level language, to motion sequences.

[0184] However, conventional methods, which directly use neural networks to automatically and potentially extract text features, may overemphasize certain details in the text while ignoring other important information. As a result, the network is not sensitive to subtle changes in the input text and lacks fine-grained control. Furthermore, conventional methods are unable to generate action details well. On the one hand, a single action text description always involves multiple actions and attributes. However, the global text representations extracted by current methods usually do not convey the clarity and detail necessary for a good understanding of the text, and therefore cannot effectively guide the synthesis of action details. On the other hand, the direct mapping from global text representations with high-level language to action sequences in conventional methods further hinders the generation of action details.

[0185] Based on this, the present invention proposes a text-driven virtual object motion generation method that allows for fine control based on hierarchical meaning. By utilizing the characteristic of motion description text having a hierarchical structure, the motion description text is analyzed at a predetermined number of semantic levels through semantic analysis to obtain motion description information for each of the multiple semantic levels, and controllable motion generation is performed by using the motion description information for each of the multiple semantic levels as fine-grained signals. Specifically, the sentences in the motion description text describe an overall motion that includes multiple motions, and the overall motion consists of several local motions, each local motion consisting of different motion details that are its attributes (e.g., direction of movement and velocity of the motion). This structure from global to local helps to ensure a reliable and comprehensive understanding of the motion description and realizes fine-grained control of virtual object motion.

[0186] In one embodiment, the virtual object motion generation method of the present invention will be described using as an example a case in which the fine granularity of virtual object motion generation is controlled based on a hierarchical semantic graph constructed from motion description information of multiple semantic levels, and in which the multiple semantic levels include an overall motion level, a local motion level, and a motion detail level.

[0187] Specifically, the overall framework of the virtual object motion generation method of the present invention, as shown in Figure 13, mainly includes two core components: a graph inference module and a coarse-to-fine motion sequence generation module. For motion description text to describe virtual object motion, the server extracts at least one verb appearing in the motion description text and a phrase modifying the verb corresponding to each of the at least one verb, based on a semantic role analysis tool, and obtains motion description information for each of multiple semantic levels by determining the semantic role of each phrase modifying the verb. After obtaining motion description information for each of multiple semantic levels, the server makes the motion description text a global motion node at the global motion level in a hierarchical semantic graph, and connects at least one verb to the global motion node directly by an edge as a local motion node at the local motion level in the hierarchical semantic graph. The server also makes the phrases modifying the verb corresponding to each of the at least one verbs motion detail nodes at the motion detail level, which are connected to the corresponding local motion nodes. The server then uses a pre-trained text encoder to encode the action description text, at least one verb, and the phrases that modify each of the at least one verb as node representations of the corresponding semantic nodes.

[0188] In the graph inference module, the server uses a pre-trained graph attention network to construct interactions between different levels in a hierarchical semantic graph, with the aim of reducing ambiguity in each semantic node. For example, the verb "to pick up" can represent different actions when there is no context, but the phrase "with both hands" that modifies the verb removes the possible ambiguity of this verb, so the action should be "to pick up with both hands" and not "to pick up with one hand." Therefore, by inferring interactions in the hierarchical semantic graph using a pre-trained graph attention network, it is possible to obtain three levels of textual representations (i.e., action description representations at multiple semantic levels, each responsible for capturing control information of the overall motion, control information of local actions, and control information of action details, respectively).

[0189] In a specific application, a graph attention network can use a graph attention mechanism to update the node representation of each semantic node in a hierarchical semantic graph. After obtaining the updated node representation of each semantic node, the server uses the updated node representation of each semantic node as the second feature vector of each action description information, concatenates the second feature vectors of at least two action description information belonging to each semantic level, and uses the concatenated second feature vector as the action description representation of the semantic level to obtain the action description representations for each of the multiple semantic levels.

[0190] In the coarse-to-fine motion sequence generation module, the process of generating motion from text is broken down into three semantic levels, each responsible for capturing the overall motion, local motion, and the details of the motion, respectively.

[0191] First, during the training phase, the server constructs three levels of action encoders. That is, the server trains one action autoencoder for each of the three semantic levels, and achieves action representation learning in an encoding-decoding manner, and latent action representations at each semantic level.

[0192]

number

[0193]

number

[0194]

number

[0195]

number

[0196]

number

[0197]

number

[0198]

number

[0199]

number

[0200] After that, in the training stage, a hierarchical motion generation model that generates an operation sequence based on the diffusion model is further designed. Compared with other generative frameworks, the diffusion model is a generative model based on the thermodynamic random diffusion process. This process includes a forward process of sequentially adding noise to samples from the data distribution and a backward process of training a neural network to reverse the forward process by sequentially removing noise. In the forward process, the noise addition process in the latent space is

[0201] [Number] is defined as. Here,

[0202] [Number] represents the i-th semantic level of the latent motion representation at the t-th noise addition step,

[0203] [Number] represents the i-th semantic level of the latent motion representation at the (t - 1)-th noise addition step,

[0204] [Number] is a preset hyperparameter related to the noise addition step t and can be obtained based on the noise addition step t.

[0205] In this embodiment, in the training stage, noise reducers at three semantic levels

[0206] [Number] are connected in series. After the training is completed, the noise reducers at the three trained and serially connected semantic levels

[0207] [Number] By means of, from coarse to fine, the sampling noise signal for generating the virtual object motion and the motion description text for describing the virtual object motion, the motion potential code with the finest granularity, that is, the motion feature vector after cascade noise reduction through multiple semantic levels, can be obtained.

[0208]

[0209] [Number] ) are only used, and the coarse-grained motion feature vector Z , , , 11 ,

[0211] , , , 22 is generated. At the local motion level, as the conditions of the diffusion model, the features of the global motion nodes (that is, the motion description representation at the global motion level

[0210] [Number] ), the features of the local motion nodes (that is, the motion description representation at the local motion level

[0211] [Number] ), and Z 11 are jointly used, and the potential motion code Z 22is further generated. At the operation detail level, as conditions of the diffusion model, the features of all nodes in the hierarchical semantic graph (as shown in FIG. 14, the operation description expression at the overall motion level is

[0212]

Number

[0213]

Number

[0214]

Number

[0215]

Number

[0216] In one embodiment, Tables 1 and 2 show the quantitative experimental results of the present invention on the HumanML3D and KIT-ML datasets, respectively, and the optimal results among them are all from the present invention's method. In Tables 1 and 2, methods compared with the present invention's method include Real motion, Seq2Seq (sequence-to-sequence), Language2Pose (joint language posture), Text2Gesture (text-gesture), Hier (multi-hierarchical attention model), MoCoGAN (video generation model), Dance2Music (dance-music model), TM2T (human motion generation model), T2M (text-to-animation generation), MDM (human motion diffusion model), MLD (movement latency diffusion), and others.

[0217] Currently, five evaluation metrics are widely used in cross-modal generation tasks: R-Precision (reflecting the accuracy of text-motion matching during the search), FID (Frechet Inception Distance, a metric for calculating the distance between feature vectors of real and generated images), MM Dist (Multi-Modal Distance), Diversity (defined as the variance of motion feature vectors of generated motions across all text descriptions, reflecting the diversity of motions synthesized from a set of different descriptions), and MModality (measures the diversity of motions generated from each text description, reflecting the diversity of motions synthesized from a particular description).

[0218] Of these five quantization metrics, R-Precision, FID, and MM Dist primarily reflect the realism of the generated 3D human motion compared to actual motion. Diversity and MModality primarily reflect the degree of diversification of the generated 3D human motion. The results in Tables 1 and 2 demonstrate that the present invention achieves optimal performance in terms of realism and diversity of generated results, surpassing conventional methods, on two major datasets.

[0219] [Table 1]

[0220] [Table 2] The inventors believe that the present invention has two significant advantages compared to conventional methods. First, by explicitly decomposing and representing the semantic space, the present invention can establish a fine-grained correspondence between text data and motion sequences, avoiding unbalanced learning of different text components and coarse-grained representation of control signals. Second, the hierarchically subdivided motion sequence generation strengthens the generation results step by step from coarse to fine, avoiding the generation results being too coarse, ensuring model generation quality, and improving the versatility of the results.

[0221] In one embodiment, in order to further fine-tune the generated virtual object behavior and achieve finer granular control, the configuration of the present invention may continuously improve the generated virtual object behavior by modifying the edge weights of the hierarchical semantic graph, thereby generating virtual object behavior that better suits the needs.

[0222] Specifically, when the virtual object behavior is acquired, the server responds to edge weight adjustment events of the connecting edges that connect each semantic node in the hierarchical semantic graph by adjusting the edge weights of the connecting edges indicated by the edge weight adjustment events to acquire an updated hierarchical semantic graph, updates the node representation of each semantic node in the updated hierarchical semantic graph using the graph attention mechanism, acquires the third feature vector of each behavior description information based on the updated node representation of each semantic node, concatenates the third feature vectors of at least two behavior description information belonging to each semantic level, obtains updated behavior description representations for multiple semantic levels using the concatenated third feature vector as the updated behavior description representation of the semantic level, and generates the adjusted virtual object behavior based on the updated behavior description representations for multiple semantic levels.

[0223] It should be understood that, although the steps in the flowcharts for each embodiment described above are shown sequentially according to the arrows, these steps are not necessarily performed sequentially in the order indicated by the arrows. Unless explicitly stated otherwise in this specification, there are no strict restrictions on the order in which these steps are performed. These steps may be performed in other orders. Furthermore, at least some of the steps in the flowcharts for each embodiment described above may include multiple steps or stages. These steps or stages do not necessarily have to be completed at the same time, but may be performed at different times. The order in which these steps or stages are performed also does not necessarily have to be sequential, but may be performed alternately with or in exchange for other steps, or at least some of the steps or stages in other steps.

[0224] Based on a similar inventive concept, embodiments of the present application further provide a virtual object motion generation device for realizing the virtual object motion generation method described above. Since the implementation configuration for solving the problems provided by this device is similar to the implementation configuration described in the above method, specific limitations in the embodiments of one or more virtual object motion generation devices provided below should be referred to with respect to the limitations of the virtual object motion generation method described above. Further explanation is omitted here.

[0225] In one embodiment, as shown in Figure 14, a virtual object motion generation device is provided that includes an acquisition module 1402, a semantic analysis module 1404, an encoding module 1406, a first noise reduction processing module 1408, a second noise reduction processing module 1410, and a decoding module 1412. The acquisition module 1402 acquires the behavior description text for describing the behavior of a virtual object.

[0226] The semantic analysis module 1404 analyzes the action description text at a predetermined number of semantic levels through semantic analysis, thereby obtaining action description information for each of the multiple semantic levels and acquiring a sampling noise signal for generating virtual object actions.

[0227] The encoding module 1406 obtains the operation description representation for each of the multiple semantic levels by encoding the operation description information for each of the multiple semantic levels.

[0228] The first noise reduction processing module 1408 obtains the operation feature vector output from the first semantic level by performing noise reduction processing on the sampled noise signal based on the operation description representation of the first semantic level among multiple semantic levels.

[0229] The second noise reduction processing module 1410 performs noise reduction processing on the sampled noise signal for each semantic level from the first semantic level onward among multiple semantic levels, based on the operation feature vector output from the previous semantic level and the operation description representations of at least two semantic levels from the first semantic level to the current semantic level. This process obtains an operation feature vector after cascading noise reduction through multiple semantic levels, and the operation granularity level represented by the operation feature vector output from the noise reduction processing of multiple semantic levels is reduced in order from the highest semantic level to the lowest semantic level.

[0230] The decoding module 1412 obtains the virtual object behavior by decoding the behavior feature vector after cascaded noise reduction.

[0231] In the virtual object motion generation device described above, motion description text for describing virtual object motion is acquired, and motion description information for each of the multiple semantic levels is acquired by analyzing the motion description text at predetermined multiple semantic levels through semantic analysis, a sampling noise signal for generating virtual object motion is acquired, and motion description representations for each of the multiple semantic levels are acquired by encoding the motion description information for each of the multiple semantic levels, and a motion feature vector output from the first semantic level is acquired by performing noise reduction processing on the sampling noise signal based on the motion description representation of the first semantic level among the multiple semantic levels, and a motion feature vector output from the first semantic level is acquired by performing noise reduction processing on the sampling noise signal for each semantic level after the first semantic level, using the motion feature vector output from the previous semantic level and the motion description representations for at least two semantic levels from the first semantic level to the current semantic level as common conditions, thereby using the motion description representations for each of the multiple semantic levels to successively enrich the finer granularity of motion details and acquire a motion feature vector after cascade noise reduction that has passed through multiple semantic levels and accurately represents the virtual object motion at a finer granularity, and furthermore, the virtual object motion can be acquired by decoding the motion feature vector after cascade noise reduction. The overall process involves using the behavioral description information from multiple semantic levels as fine-grained control signals, capturing the behavioral characteristics of multiple semantic levels to subdivide and generate virtual object behavior, thereby improving the accuracy of the generated virtual object behavior.

[0232] In one embodiment, the multiple semantic levels include a global motion level, a local motion level, and a motion detail level. The semantic analysis module further uses motion description text as global motion level motion description information, extracts at least one verb and a phrase for modifying each of the at least one verb from the motion description text, uses the at least one verb as local motion level motion description information, and uses the phrase for modifying each of the at least one verb as motion detail level motion description information.

[0233] In one embodiment, the encoding module further obtains a first feature vector for each action description by encoding each action description information of each of the multiple semantic levels, and obtains a second feature vector for each action description by performing an update process based on an attention mechanism on the first feature vector of each action description based on the semantic relationship between action description information of at least one pair of different semantic levels, and for each semantic level, it concatenates the second feature vectors of at least two action description information belonging to that semantic level, and uses the concatenated second feature vector as the action description representation of the semantic level, thereby obtaining the action description representations for each of the multiple semantic levels.

[0234] In one embodiment, the encoding module further uses each action description information as a semantic node, connects two semantic nodes representing a pair of action description information with a semantic relationship based on the semantic relationship between action description information at least one pair of different semantic levels, determines the connecting edge connecting each semantic node, uses the first feature vector of each action description information as the node representation of each semantic node, constructs a hierarchical semantic graph based on each semantic node, the connecting edge connecting each semantic node, and the node representation of each semantic node, updates the node representation of each semantic node in the hierarchical semantic graph using a graph attention mechanism, and obtains the second feature vector of each action description information based on the updated node representation of each semantic node.

[0235] In one embodiment, the encoding module further determines at least one adjacent node of the target semantic node for each semantic node in the hierarchical semantic graph, performs interaction processing based on a graph attention mechanism on the node representation of at least one adjacent node and the node representation of the target semantic node, determines attention weight coefficients for at least one adjacent node and the target semantic node, and obtains the updated node representation of the target semantic node by weighted addition of the node representation of at least one adjacent node and the node representation of the target semantic node based on the attention weight coefficients.

[0236] In one embodiment, the virtual object motion generation device further includes an adjustment module. When the adjustment module acquires virtual object motion, it acquires an updated hierarchical semantic graph by adjusting the edge weights of the connecting edges indicated by the edge weight adjustment events in response to edge weight adjustment events of the connecting edges that connect each semantic node in the hierarchical semantic graph, updates the node representation of each semantic node in the updated hierarchical semantic graph using a graph attention mechanism, acquires a third feature vector of each motion description information based on the updated node representation of each semantic node, concatenates the third feature vectors of at least two motion description information belonging to each semantic level, and uses the concatenated third feature vector as the updated motion description representation of the semantic level to acquire updated motion description representations for each of the multiple semantic levels, and generates an adjusted virtual object motion based on each of the updated motion description representations for the multiple semantic levels.

[0237] In one embodiment, the first noise reduction processing module further processes the sampled noise signal into a noise signal that has undergone multi-step noise addition, and from the last step of the multi-step noise addition process, performs reverse noise reduction processing on the input noise signal of each step based on the operation description representation of the first semantic level among multiple semantic levels, and uses the noise-reduced signal obtained by processing the input noise signal of the first step as the operation feature vector output from the first semantic level.

[0238] In one embodiment, the first noise reduction processing module obtains noise addition step features by encoding the step number of the target noise addition step, obtains noise reduction condition features by fusing the initial semantic level operation description representation with the noise addition step features, and obtains a noise reduction signal by performing noise reduction processing on the input noise signal of the target noise addition step based on the noise reduction condition features.

[0239] In one embodiment, the first noise reduction processing module obtains a first predicted additional noise corresponding to the target noise addition step by predicting the additional noise corresponding to the target noise addition step based on the noise reduction condition features and the input noise signal of the target noise addition step, and obtains a noise reduction signal by subtracting the first predicted additional noise from the input noise signal of the target noise addition step and performing noise reduction processing.

[0240] In one embodiment, the virtual object's behavior is determined by a pre-trained behavior sequence generation model, which includes a cascaded noise reduction network and a decoder. The cascaded noise reduction network obtains a behavior feature vector after cascaded noise reduction across multiple semantic levels by performing noise reduction processing on each of the multiple semantic levels. The decoder obtains the virtual object's behavior by decoding the behavior feature vector after cascaded noise reduction.

[0241] In one embodiment, the virtual object motion generation device further includes a training module. The training module acquires a plurality of training samples and, for each of the plurality of training samples, acquires a cascaded noise reduction network by training an initial noise reduction network based on the sample description text and motion sequence in the target training sample.

[0242] In one embodiment, the training module further obtains sample description information for each of the multiple semantic levels by analyzing the sample description text in the target training sample at multiple semantic levels through semantic analysis, obtains sample description representations for each of the multiple semantic levels by encoding the sample description information for each of the multiple semantic levels, and obtains a cascaded noise reduction network by training an initial noise reduction network based on the sample description representations for each of the multiple semantic levels and the operation sequence in the target training sample.

[0243] In one embodiment, the training module further obtains latent action representations corresponding to each of several semantic levels by performing action coding at multiple coding levels on the action sequences in the target training samples, and obtains a cascaded noise reduction network by training an initial noise reduction network based on the sample description representations for each of the multiple semantic levels and the latent action representations corresponding to each of the multiple semantic levels.

[0244] In one embodiment, there is a one-to-one correspondence between multiple coding levels and multiple semantic levels, and the coding dimension of each coding level increases sequentially from the first coding level to the last coding level. The training module further obtains the motion latent space features of each of the multiple coding levels by performing motion coding of each of the multiple coding levels for the motion sequence in the target training sample, and obtains the latent motion representation corresponding to each of the multiple semantic levels by decoding each of the motion latent space features of each of the multiple coding levels.

[0245] In one embodiment, the initial noise reduction network includes a cascaded set of initial noise reducers, each corresponding to one semantic level. The training module further obtains a trained noise reducer by training a target initial noise reducer for each of the set of initial noise reducers based on sample description representations for at least two semantic levels, from the first semantic level to the target semantic level corresponding to the target initial noise reducer, and a latent operation representation corresponding to the target semantic level, and then obtains a cascaded noise reduction network based on the trained noise reducers corresponding to each of the set of initial noise reducers.

[0246] In one embodiment, the training module further obtains a noise addition step number for noise addition, samples a random noise signal, and adds the random noise signal to a latent operating representation corresponding to a target semantic level based on the noise addition step number to obtain a noise operating representation. The training module inputs the noise operating representation, the noise addition step number, and sample description representations for at least two semantic levels from the first semantic level to the target semantic level corresponding to the target initial noise reducer into the target initial noise reducer, and obtains a second predicted additional noise by predicting the added noise using the target initial noise reducer. A trained noise reducer is then obtained by adjusting the parameters of the target initial noise reducer based on the second predicted additional noise.

[0247] In one embodiment, the training module further inputs the noise operation representation, the noise addition step number, the sample description representations for at least two semantic levels from the first semantic level to the target semantic level corresponding to the target initial noise reducer, and the reconstructed operation representation output from the previous noise reducer to the target initial noise reducer, and obtains a second predicted additional noise by predicting the added noise using the target initial noise reducer.

[0248] Each module of the virtual object behavior generator described above may be implemented in whole or in part by software, hardware, or a combination thereof. Each module may be embedded in the processor of the computer device in hardware form, independently, or stored in the memory of the computer device in software form. This facilitates the processor to call and execute the behavior corresponding to each module.

[0249] In one embodiment, a computer device is provided. This computer device may be a server or a terminal. Taking the case where the computer device is a server as an example, its internal configuration diagram may be as shown in Figure 15. The computer device includes a processor, memory, an input / output interface (I / O), and a communication interface. Here, the processor, memory, and I / O interface are connected via a system bus, and the communication interface is connected to the system bus via the I / O interface. Here, the processor of the computer device provides computation and control functions. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the execution of the operating system and computer programs on the non-volatile storage medium. The database of the computer device is for storing data such as training samples. The I / O interface of the computer device is for exchanging information between the processor and external devices. The communication interface of the computer device is for communicating with external terminals via a network connection. When executed by the processor, this computer program implements a method for generating virtual object behavior.

[0250] As those skilled in the art will understand, the configuration shown in Figure 15 is merely a block diagram of some of the configurations relating to the present invention and does not limit the computer equipment to which the present invention applies. Specific computer equipment may include more or fewer components than those shown, or may be a combination of some components, or may have different arrangements of components.

[0251] In one embodiment, a computer device comprising memory and a processor is further provided. The memory stores a computer program, and the processor executes the computer program to perform the steps in each of the above-described embodiment of the method.

[0252] In one embodiment, a computer-readable storage medium storing a computer program is provided, and when the computer program is executed by a processor, it realizes the steps in each of the above-described embodiment of the method.

[0253] In one embodiment, a computer program product is provided which, when executed by a processor, realizes the steps in each of the above-described method embodiments.

[0254] As those skilled in the art will understand, all or part of the steps of the methods according to the above embodiments may be executed by instructing the relevant hardware via a computer program. The computer program may be stored in a non-volatile computer-readable storage medium. When the computer program is executed, the steps of each embodiment of the above methods are executed. Here, any reference to memory, database, or other medium used in each embodiment provided herein may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache, etc. For illustrative purposes only, rather than being an limitation, RAM may take many forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases relating to each embodiment provided herein may include at least one of relational databases and non-relational databases. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors relating to each embodiment provided herein may include, but are not limited to, general-purpose processors, central processing units, graphics processors, digital signal processors, programmable logic units, quantum computing-based data processing logic units, and the like.

[0255] The constituent elements of the embodiments described above may be combined in any way. For the sake of brevity, not all possible combinations of the constituent elements of the embodiments described above are described. However, these combinations of constituent elements should be considered to fall within the scope described herein, as long as they do not contradict each other.

[0256] The embodiments described above are merely examples of some embodiments of the present application, and although the descriptions are specific and detailed, they should not be understood as limiting the scope of the patent. It should be noted that, for those skilled in the art, some modifications and improvements are possible without departing from the concept of the present application. All of these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application should be in accordance with the appended claims.

Claims

1. A method for generating virtual object behaviors executed by a computer device, Steps include obtaining behavioral description text to describe the behavior of a virtual object, The steps include: obtaining action description information for each of the multiple semantic levels by analyzing the action description text at predetermined multiple semantic levels through semantic analysis, and obtaining a sampling noise signal for generating the virtual object action; The steps include: obtaining an action description representation for each of the multiple semantic levels by encoding the action description information for each of the multiple semantic levels; The steps include: obtaining an operation feature vector output from the first semantic level by performing noise reduction processing on the sampling noise signal based on the operation description representation of the first semantic level among the plurality of semantic levels; A step to obtain a cascaded noise-reduced operation feature vector that has passed through the multiple semantic levels by performing noise reduction processing on the sampling noise signal based on the operation feature vector output from the previous semantic level and the operation description representations of at least two semantic levels from the first semantic level to the current semantic level, wherein the operation granularity level represented by the operation feature vector output from the noise reduction processing of the multiple semantic levels decreases in order from the highest semantic level to the lowest semantic level. The steps include obtaining the virtual object behavior by decoding the operational feature vector after cascading noise reduction, A method for generating virtual object behavior, characterized by the following features.

2. The aforementioned multiple semantic levels include a global motion level, a local motion level, and a motion detail level, and by semantic analysis, motion description information for each of the aforementioned multiple semantic levels is obtained by analyzing the motion description text at a predetermined multiple semantic levels. The aforementioned step is: The steps include: using the aforementioned motion description text as the overall motion-level motion description information; and extracting from the aforementioned motion description text at least one verb and a phrase for modifying the verb that corresponds to each of the at least one verb; The steps include: using the at least one verb as the local action level action description information, and using a phrase for modifying each of the at least one verbs as the action detail level action description information; The method for generating virtual object behavior according to feature 1.

3. The step of obtaining an action description representation for each of the multiple semantic levels by encoding the action description information for each of the multiple semantic levels is as follows: The steps include: obtaining a first feature vector for each of the operation description pieces by encoding each of the operation description pieces for each of the multiple operation description pieces; The steps include obtaining a second feature vector for each of the operation description information by performing an update process based on an attention mechanism on the first feature vector of each of the operation description information based on the semantic relationship between operation description information at least one pair of different semantic levels, The process includes the step of obtaining an action description representation for each of the multiple semantic levels by concatenating the second feature vectors of at least two action description information belonging to each semantic level, and using the concatenated second feature vector as the action description representation for the semantic level. The method for generating virtual object behavior according to feature 1.

4. The step of obtaining a second feature vector for each of the action description information by performing an update process based on an attention mechanism on the first feature vector of each of the action description information based on the semantic relationship between action description information at least one pair of different semantic levels is as follows: Each of the aforementioned action description information is treated as a semantic node, and based on the semantic relationship between at least one pair of action description information at different semantic levels, a step is to determine the connecting edge that connects each of the aforementioned semantic nodes by connecting two semantic nodes that represent a pair of action description information with a semantic relationship. The steps include: setting the first feature vector of each of the aforementioned operation description information as the node representation of each of the aforementioned semantic nodes; A step of constructing a hierarchical semantic graph based on each of the aforementioned semantic nodes, the connecting edges that connect each of the aforementioned semantic nodes, and the node representation of each of the aforementioned semantic nodes. The process includes the steps of: updating the node representation of each semantic node in the hierarchical semantic graph using a graph attention mechanism; and obtaining a second feature vector of each action description information based on the updated node representation of each semantic node. The method for generating virtual object behavior according to feature 3.

5. The step of updating the node representation of each of the semantic nodes in the hierarchical semantic graph using a graph attention mechanism is: The steps include determining at least one adjacent node of the target semantic node for each semantic node in the hierarchical semantic graph, The steps include performing interaction processing based on a graph attention mechanism on the node representation of at least one adjacent node and the node representation of the target semantic node to determine the attention weight coefficients of the at least one adjacent node and the target semantic node, The step of obtaining the updated node representation of the target semantic node by weighting and adding the node representation of the at least one adjacent node and the node representation of the target semantic node based on the attention weight coefficient, The method for generating virtual object behavior according to feature 4.

6. When the virtual object behavior is obtained, the step of obtaining an updated hierarchical semantic graph is to adjust the edge weights of the connecting edges indicated by the edge weight adjustment events in response to the edge weight adjustment events of the connecting edges that connect each of the semantic nodes in the hierarchical semantic graph, The steps include: updating the node representation of each semantic node in the updated hierarchical semantic graph using a graph attention mechanism, and obtaining a third feature vector of each action description information based on the updated node representation of each semantic node; The steps include: obtaining updated action description representations for each of the multiple semantic levels by concatenating the third feature vectors of at least two action description information belonging to each semantic level, and using the concatenated third feature vector as the updated action description representation for the semantic level; The step of generating adjusted virtual object behavior based on the updated behavioral description representations of each of the aforementioned multiple semantic levels, further includes: The method for generating virtual object behavior according to feature 4.

7. The step of obtaining an operation feature vector output from the first semantic level by performing noise reduction processing on the sampling noise signal based on the operation description representation of the first semantic level among the plurality of semantic levels is as follows: The sampling noise signal is converted into a noise signal after multi-step noise addition; from the last step of the multi-step noise addition, reverse noise reduction processing is performed on the input noise signal of each step based on the operation description expression of the first semantic level among the multiple semantic levels; and the noise-reduced signal obtained by processing the input noise signal of the first step is used as the operation feature vector output from the first semantic level. The method for generating virtual object behavior according to feature 1.

8. The step of performing noise reduction processing on the input noise signal of the target noise addition step, for each of the steps of adding noise in multiple steps, is as follows: The steps include: obtaining noise addition step features by encoding the step number of the target noise addition step; A step of obtaining noise reduction condition features by fusing the aforementioned initial semantic level action description expression and the noise addition step features, The process includes a step of obtaining a noise-reduced signal by performing noise reduction processing on the input noise signal of the target noise addition step based on the noise reduction condition characteristics. The method for generating virtual object behavior according to feature 7.

9. The step of obtaining a noise-reduced signal by performing noise reduction processing on the input noise signal of the target noise addition step based on the noise reduction condition characteristics is as follows: A step of obtaining a first predicted additional noise corresponding to the target noise addition step by predicting the additional noise corresponding to the target noise addition step based on the noise reduction condition characteristics and the input noise signal of the target noise addition step, The process includes a step of obtaining a noise-reduced signal by subtracting the first predicted additional noise from the input noise signal of the target noise addition step and performing noise reduction processing. The method for generating virtual object behavior according to feature 8.

10. The virtual object behavior is determined by a pre-trained behavior sequence generation model, which includes a cascaded noise reduction network and a decoder, wherein the cascaded noise reduction network obtains a behavior feature vector after cascaded noise reduction through the multiple semantic levels by performing noise reduction processing on each of the multiple semantic levels, and the decoder obtains the virtual object behavior by decoding the behavior feature vector after cascaded noise reduction. The method for generating virtual object behavior according to feature 1.

11. The aforementioned cascade noise reduction network was acquired through a training step, and the training step is Steps to obtain multiple training samples, The process includes the step of obtaining a cascaded noise reduction network by training an initial noise reduction network for each training sample among the plurality of training samples based on the sample description text and operation sequence in the target training sample. The method for generating virtual object behavior according to feature 10.

12. The step of obtaining a cascaded noise reduction network by training an initial noise reduction network based on the sample description text and operation sequence in the target training sample is as follows: The steps include: obtaining sample description information for each of the multiple semantic levels by analyzing the sample description text in the target training sample at the multiple semantic levels through semantic analysis; The steps include: obtaining a sample description representation for each of the multiple semantic levels by encoding the sample description information for each of the multiple semantic levels; The steps include: obtaining a cascaded noise reduction network by training the initial noise reduction network based on the sample description representations of each of the multiple semantic levels and the operation sequence in the target training sample; The method for generating virtual object behavior according to feature 11.

13. The step of obtaining a cascaded noise reduction network by training the initial noise reduction network based on the sample description representations of each of the multiple semantic levels and the operation sequence in the target training sample is as follows: The steps include: obtaining latent action representations corresponding to each of the multiple semantic levels by performing action coding at multiple coding levels on each of the action sequences in the target training sample; The step of obtaining a cascaded noise reduction network by training the initial noise reduction network based on the sample description representations for each of the multiple semantic levels and the latent operation representations corresponding to each of the multiple semantic levels, includes the step of obtaining a cascaded noise reduction network. The method for generating virtual object behavior according to feature 12.

14. The step of obtaining a latent action representation corresponding to each of the multiple semantic levels is as follows: The multiple coding levels correspond one-to-one with the multiple semantic levels, the coding dimension of each coding level among the multiple coding levels increases sequentially from the first coding level to the last coding level, and the action coding of each of the multiple coding levels is performed on the action sequence in the target training sample. The steps include: obtaining the motion latent space features of each of the multiple coding levels by performing motion coding at multiple coding levels on the motion sequence in the target training sample; The process includes the step of obtaining a latent action representation corresponding to each of the multiple semantic levels by decoding the motion latent space features of each of the multiple coding levels. The method for generating virtual object behavior according to feature 13.

15. The aforementioned initial noise reduction network includes a cascaded set of initial noise reducers, each of which corresponds to one semantic level. The step of obtaining a cascaded noise reduction network by training the initial noise reduction network based on the sample description representations for each of the multiple semantic levels and the latent operation representations corresponding to each of the multiple semantic levels is as follows: A step of obtaining a trained noise reducer by training the target initial noise reducer for each of the plurality of initial noise reducers based on sample description representations for at least two semantic levels from the first semantic level to the target semantic level corresponding to the target initial noise reducer, and a latent operation representation corresponding to the target semantic level. The steps include: obtaining a cascaded noise reduction network based on trained noise reducers corresponding to each of the plurality of initial noise reducers; The method for generating virtual object behavior according to feature 13.

16. The step of obtaining a trained noise reducer by training the target initial noise reducer for each of the plurality of initial noise reducers based on sample description representations for at least two semantic levels from the first semantic level to the target semantic level corresponding to the target initial noise reducer, and a latent operation representation corresponding to the target semantic level, The steps include obtaining a noise addition step number for noise addition and sampling a random noise signal, A step of obtaining a noise operation representation by adding the random noise signal to the latent operation representation corresponding to the target semantic level based on the noise addition step number, The steps include inputting the noise operation representation, the noise addition step number, and the sample description representations for at least two semantic levels from the first semantic level to the target semantic level corresponding to the target initial noise reducer into the target initial noise reducer, and obtaining a second predicted additional noise by predicting the added noise using the target initial noise reducer; The step of obtaining a trained noise reducer by adjusting the parameters of the target initial noise reducer based on the second predicted additional noise, The method for generating virtual object behavior according to feature 15.

17. The step of inputting the noise operation representation, the noise addition step number, and the sample description representations for at least two semantic levels from the first semantic level to the target semantic level corresponding to the target initial noise reducer into the target initial noise reducer, and obtaining a second predicted additional noise by predicting the added noise using the target initial noise reducer, is as follows: If a preceding noise reducer is present in series with the target initial noise reducer, the process includes inputting the noise operation expression, the noise addition step number, the sample description expressions for at least two semantic levels from the initial semantic level to the target semantic level corresponding to the target initial noise reducer, and the reconstruction operation expression output from the preceding noise reducer to the target initial noise reducer, and obtaining a second predicted additional noise by predicting the added noise using the target initial noise reducer. The method for generating virtual object behavior according to feature 16.

18. A virtual object behavior generation device, A module for obtaining behavioral description text to describe the behavior of virtual objects, A semantic analysis module that analyzes the operation description text at a predetermined number of semantic levels through semantic analysis, obtains operation description information for each of the multiple semantic levels, and obtains a sampling noise signal for generating the virtual object operation, An encoding module that obtains an action description representation for each of the multiple semantic levels by encoding the action description information for each of the multiple semantic levels, A first noise reduction processing module obtains an operation feature vector output from the first semantic level by performing noise reduction processing on the sampling noise signal based on the operation description representation of the first semantic level among the plurality of semantic levels, A second noise reduction processing module obtains an operation feature vector after cascaded noise reduction through the multiple semantic levels by performing noise reduction processing on the sampling noise signal based on the operation feature vector output from the previous semantic level and the operation description representations of at least two semantic levels from the first semantic level to the current semantic level, wherein the operation granularity level represented by the operation feature vector output in the noise reduction processing of the multiple semantic levels decreases in order from the highest semantic level to the lowest semantic level, A decoding module that obtains the virtual object behavior by decoding the operation feature vector after cascade noise reduction, is included. A virtual object motion generation device characterized by the following features.

19. A computer program that causes a computer to execute the virtual object operation generation method described in any one of claims 1 to 17.

20. A computer device comprising memory and a processor, wherein a computer program is stored in the memory, and the processor, upon executing the computer program, realizes the virtual object operation generation method described in any one of claims 1 to 17.

Images