Large model training method for discriminating and generating action sequence based on text voice correlation

Abstract

The invention provides a large model training method for discriminating and generating an action sequence based on text and voice correlation, and the method comprises the steps: obtaining a target semantic feature which is generated by fusing a text semantic feature and a voice semantic feature corresponding to the same text content; inputting the target semantic feature into a multi-modal fusion feature generation model, and outputting a corresponding action sequence; calculating a loss function value of the multi-modal fusion feature generation model based on the target semantic feature and the action sequence; and adjusting a model parameter value of the multi-modal fusion feature generation model based on the loss function value, continuing training until a preset training completion condition is met, and obtaining a trained multi-modal fusion feature generation model. According to the embodiment of the invention, the semantic feature generated by fusing the text semantic feature and the voice semantic feature corresponding to the same text content is utilized to guide the generation of the human body speaking action, so that the natural, smooth and reasonable human body animation is output.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and specifically relates to a large model training method for generating action sequences based on text-speech correlation discrimination. Background Art

[0002] Human motion animation plays an important role in 3D film and television production. Natural and smooth human motion animation sequences require experienced animators to spend a lot of time and rely on professional software to adjust frame by frame, and the production threshold is relatively high. With the development of the 3D film industry and the prosperity of the short video industry, the demand for animation production is increasing. Generative technology based on artificial intelligence can greatly improve the output efficiency of professionals, reduce the cost of getting started for non-professionals, and lower the threshold for human animation production. At present, human motion generation technology is still in the development stage, and the generated human motion has problems of unnatural and uncoordinated limbs. Summary of the invention

[0003] The present application proposes a large model training method for generating action sequences based on text-speech correlation discrimination, which can solve the technical problem that human body movements generated by human body movement generation technology in related technologies have unnatural and uncoordinated limbs.

[0004] The first embodiment of the present application proposes a large model training method for generating action sequences based on text-speech correlation discrimination, including:

[0005] Acquire a target semantic feature, where the target semantic feature is generated by fusing a text semantic feature and a speech semantic feature corresponding to the same text content;

[0006] Inputting the target semantic features into a multimodal fusion feature generation model and outputting a corresponding action sequence;

[0007] Based on the target semantic features and the action sequence, calculating the loss function value of the multimodal fusion feature generation model;

[0008] The model parameter values ​​of the multimodal fusion feature generation model are adjusted based on the loss function value, and the training is continued until a preset training completion condition is met to obtain a trained multimodal fusion feature generation model.

[0009] The embodiment of the second aspect of the present application provides an action sequence generation method, which is applied to the multimodal fusion feature generation model in the distributed system of the first aspect, and the method includes:

[0010] The target semantic features generated by fusing the text semantic features and the speech semantic features corresponding to the same text content are input into the multimodal fusion feature generation model, and the corresponding action sequence is output.

[0011] The embodiment of the third aspect of the present application provides a large model training device for generating action sequences based on text-speech correlation discrimination, including:

[0012] An acquisition module, used to acquire a target semantic feature, wherein the target semantic feature is generated by fusing a text semantic feature and a speech semantic feature corresponding to the same text content;

[0013] An input module, used to input the target semantic features into a multimodal fusion feature generation model and output a corresponding action sequence;

[0014] A calculation module, used for calculating the loss function value of the multimodal fusion feature generation model based on the target semantic feature and the action sequence;

[0015] An adjustment module is used to adjust the model parameter values ​​of the multimodal fusion feature generation model based on the loss function value, and continue training until a preset training completion condition is met to obtain a trained multimodal fusion feature generation model.

[0016] The embodiment of the fourth aspect of the present application provides an action sequence generation device, which is applied to the multimodal fusion feature generation model described in the first aspect, and the device includes:

[0017] The input module is used to input the target semantic features generated by fusing the text semantic features and speech semantic features corresponding to the same text content into the multimodal fusion feature generation model, and output the corresponding action sequence.

[0018] An embodiment of the fifth aspect of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect or the second aspect above.

[0019] The embodiment of the sixth aspect of the present application provides a computer-readable storage medium on which a computer program is stored, and the program is executed by a processor to implement the method described in the first aspect or the second aspect above.

[0020] The technical solution provided in the embodiments of the present application has at least the following technical effects or advantages:

[0021] The present application proposes a large model training method for generating action sequences based on text-speech correlation discrimination, including: obtaining target semantic features, which are generated by fusion of text semantic features and speech semantic features corresponding to the same text content; inputting the target semantic features into a multimodal fusion feature generation model, and outputting the corresponding action sequence; calculating the loss function value of the multimodal fusion feature generation model based on the target semantic features and the action sequence; adjusting the model parameter values ​​of the multimodal fusion feature generation model based on the loss function value, and continuing training until the preset training completion conditions are met to obtain a trained multimodal fusion feature generation model. The embodiment of the present application guides the generation of human speaking actions by using semantic features generated by fusion of text semantic features and speech semantic features corresponding to the same text content, thereby outputting natural, smooth and reasonable human animations.

[0022] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] By reading the detailed description of the preferred embodiment below, various other advantages and benefits will become clear to those of ordinary skill in the art. The accompanying drawings are only used for the purpose of illustrating the preferred embodiment and are not considered to be limitations of the present application. In addition, the same reference symbols are used to represent the same components throughout the accompanying drawings.

[0024] In the attached picture:

[0025] Figure 1 A flow chart of a large model training method for generating action sequences based on text-speech correlation discrimination provided by an embodiment of the present application is shown;

[0026] Figure 2 A schematic diagram of the structure of an encoder provided by an embodiment of the present application is shown;

[0027] Figure 3 A schematic diagram of the structure of a semantic correlation fuser provided in one embodiment of the present application is shown;

[0028] Figure 4 A schematic diagram of the structure of a target text large model provided by an embodiment of the present application is shown;

[0029] Figure 5 A schematic diagram of the structure of a target speech large model provided by an embodiment of the present application is shown;

[0030] Figure 6 A flow chart of a large model training method for generating action sequences based on text-speech correlation discrimination provided by an embodiment of the present application is shown;

[0031] Figure 7 A schematic diagram of a flow chart of an action sequence generation method provided by an embodiment of the present application is shown;

[0032] Figure 8 A schematic diagram showing a flow chart of another method for generating an action sequence provided by an embodiment of the present application;

[0033] Fig. 9 A schematic diagram of the structure of a large model training device for generating action sequences based on text-speech correlation discrimination provided by an embodiment of the present application is shown;

[0034] Fig.10 A schematic diagram of the structure of an action sequence generating device provided by an embodiment of the present application is shown;

[0035] Fig.11 A schematic diagram of the structure of an electronic device provided by an embodiment of the present application is shown;

[0036] Fig.12 A schematic diagram of a storage medium provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0037] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0038] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in this application should have the common meanings understood by technicians in the field to which this application belongs.

[0039] The following describes a large model training method for generating action sequences based on text-speech correlation discrimination according to an embodiment of the present application in conjunction with the accompanying drawings.

[0040] like Figure 1 As shown, the method includes the following steps.

[0041] S101, obtaining target semantic features.

[0042] The target semantic features are generated by fusing the text semantic features and speech semantic features corresponding to the same text content.

[0043] It is understandable that the generation of human movements often has specific requirements, which usually include an important conditional signal. This conditional signal can be varied, such as a vivid text description, which can provide a specific context and direction for the generation of the movement through the power of words; or an infectious background voice, the rhythm and melody of the music can inspire the inspiration and rhythm of the movement; these will play a guiding role in the generation of the movement. Therefore, the generated movement should not only be reasonable in itself, and be able to conform to the laws and logic of human movement in its own form of expression, but also be coordinated with the corresponding conditional signal. Only in this way can the movement be better integrated into the specific situation and play the greatest effect.

[0044] In order to improve the accuracy of the target semantic features, the voice information corresponding to the character when displaying human body movements and the text content corresponding to the voice information can be obtained at the same time. The voice information and text content are further output to the corresponding models for feature extraction to obtain the voice semantic features corresponding to the voice information and the text semantic features corresponding to the text content. The voice semantic features and the text semantic features are further fused to obtain the target semantic features.

[0045] Compared with a single semantic feature, this combination of multiple semantic features has a more comprehensive auxiliary guidance capability. The text content can convey rich information such as specific action requirements, situation settings, and emotional tendencies in clear and accurate language. For example, a vivid text description can depict a fierce battle scene, clearly pointing out the attacking actions, defensive postures, and nervous facial expressions that the characters need to make.

[0046] Voice information provides strong support for the generation of human movements in its own unique way. Rhythmic voice instructions can accurately grasp the rhythm and strength of the movements like a professional coach. Different tones, speeds and timbres can convey different emotional atmospheres and urgency, thereby guiding the performance of human movements to better fit specific situations. For example, in a tense sports game, passionate voice shouts can inspire athletes to make more decisive and powerful movements.

[0047] Since the multimodal input of text description and voice guidance can provide information from both language and sound levels, the information is more complete. This complete information input makes the generated human action sequence more consistent with expectations in all aspects.

[0048] S102: Input the target semantic features into the multimodal fusion feature generation model and output the corresponding action sequence.

[0049] Among them, the multimodal fusion feature generation model consists of an encoder and a decoder.

[0050] In some embodiments, the target semantic feature is input into a multimodal fusion feature generation model, and the corresponding action sequence is output, including: using the encoder of the multimodal fusion feature generation model to encode the target semantic feature to obtain the encoded semantic feature; using the decoder of the multimodal fusion feature generation model to determine the action of each time step corresponding to the encoded semantic feature; based on the action of each time step, combining and generating an action sequence corresponding to the encoded semantic feature.

[0051] Among them, the encoder encodes the target semantic features and extracts representative features, that is, encodes semantic features, so as to improve the expressive power of the model and reduce the amount of data of semantic features.

[0052] Furthermore, the decoder decodes the encoded semantic features to obtain the target semantic features, thereby determining the action corresponding to the current time step in the semantic features. The generated action will be fed back to the decoder as a new input, and the above steps are repeated until a complete action sequence is generated.

[0053] In some embodiments, the present application provides a structure of an encoder, such as Figure 2 As shown, the structure includes an input layer, multiple hidden layers, and an output layer.

[0054] The input layer is used to receive the target semantic feature, which may be a word embedding vector.

[0055] The plurality of hidden layers include a first hidden layer and a second hidden layer.

[0056] The parameters of the first hidden layer are input dimension fusion_dim and output dimension hidden_dim_1. The LeakyReLU activation function is used with a slope of 0.2.

[0057] The input fusion_dim dimension data is mapped to the hidden_dim_1 dimension through a linear transformation. This process usually involves the product and addition of the weight matrix and the bias vector, which can capture the linear combination of the input features.

[0058] Using the LeakyReLU activation function can introduce nonlinearity, allowing the model to learn more complex functional relationships. Unlike traditional ReLU, LeakyReLU still allows a small gradient (slope of 0.2) when the input is less than zero, which can help the model maintain information flow during training.

[0059] The parameters of the second hidden layer are input dimension hidden_dim_1 and output dimension hidden_dim_2. The ReLU activation function is used.

[0060] Through linear transformation, the input hidden_dim_1 dimension data is mapped to the hidden_dim_2 dimension.

[0061] The ReLU (Rectified Linear Unit) activation function is defined as (f(x) = \max(0,x)), which changes all negative values ​​to zero and keeps positive values ​​unchanged. This simple nonlinear change enables the model to learn more complex features and functional relationships.

[0062] The parameters of the output layer are input dimension hidden_dim_2 and output dimension encoding_dim, which is the final encoding dimension. That is, through linear transformation, the input hidden_dim_2 dimension data is mapped to the encoding_dim dimension, thereby generating the encoding semantic features of the encoding_dim dimension.

[0063] The structure of the decoder is similar to that of the encoder, but it needs to use appropriate parameter settings to target semantic features to determine the action corresponding to the current time step in the semantic features. The generated action will be fed back to the decoder as a new input, and the above steps are repeated until a complete action sequence is generated.

[0064] In some embodiments, a decoder of a multimodal fusion feature generation model is used to determine the action of each time step corresponding to the encoded semantic feature, including: for each time step, the decoder generates a probability distribution of all actions in the current time step based on the internal state and the action of the previous time step; and selects the target action from the probability distribution based on a preset selection strategy.

[0065] Among them, the preset selection strategy can be a greedy strategy, a random strategy, a game strategy, etc.

[0066] During the decoding process, the decoder initializes its internal state, usually using a special start marker (such as <start>) to indicate the start of the generated sequence. The decoder can be a recurrent neural network (RNN) or other sequence models (such as LSTM, GRU or Transformer). At each time step, the decoder generates the next action based on the current internal state and the previously generated output (such as the previous action). At each time step, the decoder also uses the target semantic features to focus on the important features related to the current generated action through the attention mechanism.

[0067] Further through the softmax layer, the decoder will calculate the probability distribution of the next action, and then determine the target action through a preset selection strategy based on this distribution.

[0068] S103. Based on the target semantic features and action sequence, calculate the loss function value of the multimodal fusion feature generation model.

[0069] In some embodiments, based on the target semantic features and action sequences, the loss function value of the multimodal fusion feature generation model is calculated, including: calculating the correlation score between the target semantic features and the action sequence; based on the correlation score and the preset correlation, calculating the loss function value of the multimodal fusion feature generation model.

[0070] Among them, the target semantic features and action sequences can be input into the semantic relevance discriminator to obtain the semantic relevance score and calculate the relevance loss. The network parameters are gradually corrected through back propagation to make the generated results more consistent with the input text and speech.

[0071] The semantic relevance discriminator adopts a multi-layer perceptron structure, receives the fusion features of text and speech and the corresponding generated action sequence as input, and finally outputs a scalar value through a combination of a series of linear layers and activation functions, indicating the degree of relevance of the input text, speech and action sequence.

[0072] S104. Adjust the model parameter values ​​of the multimodal fusion feature generation model based on the loss function value, and continue training until the preset training completion condition is met to obtain a trained multimodal fusion feature generation model.

[0073] Among them, the training completion condition may be that the loss function value is less than the loss function value threshold, or the number of training iterations is greater than the preset number of training iterations. The loss function value threshold and the preset number of training iterations can be flexibly set based on actual conditions.

[0074] The present application proposes a large model training method for generating action sequences based on text-speech correlation discrimination, including: obtaining target semantic features, which are generated by fusion of text semantic features and speech semantic features corresponding to the same text content; inputting the target semantic features into a multimodal fusion feature generation model, and outputting the corresponding action sequence; based on the target semantic features and the action sequence, calculating the loss function value of the multimodal fusion feature generation model; adjusting the model parameter value of the multimodal fusion feature generation model based on the loss function value, and continuing training until the preset training completion conditions are met to obtain a trained multimodal fusion feature generation model. The embodiment of the present application guides the generation of human speaking actions by using the semantic features generated by the fusion of text semantic features and speech semantic features corresponding to the same text content, thereby outputting natural, smooth and reasonable human animations.

[0075] In some embodiments, obtaining the target semantic feature includes: obtaining a preset number of understanding dimensions of text semantic features and speech semantic features, wherein the text description corresponding to the text semantic feature is a text description of the speech information corresponding to the speech semantic feature; determining the correlation coefficients of the text semantic features and the speech semantic features under each understanding dimension; and fusing the text semantic features and the speech semantic features based on the correlation coefficients of the text semantic features and the speech semantic features under each understanding dimension to obtain the target semantic feature.

[0076] Among them, the semantic feature of the number of understanding dimensions is a high-dimensional vector, each understanding dimension can represent an attribute of the semantic feature, and each attribute can affect the overall semantic understanding of the semantic feature.

[0077] The preset number of understanding dimensions of text semantic features and speech semantic features in the embodiment of the present application can be 1024 dimensions or 4096 dimensions, etc.

[0078] In some embodiments, text semantic features and speech semantic features of a preset number of understanding dimensions may be input into a semantic correlation fuser, so that correlation coefficients of the text semantic features and speech semantic features at each understanding dimension may be output.

[0079] Furthermore, the text semantic features and the speech semantic features can be fused based on the correlation coefficients of the text semantic features and the speech semantic features in each understanding dimension to obtain the target semantic features.

[0080] In some embodiments, text semantic features and speech semantic features are fused based on the correlation coefficients of the text semantic features and speech semantic features in each understanding dimension to obtain target semantic features, including: determining weight coefficients of the text semantic features and speech semantic features in each understanding dimension based on the correlation coefficients of the text semantic features and speech semantic features in each understanding dimension, the larger the correlation coefficient, the stronger the correlation between the features, and the larger the weight coefficient; generating target semantic features based on the feature attributes in each understanding dimension and the weight coefficient in each understanding dimension.

[0081] It can be understood that each understanding dimension can affect the overall semantic understanding of semantic features, and the larger the correlation coefficient of text semantic features and speech semantic features under each understanding dimension, the stronger the correlation between features, that is, the two semantic features under this understanding dimension are relatively similar, indicating that the two semantic features under this understanding dimension are relatively correct, and a higher weight coefficient can be set for this understanding dimension. On the contrary, the smaller the correlation coefficient of text semantic features and speech semantic features under each understanding dimension, the lower the correlation between features, the accuracy of one of the two semantic features under this understanding dimension is relatively low or the accuracy of both semantic features is relatively low, and a lower weight coefficient can be set for this understanding dimension.

[0082] Furthermore, the target semantic features may be generated based on the feature attributes of the text semantic features and the speech semantic features under each understanding dimension and the weight coefficients under each understanding dimension.

[0083] Among them, the feature attributes of text semantic features and speech semantic features under each understanding dimension can be fused, and further calculations can be performed based on the fused feature attributes and corresponding weight coefficients to generate target semantic features.

[0084] Alternatively, the feature attributes of the text semantic features and speech semantic features under each understanding dimension are calculated with the corresponding weight coefficients to obtain the calculated feature attributes, and further, the calculated feature attributes under each understanding dimension are fused to generate the target semantic features.

[0085] The calculation method can be addition, multiplication, etc. For example, one step of calculation based on the fused feature attributes and the corresponding weight coefficients can be to multiply the fused feature attributes under each understanding dimension with the corresponding weight coefficients respectively, and then add the multiplied calculation results to obtain the target semantic feature.

[0086] In some embodiments, the present application provides a structural diagram of a semantic relevance fuser.

[0087] like Figure 3 As shown, the structure includes multiple convolutional layers, fully connected layers and activation layers.

[0088] The multiple convolutional layers include a first convolutional layer, a second convolutional layer and a third convolutional layer.

[0089] The parameters of multiple convolutional layers are set as follows:

[0090] First convolutional layer:

[0091] Input channels:3.

[0092] Output channels: 64.

[0093] Convolution kernel size: 4x4.

[0094] Step length: 2.

[0095] Padding:1.

[0096] Use the LeakyReLU activation function with a slope of 0.2.

[0097] Second convolutional layer:

[0098] Input channels: 64.

[0099] Output channels: 128.

[0100] Convolution kernel size: 4x4.

[0101] Step length: 2.

[0102] Padding:1.

[0103] LeakyReLU activation with slope 0.2.

[0104] The third convolutional layer:

[0105] Input channels: 128.

[0106] Output channels: 256.

[0107] Convolution kernel size: 4x4.

[0108] Step length: 2.

[0109] Padding:1.

[0110] LeakyReLU activation with slope 0.2.

[0111] Through these multiple convolutional layers, the text semantic features and speech semantic features of the learning input can be combined into local features. When processing text, the convolution operation can capture the relationship between words and phrases, thereby extracting effective semantic features. Through the pooling operation, the convolution layer can effectively reduce the feature dimension and compress information, making subsequent calculations more efficient. Using convolution kernels of different sizes, features can be extracted from multiple scales, which helps to capture semantic information at different levels. This is especially important for complex semantic tasks, because different contexts may require different semantic understandings. Further combined with the activation function, nonlinear transformations are introduced, so that the model can better learn complex semantic relationships. This nonlinear characteristic enhances the expressive power of the model. Further output a feature map, which includes the relationship between text semantic features and speech semantic features.

[0112] The parameters of the fully connected layer are set as follows:

[0113] Number of input nodes: 256*4*4: means that after three convolutional layers, the size of the feature map is 4x4 and the number of channels is 256.

[0114] Number of output nodes: 1.

[0115] Using the Sigmoid activation function means: the output value is between 0 and 1, indicating the correlation between the input features.

[0116] The fully connected layer flattens the feature map into a one-dimensional vector and performs a linear transformation through the weight matrix and bias. This process can comprehensively consider all input features and realize the association between different features, so as to better express complex semantic information. After the fully connected layer, a nonlinear activation function is usually applied. The introduction of nonlinearity helps the model learn complex patterns and relationships, so that the output can capture more feature interactions, and finally output the correlation coefficients of text semantic features and speech semantic features under each understanding dimension.

[0117] In some embodiments, obtaining text semantic features and speech semantic features of preset dimensions includes:

[0118] The text description is input into the target text model and the speech information is input into the target language model, and the text semantic features and speech semantic features of the preset understanding dimension number are output.

[0119] First, since it is necessary to determine the correlation between text semantic features and speech semantic features, word embedded features refer to a set of numbers that carry specific abstract information, usually stored in a vector with a fixed length. The greatest use of word embedded features is in similarity calculation. Therefore, text semantic features and speech semantic features are word embedded features.

[0120] The target text big model and the target language big model are multi-round conversational outputs, which are not directly suitable for feature extraction. The present invention uses the open source target text big model to extract text features, and transforms the final output result on this basis, cutting off the last linear layer and replacing it with a fully connected layer to obtain text semantic features.

[0121] Compared with the text model, the speech model has a more obvious advantage in processing speech information. After training with large-scale speech data, it can better capture the information contained in the speech data. The present invention is based on the WavLM speech model, and the final output layer is transformed on this basis to extract the semantic features of the sound.

[0122] Specifically, the WavLM model comes from the Natural Language Computing Group of Microsoft Research Asia. In the natural language pre-training, based on the Transformer model architecture, a pre-training scheme of Denoising Masked Speech Modeling is proposed. The WavLM model contains a convolutional encoder and a Transformer encoder. Among them, the convolutional encoder has a total of 7 layers, each of which consists of a time domain convolution layer, a normalization layer and a GELU activation function layer. In the Transformer encoder, WavLM uses gated relative position bias to introduce relative position into the calculation of the attention network in order to better model local information. During the training process, WavLM randomly transforms the input wav, such as mixing two wavs or adding background noise. Then, about 50% of the speech signal is randomly masked, and the label corresponding to the masked position is predicted at the output. WavLM follows the idea proposed by HuBERT, converts continuous signals into discrete labels through the Kmeans method, and uses discrete labels as targets for modeling. In formal terms, given an input speech X, its label Y is first extracted, and then X is denoised and masked to generate X, and the Transformer model needs to predict the label Y of the masked position by inputting X. In the present invention, the prior understanding ability of the speech large model WavLM on massive speech data is used to extract the sound semantic features, and the probability features of its output are progressively combined and truncated, and finally spliced ​​into word embedding features of the same dimension as the semantic features of the previous text.

[0123] In some embodiments, the structure of the target text model is as follows: Figure 4 As shown, it includes a construction layer, a word segmenter, an output embedding layer, and an output layer.

[0124] The process of obtaining text semantic features can be implemented as follows:

[0125] Initialize the target text model and word segmenter.

[0126] Construct a text sequence according to the data format of Q, K, and V in the attention module in the construction layer.

[0127] The text sequence is input into the tokenizer for encoding and then into the output embedding layer for processing.

[0128] Based on the output of the output embedding layer, the output layer is processed by the function last_token_pool to finally obtain the text semantic features.

[0129] In some embodiments, the structure of the target speech model is as follows: Figure 5 As shown, it includes convolutional layers, construction layers, word segmenters, output embedding layers and multiple output layers.

[0130] The process of obtaining speech semantic features can be implemented as follows:

[0131] Initialize the target text model and word segmenter.

[0132] The speech information is constructed into a text sequence according to the data format of Q, K, and V in the attention module in the construction layer.

[0133] The text sequence is input into the tokenizer for encoding and then into the output embedding layer for processing.

[0134] Based on the output of the output embedding layer, the embedded features of each output layer are obtained through processing by the function model.extract_features.

[0135] The output results of each layer obtained in the above steps are linearly concatenated to obtain a 4096-dimensional embedded feature with the same dimension as the text semantic feature.

[0136] For ease of understanding, the present application embodiment provides a flowchart of a large model training method for generating action sequences based on text-speech correlation discrimination, such as Figure 6 As shown:

[0137] First, the text description is input into the target text model and the speech information is input into the target language model, and the text semantic features and speech semantic features of the preset understanding dimension number are output.

[0138] The text semantic features and speech semantic features are input into the semantic correlation fuser to obtain the target semantic features.

[0139] The target semantic features are input into the multimodal fusion feature generation model and the corresponding action sequence is output.

[0140] The action sequence and target semantic features are input into the semantic relevance discriminator to obtain the relevance score.

[0141] Based on the target semantic features and action sequences, the loss function value of the multimodal fusion feature generation model is calculated; based on the loss function value, the model parameter values ​​of the multimodal fusion feature generation model are adjusted, and training is continued until the preset training completion conditions are met to obtain a trained multimodal fusion feature generation model.

[0142] An action sequence generation method proposed according to an embodiment of the present application is described below in conjunction with the accompanying drawings. The method is applied to the above-mentioned multimodal fusion feature generation model.

[0143] like Figure 7 As shown, the method comprises the following steps:

[0144] S701, input the target semantic features generated by fusing the text semantic features and speech semantic features corresponding to the same text content into a multimodal fusion feature generation model, and output the corresponding action sequence.

[0145] For ease of understanding, the present application embodiment provides a flow chart of an action sequence generation method, such as Figure 8 As shown:

[0146] First, the text description is input into the target text model and the speech information is input into the target language model, and the text semantic features and speech semantic features of the preset understanding dimension number are output.

[0147] The text semantic features and speech semantic features are input into the semantic correlation fuser to obtain the target semantic features.

[0148] The target semantic features are input into the multimodal fusion feature generation model and the corresponding action sequence is output.

[0149] The present application also provides a large model training based on text-speech correlation discrimination to generate action sequences, and the device is used to execute the large model training method based on text-speech correlation discrimination to generate action sequences provided in any of the above embodiments. Fig. 9 As shown, the device includes: an acquisition module 901, an input module 902, a calculation module 903 and an adjustment module 904.

[0150] An acquisition module 901 is used to acquire a target semantic feature, where the target semantic feature is generated by fusing a text semantic feature and a speech semantic feature corresponding to the same text content;

[0151] An input module 902 is used to input the target semantic features into the multimodal fusion feature generation model and output the corresponding action sequence;

[0152] A calculation module 903 is used to calculate the loss function value of the multimodal fusion feature generation model based on the target semantic features and the action sequence;

[0153] The adjustment module 904 is used to adjust the model parameter values ​​of the multimodal fusion feature generation model based on the loss function value, and continue training until the preset training completion condition is met to obtain a trained multimodal fusion feature generation model.

[0154] In some embodiments, the acquisition module 901 is specifically used to:

[0155] Obtaining a preset number of understanding dimension text semantic features and speech semantic features, wherein the text description corresponding to the text semantic feature is a text description of the speech information corresponding to the speech semantic feature;

[0156] Determine the correlation coefficients of text semantic features and speech semantic features under each comprehension dimension;

[0157] The text semantic features and speech semantic features are fused based on the correlation coefficients of the text semantic features and speech semantic features in each understanding dimension to obtain the target semantic features.

[0158] In some embodiments, the acquisition module 901 is further specifically configured to:

[0159] The text description is input into the target text model and the speech information is input into the target language model, and the text semantic features and speech semantic features of the preset understanding dimension number are output.

[0160] In some embodiments, the acquisition module 901 is further specifically configured to:

[0161] Determine the weight coefficient of each comprehension dimension of text semantic features and speech semantic features based on the correlation coefficient of each comprehension dimension of text semantic features and speech semantic features. The larger the correlation coefficient, the stronger the correlation between the features and the larger the weight coefficient.

[0162] Based on the feature attributes under each understanding dimension and the weight coefficient under each understanding dimension, the target semantic features are generated.

[0163] In some embodiments, the input module 902 is specifically configured to:

[0164] Encode the target semantic features using the encoder of the multimodal fusion feature generation model to obtain encoded semantic features;

[0165] Determine the action at each time step corresponding to the encoded semantic feature using the decoder of the multimodal fusion feature generation model;

[0166] Based on the actions at each time step, the combination generates an action sequence corresponding to the encoded semantic features.

[0167] In some embodiments, the input module 902 is further specifically configured to:

[0168] For each time step, the decoder generates a probability distribution of all actions at the current time step based on the internal state and the action at the previous time step;

[0169] The target action is selected from the probability distribution based on a preset selection strategy.

[0170] In some embodiments, the calculation module 903 is specifically configured to:

[0171] Calculate the correlation score between the target semantic features and the action sequence;

[0172] Based on the correlation score and the preset correlation, the loss function value of the multimodal fusion feature generation model is calculated.

[0173] The present application also provides an action sequence generation device, which is used to execute the action sequence generation method provided in any of the above embodiments. Fig.10 As shown, the device includes: an input module 1001.

[0174] The input module 1001 is used to input the target semantic features generated by fusing the text semantic features and the speech semantic features corresponding to the same text content into the multimodal fusion feature generation model, and output the corresponding action sequence.

[0175] The present application also provides an electronic device to perform the above-mentioned large model training method or action sequence generation method based on text-speech correlation discrimination to generate action sequences. Fig.10 It shows a schematic diagram of an electronic device provided by some embodiments of the present application. Fig.10 As shown, the electronic device 7 includes: a processor 700, a memory 701, a bus 702 and a communication interface 703, and the processor 700, the communication interface 703 and the memory 701 are connected via the bus 702; the memory 701 stores a computer program that can be run on the processor 700, and when the processor 700 runs the computer program, it executes the large model training method or action sequence generation method based on text-speech correlation discrimination to generate an action sequence provided in any of the aforementioned embodiments of the present application.

[0176] The memory 701 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. The communication connection between the device network element and at least one other network element is realized through at least one communication interface 703 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0177] The bus 702 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. Among them, the memory 701 is used to store programs, and the processor 700 executes the program after receiving the execution instruction. The large model training method or the action sequence generation method based on text-speech correlation discrimination to generate an action sequence disclosed in any of the embodiments of the present application can be applied to the processor 700, or implemented by the processor 700.

[0178] The processor 700 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 700. The above processor 700 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a readily available programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 701, and the processor 700 reads the information in the memory 701 and completes the steps of the above method in combination with its hardware.

[0179] The electronic device provided in the embodiment of the present application and the large model training method or action sequence generation method based on text-speech correlation discrimination to generate action sequences provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented therein.

[0180] The present application also provides a computer-readable storage medium corresponding to the large model training method or action sequence generation method based on text-speech correlation discrimination to generate action sequences provided in the above-mentioned embodiments, please refer to Fig.11 The computer-readable storage medium shown is a CD 30, on which a computer program (i.e., a program product) is stored. When the computer program is run by the processor, it will execute the large model training method or action sequence generation method based on text-speech correlation discrimination to generate an action sequence provided by any of the aforementioned embodiments.

[0181] It should be noted that examples of computer-readable storage media may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0182] The computer-readable storage medium provided by the above-mentioned embodiments of the present application and the large model training method or action sequence generation method based on text-speech correlation discrimination for generating action sequences provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0183] It should be noted that:

[0184] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known structures and technologies are not shown in detail so as not to obscure the understanding of this description.

[0185] Similarly, it should be understood that in order to streamline the present application and help understand one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present application, the various features of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, the disclosed method should not be interpreted as reflecting the following schematic diagram: the claimed application requires more features than the features clearly stated in each claim. More specifically, as reflected in the claims below, the inventive aspects are less than all the features of the single embodiment disclosed above. Therefore, the claims following the specific embodiment are hereby expressly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present application.

[0186] In addition, those skilled in the art will appreciate that, although some embodiments herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present application and form different embodiments. For example, in the claims below, any one of the claimed embodiments may be used in any combination.

[0187] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.< / start>

Claims

1. A large model training method for generating action sequences based on text-speech correlation discrimination, characterized in that: include: Acquire a target semantic feature, where the target semantic feature is generated by fusing a text semantic feature and a speech semantic feature corresponding to the same text content; Inputting the target semantic features into a multimodal fusion feature generation model and outputting a corresponding action sequence; Based on the target semantic features and the action sequence, calculating the loss function value of the multimodal fusion feature generation model; The model parameter values ​​of the multimodal fusion feature generation model are adjusted based on the loss function value, and the training is continued until a preset training completion condition is met to obtain a trained multimodal fusion feature generation model.

2. The method according to claim 1, characterized in that The obtaining of the target semantic features includes: Acquire a preset number of text semantic features and speech semantic features of the understanding dimension, wherein the text description corresponding to the text semantic feature is a text description of the speech information corresponding to the speech semantic feature; Determine the correlation coefficients of the text semantic features and the speech semantic features under each understanding dimension; The text semantic features and the speech semantic features are fused based on the correlation coefficients of the text semantic features and the speech semantic features under each understanding dimension to obtain the target semantic features.

3. The method according to claim 2, characterized in that The obtaining of text semantic features and speech semantic features of preset dimensions includes: The text description is input into the target text model and the speech information is input into the target language model, and the text semantic features and speech semantic features of the preset understanding dimension number are output.

4. The method according to claim 2, characterized in that: The step of fusing the text semantic features and the speech semantic features based on the correlation coefficients of the text semantic features and the speech semantic features under each understanding dimension to obtain the target semantic features includes: Determine the weight coefficients of the text semantic features and the speech semantic features under each comprehension dimension based on the correlation coefficients of the text semantic features and the speech semantic features under each comprehension dimension, wherein the larger the correlation coefficient, the stronger the correlation between the features, and the larger the weight coefficient; Based on the feature attributes under each understanding dimension and the weight coefficient under each understanding dimension, the target semantic features are generated.

5. The method according to claim 1, characterized in that The step of inputting the target semantic features into a multimodal fusion feature generation model and outputting a corresponding action sequence includes: Encoding the target semantic feature using an encoder of a multimodal fusion feature generation model to obtain an encoded semantic feature; Determine the action of each time step corresponding to the encoded semantic feature using a decoder of the multimodal fusion feature generation model; Based on the actions at each time step, an action sequence corresponding to the encoded semantic feature is generated by combination.

6. The method according to claim 5, characterized in that The action of using the decoder of the multimodal fusion feature generation model to determine each time step corresponding to the encoded semantic feature includes: For each time step, the decoder generates a probability distribution of all actions in the current time step based on the internal state and the action in the previous time step; A target action is selected from the probability distribution based on a preset selection strategy.

7. The method according to claim 1, characterized in that The calculating the loss function value of the multimodal fusion feature generation model based on the target semantic feature and the action sequence includes: Calculating a correlation score between the target semantic feature and the action sequence; Based on the correlation score and the preset correlation, the loss function value of the multimodal fusion feature generation model is calculated.

8. A method for generating an action sequence, characterized in that: Applied to the multimodal fusion feature generation model according to any one of claims 1 to 7, the method comprising: The target semantic features generated by fusing the text semantic features and the speech semantic features corresponding to the same text content are input into the multimodal fusion feature generation model, and the corresponding action sequence is output.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: The processor runs the computer program to implement the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the method according to any one of claims 1 to 8.

Images