Task processing model training, task processing and emotional speech generation method

Abstract

This specification provides embodiments of a task processing model training, task processing, and emotional speech generation method. The task processing model training method relates to the field of artificial intelligence technology and includes: acquiring a first sample set and a second sample set; using the first sample set to train an initial processing unit and at least two initial encoding units in the task processing model to obtain a target processing unit and at least two target encoding units; using the second sample set to train an initial decoding unit in the task processing model to obtain a target decoding unit; and constructing a trained task processing model based on the target processing unit, the target decoding unit, and the at least two target encoding units. By setting at least two initial encoding units in the model for encoding non-textual modal data, a model capable of directly processing multimodal data is obtained.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of artificial intelligence technology, and in particular to methods for training task processing models, task processing, and generating emotional speech. Background Technology

[0002] In the field of artificial intelligence, large models have already demonstrated powerful capabilities for processing text-based data, enabling them to perform a variety of tasks from translation to content generation. With technological advancements, how to leverage large models to process non-textual data (such as images, audio, and video) has become a key research focus.

[0003] Currently, non-textual modal data is typically converted into textual modal data, and then processed by a large model. However, in this process, the large model still processes the textual modal data, and the modality conversion loses the characteristics of the non-textual modal data. Therefore, there is an urgent need for a task processing model that can handle data of different modalities. Summary of the Invention

[0004] In view of this, embodiments of this specification provide a task processing model training method. One or more embodiments of this specification simultaneously relate to a task processing method, an emotional speech generation method, an information processing method based on a task processing model, a task platform, a task processing model training device, a task processing device, an emotional speech generation device, an information processing device based on a task processing model, a computing device, an electronic device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.

[0005] According to a first aspect of the embodiments of this specification, a task processing model training method is provided, comprising: acquiring a first sample set and a second sample set; using the first sample set to train an initial processing unit and at least two initial encoding units in the task processing model to obtain a target processing unit and at least two target encoding units, wherein the at least two initial encoding units are used to encode non-textual modal data, and different initial encoding units encode data of different modalities respectively, and the initial processing unit is used to perform feature fusion on the outputs of the at least two initial encoding units; using the second sample set to train an initial decoding unit in the task processing model to obtain a target decoding unit; and constructing a trained task processing model based on the target processing unit, the target decoding unit, and the at least two target encoding units.

[0006] According to a second aspect of the embodiments of this specification, a task processing method is provided, comprising: acquiring task data of a target task, wherein the task data includes at least one of text data, voice data, and image data; inputting the task data into a task processing model to obtain a task processing result, wherein the task processing model is trained based on a task processing model training method.

[0007] According to a third aspect of the embodiments of this specification, an emotional speech generation method is provided, comprising: receiving emotional speech generation data sent by a terminal device, wherein the emotional speech generation data includes at least one of text data, voice data, and image data; inputting the emotional speech generation data into a task processing model to obtain a target emotional speech, wherein the task processing model is trained based on a task processing model training method; and feeding back the target emotional speech to the terminal device.

[0008] According to a fourth aspect of the embodiments of this specification, an information processing method based on a task processing model is provided, applied to a task platform, comprising: receiving a model request sent by a terminal device; and determining a target task processing model from multiple task processing models based on the model request, wherein the multiple task processing models are trained based on a task processing model training method.

[0009] According to a fifth aspect of the embodiments of this specification, a task platform is provided, including a request interface and a response unit; the request interface is used to receive a model request sent by a terminal device, wherein the model request includes at least one of a scene identifier of a target scene, scene input data of the target scene, and model specification parameters; the response unit is used to determine a target task processing model from multiple task processing models based on the model request, wherein the multiple task processing models are trained based on a task processing model training method.

[0010] According to a sixth aspect of the embodiments of this specification, a task processing model training apparatus is provided, comprising: a first acquisition module configured to acquire a first sample set and a second sample set; a first training module configured to use the first sample set to train an initial processing unit and at least two initial encoding units in the task processing model to obtain a target processing unit and at least two target encoding units, wherein the at least two initial encoding units are used to encode non-textual modal data, and different initial encoding units encode data of different modalities respectively, and the initial processing unit is used to perform feature fusion on the outputs of the at least two initial encoding units; a second training module configured to use the second sample set to train an initial decoding unit in the task processing model to obtain a target decoding unit; and a construction module configured to construct the trained task processing model based on the target processing unit, the target decoding unit, and the at least two target encoding units.

[0011] According to a seventh aspect of the embodiments of this specification, a task processing apparatus is provided, comprising: a second acquisition module configured to acquire task data of a target task, wherein the task data includes at least one of text data, voice data, and image data; and a first input module configured to input the task data into a task processing model to obtain a task processing result, wherein the task processing model is trained based on a task processing model training method.

[0012] According to an eighth aspect of the embodiments of this specification, an emotional speech generation apparatus is provided, comprising: a first receiving module configured to receive emotional speech generation data sent by a terminal device, wherein the emotional speech generation data includes at least one of text data, voice data, and image data; and a second input module configured to input the emotional speech generation data into a task processing model to obtain target emotional speech, wherein the task processing model is trained based on a task processing model training method.

[0013] The feedback module is configured to send the target emotional voice back to the terminal device.

[0014] According to a ninth aspect of the embodiments of this specification, an information processing apparatus based on a task processing model is provided, applied to a task platform, comprising: a second receiving module configured to receive a model request sent by a terminal device; and a determining module configured to determine a target task processing model from a plurality of task processing models based on the model request, wherein the plurality of task processing models are trained based on a task processing model training method.

[0015] According to a tenth aspect of the embodiments of this specification, a computing device is provided, comprising: a memory and a processor; the memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, wherein the computer programs / instructions, when executed by the processor, implement the steps of the methods provided in the first, second, third, or fourth aspects described above.

[0016] According to an eleventh aspect of the embodiments of this specification, an electronic device is provided, including: a memory and a processor, the memory and the processor being connected via a bus; the memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, wherein when the computer programs / instructions are executed by the processor, they implement the steps of the methods provided in the first, second, third, or fourth aspects described above.

[0017] According to a twelfth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores a computer program / instructions that, when executed by a processor, implement the steps of the methods provided in the first, second, third, or fourth aspects described above.

[0018] According to a thirteenth aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the methods provided in the first, second, third, or fourth aspects described above.

[0019] This specification provides a task processing model training method according to one embodiment, comprising: acquiring a first sample set and a second sample set; using the first sample set to train an initial processing unit and at least two initial encoding units in the task processing model to obtain a target processing unit and at least two target encoding units, wherein the at least two initial encoding units are used to encode non-textual modal data, and different initial encoding units encode data of different modalities respectively, and the initial processing unit is used to perform feature fusion on the outputs of the at least two initial encoding units; using the second sample set to train an initial decoding unit in the task processing model to obtain a target decoding unit; and constructing a trained task processing model based on the target processing unit, the target decoding unit, and the at least two target encoding units. By setting at least two initial encoding units in the model for encoding non-textual modal data, direct encoding of non-textual modal data is achieved, thereby preserving the characteristics and information of the original data, avoiding information loss due to modality conversion, and allowing textual modal data to be used as an intermediate bridge to achieve alignment processing of at least three modalities, resulting in a highly accurate task processing model capable of handling multimodal data. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating a task processing model training method provided in one embodiment of this specification;

[0021] Figure 2 This is a flowchart illustrating the processing procedure of a task processing model training method provided in one embodiment of this specification.

[0022] Figure 3 This is a flowchart illustrating a task processing method provided in one embodiment of this specification;

[0023] Figure 4 This is a flowchart illustrating an embodiment of an emotional speech generation method provided in this specification;

[0024] Figure 5 This is an architecture diagram of a task processing system provided in one embodiment of this specification;

[0025] Figure 6 This is a flowchart illustrating an information processing method based on a task processing model, provided in one embodiment of this specification.

[0026] Figure 7This is a schematic diagram of the structure of a task platform provided in one embodiment of this specification;

[0027] Figure 8 This is a schematic diagram of the structure of a task processing model training device provided in one embodiment of this specification;

[0028] Figure 9 This is a schematic diagram of the structure of a task processing device provided in one embodiment of this specification;

[0029] Figure 10 This is a schematic diagram of the structure of an emotional speech generation device provided in one embodiment of this specification;

[0030] Figure 11 This is a schematic diagram of the structure of an information processing device based on a task processing model provided in one embodiment of this specification;

[0031] Figure 12 This is a structural block diagram of a computing device provided in one embodiment of this specification;

[0032] Figure 13 This is a structural block diagram of an electronic device provided in one embodiment of this specification. Detailed Implementation

[0033] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0034] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0035] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0036] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0037] In one or more embodiments of this specification, a large model refers to a deep learning model with a large number of model parameters, typically containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even tens of trillions of model parameters. A large model can also be called a foundational model (Foundation Model 1). It is pre-trained using large-scale unlabeled corpora to produce a pre-trained model with hundreds of millions of parameters. Such models can adapt to a wide range of downstream tasks and have good generalization ability. Examples include Large Language Models (LLMs) and Multi-modal Pre-training Models (MLMs).

[0038] In practical applications, large models only require a small number of samples to fine-tune the pre-trained model before they can be applied to different tasks. Large models can be widely used in fields such as Natural Language Processing (NLP) and Computer Vision. Specifically, they can be applied to computer vision tasks such as Visual Question Answering (VQA), Image Captioning (IC), and Image Generation, as well as NLP tasks such as text-based sentiment classification, text summarization, and machine translation. The main application scenarios for large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.

[0039] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0040] Multimodal Large Language Model (MLLM): A large-scale pre-trained model capable of processing multiple modalities of data, including text, images, and speech.

[0041] Zero-shot alignment: Achieving alignment and transformation between different modalities when three modalities (text-image-speech) data are not available simultaneously.

[0042] Language Pivot: Utilizes language as an intermediate bridge to connect different modalities and achieve alignment between them.

[0043] Lightweight Speech Decoder: This refers to a decoder module that consumes few resources and can generate speech in real time.

[0044] Direct Preference Optimization (DPO) is a model optimization method based on user preferences, used to improve the quality and consistency of model-generated results.

[0045] Automatic Speech Recognition (ASR) is a technology in computer science that enables machines to convert human speech into text. This technology is widely used in various products and services, such as virtual assistants, voice search, and voice-controlled devices. An ASR system typically includes steps such as sound capture, feature extraction, pattern matching, and language processing.

[0046] Supervised fine-tuning (SFT) is a method of further training a model based on a pre-trained model. In this method, the model is trained on a dataset containing pairs of inputs and desired outputs so that it learns how to generate responses closer to human-level performance. Supervised fine-tuning is often used to adapt a model to task-specific or domain-specific data, thereby improving its performance on those tasks.

[0047] Transformer is a deep learning model based on self-attention mechanisms. Originally designed for natural language processing tasks, it abandons traditional recurrent neural networks and convolutional neural networks, instead employing self-attention to process sequential data. By allowing each position to interact directly with all other positions in the sequence, it can efficiently capture long-range dependencies and supports parallel training, significantly improving training speed and model performance.

[0048] This specification provides a task processing model training method, and also relates to a task processing method, an emotional speech generation method, an information processing method based on a task processing model, a task platform, a task processing model training device, a task processing device, an emotional speech generation device, an information processing device based on a task processing model, a computing device, an electronic device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0049] See Figure 1 , Figure 1 This specification illustrates a flowchart of a task processing model training method according to an embodiment, which specifically includes the following steps:

[0050] Step 102: Obtain the first sample set and the second sample set.

[0051] It should be noted that the first sample set is used to train the multimodal alignment capability of the initial processing unit and at least two initial encoding units in the task processing model. The second sample set is used to train the decoding capability of the initial decoding unit in the task processing model.

[0052] In practical applications, there are multiple ways to obtain the first and second sample sets, and the specific method chosen depends on the actual situation. This specification does not impose any limitations on these methods in its embodiments. In one possible implementation, the first and second sample sets can be received from a user's terminal device. In another possible implementation, the first and second sample sets can be read from other data acquisition devices or databases.

[0053] Step 104: Using the first sample set, train the initial processing unit and at least two initial encoding units in the task processing model to obtain the target processing unit and at least two target encoding units. The at least two initial encoding units are used to encode non-textual modal data, and different initial encoding units encode data of different modalities respectively. The initial processing unit is used to perform feature fusion on the output of the at least two initial encoding units.

[0054] It should be noted that the initial processing unit is used to receive the outputs of at least two initial coding units and perform feature fusion on these outputs. Therefore, the initial processing unit can be understood as an initial feature fusion unit. The at least two initial coding units include, but are not limited to, an initial speech coding unit, an initial image coding unit, and an initial video coding unit. Since the initial processing unit can directly process text-modal data, the task processing model may only include at least two initial coding units. Of course, in addition to at least two initial coding units, the task processing model may also include a text coding unit for encoding text-modal data. The initial processing unit can be a large model or a full-modal large model. The initial coding unit is used to encode the input data to obtain the encoded features corresponding to the input data. The structure of the initial coding unit can be selected according to the data modality being processed. The initial coding unit includes, but is not limited to, Transformer structures, convolutional neural networks, and graph neural networks. The specific selection is based on the actual situation, and the embodiments in this specification do not impose any limitations on this.

[0055] Since the task processing model includes at least two initial encoding units, which are used to encode non-textual modal data respectively, and the encoded data modalities are different, in one possible implementation of this specification, the first sample set may include sample data of at least three modalities. The sample data of at least three modalities includes text modal sample data and sample data corresponding to the modalities of each initial encoding unit. For example, the task processing model includes two initial encoding units, which are an initial speech encoding unit for encoding speech modal data and an initial image encoding unit for encoding image modal data. The first sample set includes three-modal sample data of speech, image, and text. In another possible implementation of this specification, since three-modal sample data is relatively scarce, the first sample set may include multiple sample subsets, each corresponding one-to-one with an initial encoding unit. The sample subsets include text modal sample data and sample data corresponding to the modalities of the initial encoding units. Taking the above example, the first sample set may include a first sample subset for training the initial speech encoding unit and a second sample subset for training the initial image encoding unit. The first sample subset can include two modal sample data of speech and text, and the second sample subset can include two modal sample data of image and text. In this approach, text modal data can be used as the language hub, eliminating the need to construct the first sample set based on three modal sample data, thus enabling model training based on a first sample set of zero-sample full modal sample data.

[0056] In practical applications, there are multiple ways to train the initial processing unit and at least two initial encoding units in the task processing model using the first sample set to obtain the target processing unit and at least two target encoding units. The specific method chosen depends on the actual situation, and this specification does not limit this approach. In one possible implementation, the first sample set includes sample data for the text modality and sample data for the modality corresponding to each initial encoding unit. The first sample set can be directly used to train the initial processing unit and at least two initial encoding units in the task processing model to obtain the target processing unit and at least two target encoding units. In another possible implementation, the first sample set includes a subset of samples corresponding to each initial encoding unit. This subset includes sample data for the text modality and sample data for the modality corresponding to the initial encoding unit. These subsets can be used to train the initial processing unit and its corresponding initial encoding unit.

[0057] In one optional embodiment of this specification, taking at least two initial coding units including an initial speech coding unit and an initial image coding unit, and a first sample set including a first sample subset for the initial speech coding unit and a second sample subset for the initial image coding unit as an example, the above-mentioned use of the first sample set to train the initial processing unit and at least two initial coding units in the task processing model to obtain the target processing unit and at least two target coding units may include the following steps:

[0058] The initial speech coding unit is trained using the first sample subset to obtain the target speech coding unit;

[0059] Using the second sample subset, the initial image coding unit and the initial processing unit are trained to obtain the target processing unit and the target image coding unit.

[0060] It should be noted that the initial speech coding unit can be a speech encoder based on a convolutional neural network or a speech encoder based on a Transformer architecture. Similarly, the initial image coding unit can be an image encoder based on a convolutional neural network or an image encoder based on a Transformer architecture. Experimental testing has shown that training the initial speech coding unit and the initial processing unit using the first sample subset may lead to a decrease in the processing capability of the initial processing unit. Therefore, in this embodiment, the initial speech coding unit is trained using the first sample subset, and is not trained at all.

[0061] In practical applications, there are various ways to train the initial speech coding unit using the first sample subset to obtain the target speech coding unit. The specific method chosen depends on the actual situation, and this specification does not impose any limitations on this approach. One possible implementation in this specification is to use supervised fine-tuning with the labeled first sample subset to train the model by minimizing the difference between the predicted output and the true label. Another possible implementation in this specification is to use unsupervised or self-supervised methods when there is insufficient labeled data. For example, contrastive learning can be used to teach the model to distinguish between positive sample pairs (different segments from the same speech) and negative sample pairs (segments from different speech). Furthermore, the method of training the initial image coding unit and the initial processing unit using the second sample subset to obtain the target processing unit and the target image coding unit can refer to the above implementation method of "training the initial speech coding unit using the first sample subset to obtain the target speech coding unit," and will not be elaborated further in this specification.

[0062] By applying the scheme of the embodiments of this specification, the initial speech coding unit, the initial image coding unit, and the initial processing unit are trained in a targeted manner using the first sample subset and the second sample subset, respectively. This enables the initial speech coding unit to extract speech features more accurately, the initial image coding unit to efficiently capture key information in the image, and the initial processing unit to enhance its understanding and integration capabilities of multimodal data. This not only improves the performance of the model on specific tasks but also ensures the synergy between different modal data, achieving higher quality multimodal data processing.

[0063] In one optional embodiment of this specification, taking supervised fine-tuning as an example of the training method for the task processing model, the training process of the initial speech coding unit is described. That is, the first sample subset includes a first sample speech, a first sample question for the first sample speech, and a first sample answer to the first sample question. The above-mentioned training of the initial speech coding unit using the first sample subset to obtain the target speech coding unit may include the following steps:

[0064] The first sample speech is input into the initial speech coding unit to obtain the sample speech features;

[0065] Input the sample speech features and the first sample question into the initial processing unit to obtain the first predicted answer;

[0066] Based on the first sample answer and the first predicted answer, the initial speech coding unit is trained to obtain the target speech coding unit.

[0067] It's important to note that the first sample speech and the first sample question carry a true processing label (the first sample answer). The first sample answer serves as the processing target for the initial speech coding unit (EDU) and guides its training process. The first sample speech refers to the actual audio recording file, typically from real-world scenes or synthesized audio data. It can be provided as input to the EDU for encoding. The first sample question is the question text related to the first sample speech. It can be a specific inquiry about the content of the first sample speech, such as "How many dialogue characters are included in the first sample speech?", or it can require some form of understanding or inference based on the first sample speech, such as "Please convert the first sample speech into its corresponding text content." The first sample answer is the true answer to the first sample question, usually manually labeled or predetermined based on known information. The first sample answer is a crucial basis for evaluating the prediction accuracy of the EDU. The EDU is the part of the task processing model responsible for processing the first sample speech. It converts the speech signal into a high-dimensional feature vector or other internal representation through a series of operations. Sample speech features refer to the feature representation obtained after encoding by the EDU. These features capture key information from the first sample speech and can be utilized by subsequent processing steps. The initial processing unit receives the first sample question and features output from the initial speech coding unit and attempts to combine them to generate an answer. The first predicted answer refers to the prediction produced by the task processing model based on its current understanding and learning, according to the sample speech features and the first sample question. The target speech coding unit refers to the initial speech coding unit optimized through training; the target speech coding unit can more accurately extract features from the speech data that are helpful in solving the problem.

[0068] In practical applications, there are multiple ways to input sample speech features and the first sample question into the initial processing unit to obtain the first predicted answer. The specific method chosen depends on the actual situation, and this specification does not impose any limitations on these methods. In one possible implementation, where the task processing model does not include a text encoding unit, the sample speech features and the first sample question can be directly input into the initial unit to obtain the first predicted answer. In another possible implementation, where the task processing model includes a text encoding unit, the first sample question can be input into the text encoding unit to obtain the first sample question features, and the sample speech features and the first sample question features can be input into the initial processing unit to obtain the first predicted answer.

[0069] Furthermore, based on the first sample answer and the first predicted answer, the initial speech coding unit is trained. When obtaining the target speech coding unit, the speech loss value can be calculated based on the first sample answer and the first predicted answer. The unit parameters of the initial speech coding unit are adjusted based on the speech loss value until the training process meets the preset stopping condition, thus obtaining the target speech coding unit. There are many functions for calculating the speech loss value, such as the cross-entropy loss function, the L1 norm loss function, the maximum loss function, the mean squared error loss function, and the logarithmic loss function. The specific function is selected according to the actual situation, and this embodiment does not impose any limitations on it. The preset stopping condition includes, but is not limited to, the speech loss value being less than or equal to a preset threshold and the number of iterations reaching a preset number of iterations. The preset threshold and the preset number of iterations are specifically selected according to the actual situation, and this embodiment does not impose any limitations on them.

[0070] In one possible implementation of this specification, after calculating the speech loss value, the speech loss value can be compared with a preset threshold. Specifically, if the speech loss value is greater than the preset threshold, it indicates that the difference between the first sample answer and the first predicted answer is large, and the initial speech coding unit has poor coding ability for the first sample speech. At this time, the unit parameters of the initial speech coding unit can be adjusted until the speech loss value is less than or equal to the preset threshold, indicating that the difference between the first sample answer and the first predicted answer is small, reaching the preset stopping condition, and obtaining the target speech coding unit.

[0071] In another possible implementation of this specification, in addition to comparing the speech loss value with the preset threshold, the number of iterations can also be considered to determine whether the current initial speech coding unit has been trained successfully. Specifically, if the speech loss value is greater than the preset threshold, the unit parameters of the initial speech coding unit are adjusted until the preset number of iterations is reached, at which point the iteration stops, and the target speech coding unit is obtained.

[0072] The scheme implemented in this specification involves training an initial speech coding unit based on a first sample answer and a first predicted answer. Training continues until a preset stopping condition is met, completing the training and obtaining the target speech coding unit. By continuously adjusting the parameters of the initial speech coding unit, its speech coding capability can be activated, enabling the task processing model to process speech modal data.

[0073] In one optional embodiment of this specification, the training process of the initial image coding unit and the initial processing unit is described using supervised fine-tuning as an example. Specifically, the second sample subset includes the first sample image, a second sample question for the first sample image, and a second sample answer to the second sample question. The above-mentioned training of the initial image coding unit and the initial processing unit using the second sample subset to obtain the target processing unit and the target image coding unit may include the following steps:

[0074] The first sample image is input into the initial image encoding unit to obtain the features of the first sample image;

[0075] The first sample image features and the second sample question are input into the initial processing unit to obtain the second predicted answer;

[0076] Based on the second sample answer and the second predicted answer, the initial image coding unit and the initial processing unit are trained to obtain the target processing unit and the target image coding unit.

[0077] It should be noted that the second sample image and the second sample problem carry true processing labels (second sample answers). The second sample answer is the processing target of the initial image encoding unit and is used to guide the training process of the initial image encoding unit. The first sample image refers to the actual image file, which usually comes from a real-world scene or is synthetically generated image data. The first sample image can be provided as input to the initial image encoding unit for encoding. The second sample problem refers to the question text related to the first sample image. The second sample problem can be a specific question about the content of the first sample image, such as "What color is the first sample image?", or it can require some form of understanding or inference based on the first sample image, such as "Please describe the first sample image in text." The second sample answer refers to the true answer corresponding to the second sample problem. The second sample answer is usually manually labeled or predetermined based on known information. The second sample answer is an important basis for evaluating the prediction accuracy of the initial image encoding unit. The initial image encoding unit is the part of the task processing model responsible for processing the first sample image. The initial image encoding unit can convert the image into a high-dimensional feature vector or other forms of internal representation through a series of operations. The features of the first sample image refer to the feature representation obtained after encoding processing by the initial image encoding unit. These features capture key information from the first sample image and can be utilized by subsequent processing steps. The initial processing unit receives the second sample question and features output from the initial image encoding unit and attempts to combine them to generate an answer. The second predicted answer refers to the prediction produced by the task processing model based on its current understanding and learning, according to the features of the first sample image and the second sample question. The target image encoding unit refers to the initial image encoding unit, which has been trained and optimized to extract features from the image data more accurately that are helpful in solving the problem.

[0078] In practical applications, the implementation of "inputting the first sample image features and the second sample question into the initial processing unit to obtain the second predicted answer" can refer to the implementation of "inputting the sample speech features and the first sample question into the initial processing unit to obtain the first predicted answer" described above. The implementation of "training the initial image coding unit and the initial processing unit based on the second sample answer and the second predicted answer to obtain the target processing unit and the target image coding unit" can refer to the implementation of "training the initial speech coding unit based on the first sample answer and the first predicted answer to obtain the target speech coding unit" described above. Therefore, the embodiments in this specification will not be described in detail.

[0079] By applying the scheme of the embodiments of this specification, the initial image coding unit and the initial processing unit are trained according to the second sample answer and the second predicted answer to obtain the target processing unit and the target image coding unit. This can stimulate the image coding ability of the initial image coding unit and the feature fusion ability of the initial processing unit, so that the task processing model has the ability to process image modal data.

[0080] Step 106: Use the second sample set to train the initial decoding unit in the task processing model to obtain the target decoding unit.

[0081] It should be noted that the second sample set refers to a set of sample data specifically used for training the decoding unit. The second sample set may include multiple encoded sample features (such as feature data from speech modalities, text modalities, image modalities, or other modalities) and their corresponding expected outputs, or it may include multiple unencoded sample data (such as sample data from speech modalities, text modalities, image modalities, or other modalities) and their corresponding expected outputs. The second sample set ensures that the initial decoding unit effectively converts the internal representation generated by the encoding unit back to a human-understandable form or target format. The initial decoding unit refers to the part of the task processing model responsible for converting the internal feature vectors or other forms of representation generated by the initial encoding unit into the final output. The structure of the initial decoding unit can be selected according to the data modality being processed. The initial decoding unit includes, but is not limited to, Transformer structures, convolutional neural networks, and graph neural networks, depending on the specific circumstances. This specification does not impose any limitations on this selection in the embodiments. In the embodiments of this specification, the initial decoding unit includes, but is not limited to, an initial text decoding unit, an initial image decoding unit, and an initial speech decoding unit. The target decoding unit refers to the initial decoding unit optimized through training. The target decoding unit can more accurately map the features generated by the initial encoding unit to the expected output.

[0082] In practical applications, there are various ways to train the initial decoding unit in the task processing model using the second sample set to obtain the target decoding unit. The specific method chosen depends on the actual situation, and this specification does not limit this approach. In one possible implementation, when the data in the second sample set consists of encoded sample features, the sample features can be directly input into the initial decoding unit to obtain the predicted decoding result output by the initial decoding unit. Based on the expected output and predicted decoding result of the sample features, the initial decoding unit in the task processing model is trained to obtain the target decoding unit. In another possible implementation, when the data in the second sample set consists of unencoded sample data, the sample data can be encoded using the target encoding unit in the task processing model according to the modality of the sample data to obtain sample features. These sample features are then input into the initial decoding unit to obtain the predicted decoding result output by the initial decoding unit. Based on the expected output and predicted decoding result of the sample features, the initial decoding unit in the task processing model is trained to obtain the target decoding unit.

[0083] In one optional embodiment of this specification, the second sample set may further include a subset of samples used to train different capabilities of the initial decoding unit. Taking an example where the initial decoding unit includes an initial speech decoding unit, and the second sample set includes at least one of a third and a fourth sample subset, where the third sample subset is used to train the speech decoding capability of the initial speech decoding unit, and the fourth sample subset is used to train the emotional speech generation capability of the initial speech decoding unit, the above-described method of using the second sample set to train the initial decoding unit in the task processing model to obtain the target decoding unit may include the following steps:

[0084] When the second sample set includes the third sample subset, the initial speech decoding unit in the task processing model is trained using the third sample subset to obtain the target speech decoding unit.

[0085] When the second sample set includes the fourth sample subset, the initial speech decoding unit in the task processing model is trained using the fourth sample subset to obtain the target speech decoding unit.

[0086] When the second sample set includes the third and fourth sample subsets, the initial speech decoding unit in the task processing model is trained using the third and fourth sample subsets to obtain the target speech decoding unit.

[0087] It should be noted that the initial speech decoding unit can be a lightweight decoder with a streaming, non-autoregressive structure, such as a two-layer Transformer structure, thereby supporting real-time and parallel speech decoding generation. Of course, the initial speech decoding unit can also be a non-streaming or autoregressive decoder, depending on the actual situation. This specification does not impose any limitations on this. The implementation method of "using the third sample subset to train the initial speech decoding unit in the task processing model to obtain the target speech decoding unit" can refer to the implementation method of "using the third sample subset to train the initial speech decoding unit in the task processing model to obtain the pre-trained decoding unit". The implementation method of "using the fourth sample subset to train the initial speech decoding unit in the task processing model to obtain the target speech decoding unit" can refer to the implementation method of "using the fourth sample subset to train the pre-trained decoding unit to obtain the target speech decoding unit". This specification will not elaborate further on this aspect in the embodiments.

[0088] In practical applications, there are multiple ways to train the initial speech decoding unit in the task processing model using the third and fourth sample subsets to obtain the target speech decoding unit. The specific method chosen depends on the actual situation, and this specification does not impose any limitations on this approach. In the first possible implementation, the third sample subset can be used to train the initial speech decoding unit in the task processing model to obtain a pre-trained decoding unit; the fourth sample subset can then be used to train the pre-trained decoding unit to obtain the target speech decoding unit. In the second possible implementation, the fourth sample subset can be used to train the initial speech decoding unit in the task processing model to obtain a pre-trained decoding unit; the third sample subset can then be used to train the pre-trained decoding unit to obtain the target speech decoding unit. In the third possible implementation, both the third and fourth sample subsets can be used simultaneously to train the initial speech decoding unit in the task processing model to obtain the target speech decoding unit.

[0089] By applying the scheme of the embodiments in this specification, a second sample set is configured according to the capabilities (speech decoding capability and / or emotional speech generation capability) of the target speech decoding unit, and the initial speech decoding unit in the task processing model is trained using the second sample set, thereby realizing personalized training of the target speech decoding unit and improving the flexibility of the target speech decoding unit.

[0090] In one optional embodiment of this specification, the above-described method of training the initial speech decoding unit in the task processing model using the third and fourth sample subsets to obtain the target speech decoding unit may include the following steps:

[0091] Using the third sample subset, the initial speech decoding unit in the task processing model is trained to obtain the pre-trained decoding unit;

[0092] The pre-trained decoding unit is trained using the fourth sample subset to obtain the target speech decoding unit.

[0093] It should be noted that the pre-trained decoding unit refers to the initial speech decoding unit trained on the third sample subset. At this point, the pre-trained decoding unit has learned some decoding knowledge and possesses speech decoding capabilities. The target speech decoding unit refers to the pre-trained decoding unit after fine-tuning on the fourth sample subset. The target speech decoding unit not only retains the speech decoding capabilities learned from the third sample subset but also acquires the emotional speech generation capabilities learned based on the fourth sample subset. Like the second sample set, the third and fourth sample subsets can include multiple encoded sample features (such as feature data from speech modalities, text modalities, image modalities, or other modalities) and their corresponding expected outputs, or they can include multiple unencoded sample data (such as sample data from speech modalities, text modalities, image modalities, or other modalities) and their corresponding expected outputs.

[0094] In practical applications, the implementation of "using the third sample subset to train the initial speech decoding unit in the task processing model to obtain the pre-trained decoding unit; using the fourth sample subset to train the pre-trained decoding unit to obtain the target speech decoding unit" can refer to the implementation of "using the second sample set to train the initial decoding unit in the task processing model to obtain the target decoding unit". The embodiments in this specification will not be described in detail.

[0095] The scheme implemented in this specification first uses a third sample subset to pre-train the initial speech decoding unit, and then uses a fourth sample subset to further fine-tune the pre-trained model, so that the target speech decoding unit has both speech decoding capability and emotional speech generation capability.

[0096] In one optional embodiment of this specification, taking the third sample subset as an example, which includes a second sample image, a third sample question for the second sample image, and the second sample speech corresponding to the third sample question, the above-mentioned use of the third sample subset to train the initial speech decoding unit in the task processing model to obtain a pre-trained decoding unit may include the following steps:

[0097] The second sample image is input into the target image encoding unit to obtain the features of the second sample image;

[0098] The features of the second sample image and the third sample question are input into the target processing unit for feature fusion to obtain the speech features to be decoded;

[0099] The speech features to be decoded are input into the initial speech decoding unit to obtain the first predicted speech;

[0100] Based on the second sample speech and the first predicted speech, the initial speech decoding unit is trained to obtain a pre-trained decoding unit.

[0101] It should be noted that the second sample image refers to the image dataset used to train the initial speech decoding unit. These images are used as input to the target image encoding unit to extract features. The target image encoding unit refers to the optimized and trained initial image encoding unit, which can effectively extract feature representations from the input images that are helpful for subsequent processing. The second sample image features refer to the feature vectors or representations obtained by the target image encoding unit after processing the second sample images; these features can capture key information in the second sample images. The third sample question refers to the text question related to the second sample image. The target processing unit refers to the trained and optimized initial processing unit, which can integrate data features from different modalities and generate an internal representation suitable for use by the decoding unit. The speech features to be decoded refer to the intermediate representation generated by the target processing unit based on the image features and the related question; this representation is designed for decoding into speech output. The first predicted speech refers to the preliminary speech output generated by the initial speech decoding unit based on the speech features to be decoded, serving as the model's answer to the given image and question. The second sample speech refers to the real speech answer corresponding to the third sample question, used as a standard for evaluating model performance. The second sample speech can be speech from different language types, such as Chinese speech or English speech. A pre-trained decoding unit refers to an initial speech decoding unit that has been adjusted and optimized. The pre-trained decoding unit is trained by comparing the difference between the first predicted speech and the second sample speech, thereby improving the ability of the pre-trained decoding unit to generate accurate speech output.

[0102] In practical applications, there are various methods for obtaining the third sample subset, and the specific method should be selected according to the actual situation. This specification does not impose any limitations on this method in its embodiments. In one possible implementation, the third sample subset can be read from other data acquisition devices or a database. In another possible implementation, a sample text answer to the third sample question can be generated based on the second sample image and the third sample question related to the second sample image. The sample text answer can then be converted into speech to obtain the second sample speech corresponding to the third sample question.

[0103] Furthermore, the implementation method of "inputting the second sample image features and the third sample question into the target processing unit for feature fusion to obtain the speech features to be decoded" can refer to the implementation method of "inputting the sample speech features and the first sample question into the initial processing unit to obtain the first predicted answer" described above. The implementation method of "training the initial speech decoding unit according to the second sample speech and the first predicted speech to obtain the pre-trained decoding unit" can refer to the implementation method of "training the initial speech coding unit according to the first sample answer and the first predicted answer to obtain the target speech coding unit" described above. The embodiments in this specification will not be described in detail again.

[0104] By applying the scheme of the embodiments of this specification, since the target image encoding unit and the target processing unit have been trained and have very good processing capabilities, the data for training the initial speech decoding unit is generated based on the target image encoding unit and the target processing unit, ensuring the accuracy of the second sample image features and the speech features to be decoded, and further guaranteeing the training quality of the pre-trained decoding unit.

[0105] In one optional embodiment of this specification, a fourth sample subset including various emotional information (such as happiness, sadness, anger, etc.) can be constructed. A direct preference optimization algorithm is used to fine-tune the pre-trained decoding unit, enabling it to generate high-quality emotional speech consistent with the context's emotion, improving the emotional coherence of the synthesized speech, and giving the pre-trained decoding unit the ability to generate emotions self-awarely. The fourth sample subset includes expected sample speech with consistent emotion and unexpected sample speech with inconsistent emotion; that is, the fourth sample subset includes both expected and unexpected sample speech. The above-mentioned training of the pre-trained decoding unit using the fourth sample subset to obtain the target speech decoding unit may include the following steps:

[0106] The desired sample speech and the undesired sample speech are input into the pre-trained decoding unit to obtain the second predicted speech;

[0107] Calculate the preference loss value based on the second predicted speech, the expected sample speech, and the unexpected sample speech;

[0108] Based on the preference loss value, the unit parameters of the pre-trained decoding unit are adjusted to obtain the target speech decoding unit.

[0109] It should be noted that the process of training the pre-trained decoding unit using the fourth sample subset can be viewed as an emotional speech direct preference optimization process. The expected sample speech included in the fourth sample subset can be considered as positive sample speech / target sample speech, and the unexpected sample speech can be considered as negative sample speech / control sample speech. The fourth sample subset can be called the emotional dialogue preference sample set, where each sample contains expected sample speech with consistent emotion and unexpected sample speech with inconsistent emotion. For example, if the emotion category is "happy," then the expected sample speech could be positive, with an upward intonation. The unexpected sample speech could be low, weak in tone, or neutral speech without any emotion. The second predicted speech refers to the context-consistent phoneme sequence generated by the pre-trained decoding unit itself.

[0110] In practical applications, when calculating the preference loss value based on the second predicted speech, the desired sample speech, and the undesired sample speech, the average loss of multiple optimized paths in Connectionist Temporal Classification (CTC) can be equated to the preference loss value in speech prediction, thus achieving a transfer from DPO to CTC loss modeling. Furthermore, since the initial speech decoding unit is trained during the speech generation stage, after calculating the preference loss value, the unit parameters of the pre-trained decoding unit can be adjusted based on the preference loss value to obtain the target speech decoding unit.

[0111] By applying the scheme of the embodiments in this specification, the unit parameters of the pre-trained decoding unit are adjusted according to the preference loss value to obtain the target speech decoding unit, so that the target speech decoding unit can accept the input language features and output the emotionally consistent phoneme sequence, ensuring that the subsequent speech synthesis steps can produce natural and emotionally coherent speech.

[0112] Step 108: Construct the trained task processing model based on the target processing unit, the target decoding unit, and at least two target encoding units.

[0113] In practical applications, there are various ways to construct a trained task processing model based on a target processing unit, a target decoding unit, and at least two target encoding units. The specific method chosen depends on the actual situation, and this specification does not impose any limitations on this approach. In one possible implementation, each target encoding unit can be connected to the target processing unit, allowing the encoded features output by each target encoding unit to be used as input to the target processing unit. Similarly, the target processing unit can be connected to the target decoding unit, allowing the fused features output by the target processing unit to be used as input to the target decoding unit, thus obtaining the trained task processing model. In another possible implementation, units in the trained task processing model can be selected from the target decoding unit and at least two target encoding units based on the actual task processing requirements. For example, if the actual task processing requirement is to input image data and generate corresponding speech, then a trained task processing model can be constructed based on the target processing unit, the target speech decoding unit, and the target image encoding unit.

[0114] By applying the scheme of the embodiments of this specification, at least two initial encoding units are set in the model for encoding non-text modal data, which realizes the direct encoding of non-text modal data, thereby preserving the characteristics and information of the original data and avoiding information loss caused by modality conversion. Furthermore, text modal data can be used as an intermediate bridge to realize the alignment processing of at least three modal data, resulting in a task processing model with high accuracy that can handle multimodal data.

[0115] Traditional approaches rely heavily on trimodal data for training of large full-modal models, which suffers from insufficient high-quality data, difficulties in real-time interaction, and poor emotional consistency, thus limiting the model's performance in practical applications. In this specification's embodiments, addressing the challenge of achieving efficient full-modal alignment and real-time emotional speech generation in the absence of trimodal data, a zero-shot full-modal large-model alignment scheme based on language hubs is proposed.

[0116] See Figure 2 , Figure 2 This document illustrates a flowchart of a task processing model training method according to an embodiment of this specification. The task processing model training process includes a full-modal alignment stage and a speech generation stage. The full-modal alignment stage is further divided into a speech-to-text generation stage and an image-to-text generation stage. These stages will be described in detail below.

[0117] Speech-to-text generation stage: The first sample speech is input into the initial speech coding unit to obtain the first sample speech features; the first sample speech features and the first sample question are input into the initial processing unit to obtain the first predicted answer; based on the first sample answer and the first predicted answer, the initial speech coding unit is trained to obtain the target speech coding unit. Features are extracted from the input speech signal using the initial speech coding unit to obtain a continuous speech feature representation. The speech feature representation is aligned and trained with the corresponding text input, enabling the task processing model to possess speech understanding capabilities.

[0118] Image-to-text generation stage: The first sample image is input into the initial image encoding unit to obtain the features of the first sample image; the features of the first sample image and the second sample question are input into the initial processing unit to obtain the second predicted answer; based on the second sample answer and the second predicted answer, the initial image encoding unit and the initial processing unit are trained to obtain the target processing unit and the target image encoding unit. Features are extracted from the input image using the initial image encoding unit to obtain a continuous visual feature representation. The visual feature representation is aligned and trained with the corresponding text input to enhance the image understanding and instruction following capabilities of the task processing model.

[0119] In the speech generation stage: the second sample image is input into the target image encoding unit to obtain the features of the second sample image; the features of the second sample image and the third sample image are input into the target processing unit for feature fusion to obtain the speech features to be decoded; the speech features to be decoded are input into the initial speech decoding unit to obtain the first predicted speech; the initial speech decoding unit is trained based on the second sample speech and the first predicted speech to obtain the pre-trained decoding unit. Further, desired and unwanted sample speech can be input into the pre-trained decoding unit to obtain the second predicted speech; the preference loss value is calculated based on the second predicted speech, the desired sample speech, and the unwanted sample speech; the unit parameters of the pre-trained decoding unit are adjusted based on the preference loss value to obtain the target speech decoding unit.

[0120] The complete task processing model includes a target speech coding unit, a target image coding unit, a target processing unit, and a target speech decoding unit. The target speech coding unit extracts feature representations of the input speech, enabling speech-to-text conversion and understanding. It connects with the target processing unit, giving the task processing model speech comprehension capabilities. The target image coding unit extracts visual features of the input image, enabling image-to-text conversion and understanding. It connects with the target processing unit, giving the task processing model image comprehension and description capabilities. The target processing unit, as the core text generation and comprehension module of the task processing model, processes the input to generate a text response. It can interface with both the target speech coding unit and the target image coding unit to achieve multimodal feature fusion. The target speech decoding unit receives the output of the target processing unit and generates the corresponding speech signal. The target speech decoding unit can adopt a streaming, non-autoregressive structure to support real-time speech generation and includes multiple Transformer layers to improve training stability and generation performance.

[0121] It is worth noting that the full-modal alignment stage is divided into a speech-to-text generation stage and an image-to-text generation stage. By utilizing language as an intermediate bridge and employing a large amount of speech-to-text and image-to-text data, zero-sample generalization from vision to speech is achieved, ultimately reaching zero-sample full-modal alignment. This overcomes the dependence on trimodal data and reduces the cost of model training. By introducing a lightweight initial speech decoding unit, computational resource consumption is reduced, facilitating the deployment of task processing models in resource-constrained environments. The target speech decoding unit can adopt a streaming, non-autoregressive structure, supporting parallel decoding, significantly reducing speech generation latency and achieving real-time speech generation. By training the target speech decoding unit using a direct preference optimization method, the model can incorporate emotional factors during speech generation, generating high-quality emotional speech consistent with the context, enhancing the naturalness and emotional resonance of human-computer interaction.

[0122] Experiments have demonstrated that the task processing model trained using the embodiments in this specification can receive multimodal data input, including images, text, and speech, and can stream text and emotional speech. Compared to other traditional models, the task processing model shows significant performance improvements on multiple evaluation sets.

[0123] Refer to Table 1, which shows the experimental data of the models based on the Omn i-Bench image-text-audio evaluation set. Model A is an arbitrarily generated pre-trained model; Model B is a video understanding pre-trained model; Model C is a unified input-output large model (1.1 billion parameters); Model D is a unified input-output super-large model (3.2 billion parameters); Model E is a unified input-output 2 super-large model (6.8 billion parameters); Model F is a visual converter model; and Model G is the task processing model trained in the embodiments of this specification. Table 1 shows that the average accuracy of the task processing model trained in the embodiments of this specification increased from 33.45% to 35.47%, an improvement of 2.02%.

[0124] Table 1. Experimental data of the model based on the Omn i-Bench benchmark dataset and the text-to-speech evaluation dataset.

[0125]

[0126] Refer to Table 2, which shows the experimental data of the models based on two ASR evaluation sets. Model A is a speech T5 model; Model B is a general hearing model; Model C is a mini multimodal model; Model D is an audio processing model; Models A, B, C, D, and E are large speech models. Model E is an arbitrary generation pre-trained model. Model F is a visual converter model; Model G is an emotional speech evaluation model. Model H is the task processing model trained in the embodiments of this specification. Models E, F, G, and H are multimodal large language models. Table 2 shows that the ASR error rate of the task processing model trained in the embodiments of this specification decreased from 12.2% to 7.4% on the Chinese evaluation set (including online text and conference text), a decrease of 4.8%; and the ASR error rate on the English evaluation set (including pre-processed text and other texts) decreased from 8.1% to 2.4%, a decrease of 5.7%.

[0127] Table 2. Experimental data of the model based on two ASR evaluation sets.

[0128]

[0129] See Table 3, which presents experimental data for models based on seven image-text evaluation sets (MMB, MMBCN, Hall Bench, MathVistaM, MMMUV, AI2D, and RWQA). Model A is a pre-trained generative model; Model B is a minor version pre-trained generative model; Models A and B are proprietary models. Model C is a miniaturized multimodal pre-trained model (8 billion parameters); Model D is a visual-language chat model (7 billion parameters); Model E is a multimodal model (7 billion parameters); Model F is an emotional speech evaluation model (8 billion parameters); Models C, D, E, and F are open-source weighted models. Model G is a large-scale pre-trained model (8 billion parameters). Model H is a multimodal data processing model (8 billion parameters); Model I is a visual converter model. Model J is a task processing model trained using the embodiments in this specification. As can be seen from Table 3, the task processing model trained by the embodiments of this specification increased the average accuracy from 58.5% to 60.8% on the seven image and text evaluation sets, an improvement of 2.3%.

[0130] Table 3. Experimental data of the model based on seven text and image evaluation sets.

[0131]

[0132]

[0133] See Table 4, which presents the experimental data for the model based on the Emotional Speech Generation Evaluation Set (EO2S-9K). The Emotional Speech Generation Evaluation Set is a dataset containing approximately 9000 speech samples labeled with different emotional tags, such as happiness, sadness, and anger. Table 4 shows that after optimization with the self-perceived emotion DPO, the task processing model trained in the embodiments of this specification achieved an increase in emotional speech generation accuracy from 55.7% to 59.1%, a 3.4% improvement.

[0134] Table 4. Experimental data of the model based on the evaluation set of emotion-based speech generation.

[0135] Model language anger disgust fear hapiness neutral other sad surprise overall Before DPO Chinese 47.0 49.7 78.5 88.7 94.6 33.9 45.5 23.8 57.7 After DPO Chinese 63.4 60.1 83.6 90.1 95.2 45.7 64.7 54.6 62.9 Before DPO English 43.6 44.8 63.4 90.4 95.1 31.6 39.7 21.5 53.7 After DPO Chinese 60.2 57.7 76.4 92.3 96.4 44.3 61.2 51.8 55.3

[0136] See Figure 3 , Figure 3 This specification shows a flowchart of a task processing method according to an embodiment, which specifically includes the following steps:

[0137] Step 302: Obtain the task data of the target task, wherein the task data includes at least one of text data, voice data and image data.

[0138] Step 304: Input the task data into the task processing model to obtain the task processing result. The task processing model is trained based on the task processing model training method.

[0139] It should be noted that the target task can be of different types, such as natural language text understanding and generation, image understanding and generation, speech recognition, understanding and generation, multimodal data understanding and generation, etc. The target task can also be a task in different scenarios, such as a task in an intelligent customer service scenario, a task in an emotional dialogue scenario, a task in a role-playing scenario, etc. The task processing result corresponds to the specific task requirement. If the task requirement is to generate a text result, the task processing result will be a text modal result; if the task requirement is to generate a speech result, the task processing result will be a speech modal result.

[0140] In practical applications, there are various ways to obtain task data for a target task, and the specific method chosen depends on the actual situation. This specification does not impose any limitations on these methods in its embodiments. In one possible implementation, task data for the target task can be received from a user's terminal device. In another possible implementation, task data for the target task can be read from other data acquisition devices or databases.

[0141] It is worth noting that if the task data includes text data, the text data can be input into the target processing unit in the task processing model to obtain the output of the target processing unit; the output of the target processing unit can then be input into the target decoding unit of the task processing model to obtain the task processing result. If the task data includes speech data, the speech data can be input into the target speech coding unit in the task processing model to obtain the output of the target speech coding unit; the output of the target speech coding unit can then be input into the target processing unit to obtain the output of the target processing unit; the output of the target processing unit can then be input into the target decoding unit of the task processing model to obtain the task processing result. If the task data includes image data, the image data can be input into the target image coding unit in the task processing model to obtain the output of the target image coding unit; the output of the target image coding unit can then be input into the target processing unit to obtain the output of the target processing unit; the output of the target processing unit can then be input into the target decoding unit of the task processing model to obtain the task processing result.

[0142] By applying the solutions in the embodiments of this specification, the task processing model can directly encode non-textual modal data, thereby preserving the characteristics and information of the original data and avoiding information loss caused by modality conversion. Therefore, the accuracy of the task processing results is improved.

[0143] The following is in conjunction with the appendix Figure 4Taking the application of the task processing method provided in this specification in an emotional speech generation scenario as an example, the task processing method will be further explained. Figure 4 This specification shows a flowchart of an embodiment of an emotional speech generation method, which specifically includes the following steps:

[0144] Step 402: Receive emotional speech generation data sent by the terminal device, wherein the emotional speech generation data includes at least one of text data, voice data, and image data.

[0145] Step 404: Input the emotional speech generation data into the task processing model to obtain the target emotional speech, wherein the task processing model is trained based on the task processing model training method.

[0146] Step 406: Feedback the target emotional voice to the terminal device.

[0147] It should be noted that emotional speech generation data refers to input data containing specific emotional information, used to guide the model in generating speech output with corresponding emotional coloring. The task processing model includes a target speech decoding unit, which is directly optimized based on desired and undesired sample speech. Target emotional speech refers to the speech output generated by the task processing model that conforms to the emotional speech generation data. Target emotional speech can convey predetermined emotional states, such as happiness, sadness, anger, and surprise, to enhance the realism and emotional resonance of communication.

[0148] By applying the solutions in the embodiments of this specification, since the task processing model can directly encode non-textual modal data, it retains the characteristics and information of the original data and avoids information loss caused by modality conversion. Therefore, it improves the accuracy of task processing results. Furthermore, through the task processing model, it can accurately understand the user's emotional speech generation intention and generate target emotional speech that is warm, emotional, anthropomorphic, and provides a better experience.

[0149] Considering the large number of model parameters in the task processing model and the limited computing resources of the terminal device, the task processing method proposed in the embodiments of this specification can be applied to, for example... Figure 5 The task processing system shown is not limited to this. See also Figure 5 , Figure 5 This specification illustrates an architecture diagram of a task processing system provided in one embodiment of the present specification. The task processing system may include a terminal device 502 and a server 504.

[0150] Terminal device 502 is used to send task data of a target task to server 504, wherein the task data includes at least one of text data, voice data and image data;

[0151] The server 504 is used to input task data into the task processing model, obtain task processing results, and send task processing results to the terminal device 502.

[0152] Terminal device 502 is also used to receive task processing results sent by server 504.

[0153] It is worth noting that the task processing model includes a target processing unit, a target decoding unit, and at least two target encoding units. The target decoding unit is trained on the initial decoding unit in the task processing model based on the second sample set. The target processing unit and at least two target encoding units are trained on the initial processing unit and at least two initial encoding units in the task processing model based on the first sample set. The at least two initial encoding units are used to encode non-textual modal data, and different initial encoding units encode data of different modalities. The initial processing unit is used to perform feature fusion on the outputs of the at least two initial encoding units.

[0154] like Figure 5 As shown, the task processing model is deployed in server 504. Server 504 can connect to one or more terminal devices 502 via a local area network (LAN), wide area network (WAN), Internet, or other types of data network. Terminal devices 502 may include, but are not limited to, smartphones, tablets, laptops, PDAs, personal computers, smart home devices, and in-vehicle devices. Terminal devices 502 can also interact with users through a graphical user interface to invoke the task processing model, thereby implementing the task processing method provided in the embodiments of this specification.

[0155] It is worth noting that the task processing methods provided in the embodiments of this specification are generally executed by the server. However, in other embodiments of this specification, if the terminal device's operating resources can meet the deployment and operating conditions of the task processing model, the terminal device may also have similar functions to the server, thereby executing the task processing methods provided in the embodiments of this specification. In other embodiments, the task processing methods provided in the embodiments of this specification may also be executed jointly by the terminal device and the server.

[0156] See Figure 6 , Figure 6 This specification illustrates a flowchart of an information processing method based on a task processing model, provided in one embodiment. This method, applied to a task platform, specifically includes the following steps:

[0157] Step 602: Receive the model request sent by the terminal device.

[0158] Step 604: Based on the model request, determine the target task processing model from multiple task processing models, wherein the multiple task processing models are trained based on the task processing model training method.

[0159] It should be noted that the target task processing model is a task processing model applicable to the target scenario. The model request includes at least one of the following: the scenario identifier of the target scenario, the scenario input data of the target scenario, and model specification parameters. There are multiple ways to determine the target task processing model from multiple task processing models based on the model request; the specific method chosen depends on the actual situation, and this specification does not impose any limitations on this method. In one possible implementation of this specification, the corresponding target task processing model can be searched from at least one task processing model included in the model library based on the model request. In another possible implementation, the target task processing model can be trained and obtained based on the model request. In yet another optional implementation, the target task processing model can be constructed based on the model request.

[0160] For example, based on the scene identifier of the target scene, at least one pre-trained task processing model can be searched from the model library. Then, based on the model specification parameters, an initial task processing model can be selected from the at least one task processing model. Finally, based on the scene input data of the target scene, the selected initial task processing model is trained to obtain a target task processing model suitable for user needs. The at least one task processing model can be based on... Figure 1 The task processing model shown was trained using the training method described in this specification, and will not be repeated in the embodiments.

[0161] The solutions implemented in the embodiments of this specification are adapted to user needs to obtain target task processing models, realize personalized model services, provide users with an efficient, flexible and easy-to-use model service method, and improve user experience.

[0162] In one optional embodiment of this specification, the model request includes a scene identifier of the target scene; the process of determining the target task processing model from multiple task processing models based on the model request may include the following steps:

[0163] Based on the scene identifier of the target scene, the target task processing model suitable for the target scene is searched from the model library. The model library stores multiple task processing models suitable for different task processing scenarios.

[0164] It should be noted that scene identifiers are unique or specific labels used to distinguish different task scenarios. The model library is a database for storing and managing various pre-trained deep learning models. Multiple task processing models adapted to different task scenarios cover different application scenarios and needs. The model library allows users to select appropriate models according to their needs, or directly call models for task processing through application programming interfaces.

[0165] Multiple task processing models adapted to different task scenarios are stored in the model library. Each model is optimized for a specific application environment. Any task processing model is based on... Figure 1 The training method shown is used to train the model obtained from the task processing model. For example, based on the scene identifier "emotional speech generation" of the target scene, a target task processing model suitable for the emotional speech generation scene can be found from the model library.

[0166] By applying the solutions in the embodiments of this specification, based on scenario requirements, the target task processing model adapted to the scenario is accurately found through scenario identification, making task processing more accurate and more scenario-appropriate, thereby improving user experience and task processing quality.

[0167] In one optional embodiment of this specification, the model request includes scene input data of the target scene; the above-mentioned determination of the target task processing model from multiple task processing models based on the model request may include the following steps:

[0168] From multiple task processing models, determine the task processing model that is suitable for the target scenario;

[0169] Based on the scene input data of the target scene, a task processing model adapted to the target scene is trained to obtain the target task processing model.

[0170] It should be noted that different task processing models are adapted to different scenarios. For example, task processing model 1 is suitable for scenarios 1 and 2, while task processing model 2 is suitable for scenarios 2 and 3. If the target scenario is scenario 1, then the task processing model adapted to the target scenario is task processing model 1. A task processing model adapted to a target scenario may not only be applicable to the target scenario but also to other scenarios, making it a general task processing model applicable to different scenarios. While a task processing model adapted to a target scenario can be used for task processing, the results may not be ideal. In such cases, the task processing model adapted to the target scenario can be optimized based on the scenario input data of the target scenario. For example, optimizing the task processing model adapted to the target scenario based on the scenario input data of the emotional speech generation scenario can yield a target task processing model suitable for the emotional speech generation scenario. The scenario input data of the target scenario can be understood as the sample data of the sample tasks in the target scenario, including a first sample set and a second sample set. The method for training the task processing model adapted to the target scenario based on the scenario input data of the target scenario to obtain the target task processing model can be found in [reference needed]. Figure 1 The training method of the task processing model shown in this specification will not be described again in the embodiments.

[0171] By applying the solutions in the embodiments of this specification, based on scenario requirements, a task processing model adapted to the target scenario is further trained using scenario input data to obtain a target task processing model adapted to the target scenario. This makes the target task processing model more closely fit the target scenario, thereby improving user experience and task processing quality.

[0172] In one optional embodiment of this specification, the model request includes model specification parameters; the process of determining the target task processing model from multiple task processing models based on the model request may include the following steps:

[0173] Based on the model specification parameters, the corresponding target task processing model is searched from the model library, which stores multiple task processing models with different model specification parameters.

[0174] Based on the model specification parameters, the corresponding target task processing model is searched from the model library, which stores multiple task processing models with different model specification parameters.

[0175] It's important to note that model specifications refer to the various parameters that define the model's structure and behavior. These parameters can be broadly categorized into two types: model parameters (learnable parameters) and hyperparameters. Model parameters are those automatically adjusted during model training via backpropagation, including but not limited to weight matrices and biases. For example, in a simple fully connected layer, the weight matrix is ​​a two-dimensional tensor connecting neurons in the input and output layers; the biases are one-dimensional vectors providing additional offset values ​​for each output neuron. Hyperparameters are parameters set before model training begins, controlling the model's learning process and architecture. Hyperparameters include, but are not limited to, the learning rate and the number of neurons per layer, chosen based on specific requirements.

[0176] By applying the solutions in the embodiments of this specification, based on the model specification parameters, the corresponding target task processing model can be accurately found, ensuring the efficient and stable operation of the target task processing model and improving the user experience.

[0177] In one optional embodiment of this specification, after determining the target task processing model from multiple task processing models based on the model request, the following steps may be further included:

[0178] Deploy the target task processing model, and build a task processing interface based on the target task processing model so that the terminal device can schedule the target task processing model to execute the target task.

[0179] It should be noted that the task processing interface is an interactive programming interface for the terminal device to schedule the target task processing model to process the target task, and it is usually provided in the form of an application programming interface (API). Through the task processing interface, users can input task data for the target task, such as emotional speech generation data, to perform emotional speech generation.

[0180] In practical applications, there are various ways to deploy the target task processing model, and the specific method should be chosen based on the actual situation. This specification does not impose any limitations on this approach. One possible implementation of this specification is to deploy the target task processing model on cloud-side devices using infrastructure provided by a cloud service provider. Another possible implementation of this specification is to deploy the target task processing model on edge devices using a lightweight framework. For example, the target task processing model can be deployed on a distributed system, and a task processing interface can be built based on the target task processing model and provided to terminal devices, enabling the terminal devices to schedule the target task processing model to execute the target task.

[0181] By applying the solutions provided in the embodiments of this specification, deploying the target task processing model, and building a task processing interface based on the target task processing model, terminal devices can efficiently call the target task processing model, thereby improving the processing quality and response speed of the target task.

[0182] See Figure 7 , Figure 7 This specification shows a schematic diagram of the structure of a task platform 700 provided in one embodiment of the present specification. The task platform 700 includes a request interface 702 and a response unit 704.

[0183] Request interface 702 is used to receive a model request sent by a terminal device, wherein the model request includes at least one of the following: scene identifier of the target scene, scene input data of the target scene, and model specification parameters.

[0184] The response unit 704 is used to determine the target task processing model from multiple task processing models based on the model request, wherein the multiple task processing models are trained based on the task processing model training method.

[0185] In one optional embodiment of this specification, the task platform further includes a task processing interface, which is constructed based on the target task processing model.

[0186] The task processing interface is used to allow terminal devices to schedule and execute target tasks.

[0187] By applying the solutions in the embodiments of this specification, the task platform adapts to user needs to obtain target task processing models, realizes personalized model services, provides users with an efficient, flexible and easy-to-use model service platform, and improves user experience.

[0188] The above is an illustrative scheme of a task platform according to this embodiment. It should be noted that the technical solution of this task platform and the technical solution of the information processing method based on the task processing model described above belong to the same concept. For details not described in detail in the technical solution of the task platform, please refer to the description of the technical solution of the information processing method based on the task processing model described above.

[0189] Corresponding to the above-described embodiments of the task processing model training method, this specification also provides embodiments of the task processing model training apparatus. Figure 8 A schematic diagram of a task processing model training device provided in one embodiment of this specification is shown.

[0190] like Figure 8 As shown, the device includes:

[0191] The first acquisition module 802 is configured to acquire a first sample set and a second sample set;

[0192] The first training module 804 is configured to train the initial processing unit and at least two initial encoding units in the task processing model using the first sample set to obtain the target processing unit and at least two target encoding units. The at least two initial encoding units are used to encode non-textual modal data, and different initial encoding units encode data of different modalities respectively. The initial processing unit is used to perform feature fusion on the output of the at least two initial encoding units.

[0193] The second training module 806 is configured to train the initial decoding unit in the task processing model using the second sample set to obtain the target decoding unit;

[0194] Module 808 is configured to build a trained task processing model based on a target processing unit, a target decoding unit, and at least two target encoding units.

[0195] Optionally, at least two initial coding units include an initial speech coding unit and an initial image coding unit, and the first sample set includes a first sample subset for the initial speech coding unit and a second sample subset for the initial image coding unit; the first training module 804 is further configured to train the initial speech coding unit using the first sample subset to obtain a target speech coding unit; and to train the initial image coding unit and the initial processing unit using the second sample subset to obtain a target processing unit and a target image coding unit.

[0196] Optionally, the first sample subset includes a first sample speech, a first sample question for the first sample speech, and a first sample answer for the first sample question; the first training module 804 is further configured to input the first sample speech into an initial speech coding unit to obtain sample speech features; input the sample speech features and the first sample question into an initial processing unit to obtain a first predicted answer; and train the initial speech coding unit based on the first sample answer and the first predicted answer to obtain a target speech coding unit.

[0197] Optionally, the second sample subset includes a first sample image, a second sample question for the first sample image, and a second sample answer for the second sample question; the first training module 804 is further configured to input the first sample image into an initial image encoding unit to obtain first sample image features; input the first sample image features and the second sample question into an initial processing unit to obtain a second predicted answer; and train the initial image encoding unit and the initial processing unit based on the second sample answer and the second predicted answer to obtain a target processing unit and a target image encoding unit.

[0198] Optionally, the initial decoding unit includes an initial speech decoding unit, and the second sample set includes at least one of a third sample subset and a fourth sample subset, wherein the third sample subset is used to train the speech decoding capability of the initial speech decoding unit, and the fourth sample subset is used to train the emotional speech generation capability of the initial speech decoding unit; the second training module 806 is further configured to, when the second sample set includes the third sample subset, use the third sample subset to train the initial speech decoding unit in the task processing model to obtain a target speech decoding unit; when the second sample set includes the fourth sample subset, use the fourth sample subset to train the initial speech decoding unit in the task processing model to obtain a target speech decoding unit; and when the second sample set includes both the third and fourth sample subsets, use both the third and fourth sample subsets to train the initial speech decoding unit in the task processing model to obtain a target speech decoding unit.

[0199] Optionally, the second training module 806 is further configured to train the initial speech decoding unit in the task processing model using a third sample subset to obtain a pre-trained decoding unit; and to train the pre-trained decoding unit using a fourth sample subset to obtain a target speech decoding unit.

[0200] Optionally, the third sample subset includes the second sample image, the third sample question for the second sample image, and the second sample speech corresponding to the third sample question; the second training module 806 is further configured to input the second sample image into the target image encoding unit to obtain the features of the second sample image; input the features of the second sample image and the third sample question into the target processing unit for feature fusion to obtain the speech features to be decoded; input the speech features to be decoded into the initial speech decoding unit to obtain the first predicted speech; and train the initial speech decoding unit based on the second sample speech and the first predicted speech to obtain the pre-trained decoding unit.

[0201] Optionally, the fourth sample subset includes expected sample speech and unexpected sample speech; the second training module 806 is further configured to input the expected sample speech and unexpected sample speech into the pre-trained decoding unit to obtain the second predicted speech; calculate the preference loss value based on the second predicted speech, the expected sample speech and the unexpected sample speech; and adjust the unit parameters of the pre-trained decoding unit based on the preference loss value to obtain the target speech decoding unit.

[0202] By applying the scheme of the embodiments of this specification, at least two initial encoding units are set in the model for encoding non-text modal data, which realizes the direct encoding of non-text modal data, thereby preserving the characteristics and information of the original data and avoiding information loss caused by modality conversion. Furthermore, text modal data can be used as an intermediate bridge to realize the alignment processing of at least three modal data, resulting in a task processing model with high accuracy that can handle multimodal data.

[0203] The above is a schematic scheme of a task processing model training device according to this embodiment. It should be noted that the technical solution of this task processing model training device and the technical solution of the task processing model training method described above belong to the same concept. For details not described in detail in the technical solution of the task processing model training device, please refer to the description of the technical solution of the task processing model training method described above.

[0204] Corresponding to the above-described task processing method embodiments, this specification also provides embodiments of a task processing apparatus. Figure 9 A schematic diagram of a task processing apparatus according to one embodiment of this specification is shown. Figure 9 As shown, the device includes:

[0205] The second acquisition module 902 is configured to acquire task data of the target task, wherein the task data includes at least one of text data, voice data and image data;

[0206] The first input module 904 is configured to input task data into the task processing model to obtain task processing results, wherein the task processing model is trained based on the task processing model training method.

[0207] By applying the solutions in the embodiments of this specification, the task processing model can directly encode non-textual modal data, thereby preserving the characteristics and information of the original data and avoiding information loss caused by modality conversion. Therefore, the accuracy of the task processing results is improved.

[0208] The above is an illustrative scheme of a task processing device according to this embodiment. It should be noted that the technical solution of this task processing device and the technical solution of the task processing method described above belong to the same concept. For details not described in detail in the technical solution of the task processing device, please refer to the description of the technical solution of the task processing method described above.

[0209] Corresponding to the above embodiments of the emotional speech generation method, this specification also provides embodiments of the emotional speech generation apparatus. Figure 10 A schematic diagram of an emotional speech generation device according to one embodiment of this specification is shown. Figure 10 As shown, the device includes:

[0210] The first receiving module 1002 is configured to receive emotional speech generation data sent by the terminal device, wherein the emotional speech generation data includes at least one of text data, voice data and image data;

[0211] The second input module 1004 is configured to input the emotional speech generation data into the task processing model to obtain the target emotional speech, wherein the task processing model is trained based on the task processing model training method.

[0212] Feedback module 1006 is configured to feed back the target emotional voice to the terminal device.

[0213] By applying the solutions in the embodiments of this specification, since the task processing model can directly encode non-textual modal data, it retains the characteristics and information of the original data and avoids information loss caused by modality conversion. Therefore, it improves the accuracy of task processing results. Furthermore, through the task processing model, it can accurately understand the user's emotional speech generation intention and generate target emotional speech that is warm, emotional, anthropomorphic, and provides a better experience.

[0214] The above is an illustrative scheme of an emotional speech generation device according to this embodiment. It should be noted that the technical solution of this emotional speech generation device and the technical solution of the emotional speech generation method described above belong to the same concept. For details not described in detail in the technical solution of the emotional speech generation device, please refer to the description of the technical solution of the emotional speech generation method described above.

[0215] Corresponding to the above-described embodiments of the information processing method based on the task processing model, this specification also provides embodiments of the information processing apparatus based on the task processing model. Figure 11 A schematic diagram of the structure of an information processing device based on a task processing model provided in one embodiment of this specification is shown. Figure 11 As shown, the device is applied to a mission platform and includes:

[0216] The second receiving module 1102 is configured to receive model requests sent by the terminal device;

[0217] The determination module 1104 is configured to determine the target task processing model from multiple task processing models based on a model request, wherein the multiple task processing models are trained based on a task processing model training method.

[0218] Optionally, the model request includes a scene identifier of the target scene; the determination module 1104 is further configured to search for a target task processing model suitable for the target scene from the model library based on the scene identifier of the target scene, wherein the model library stores multiple task processing models suitable for different task processing scenes.

[0219] Optionally, the model request includes scene input data of the target scene; the determination module 1104 is further configured to determine a task processing model suitable for the target scene from multiple task processing models; and to train the task processing model suitable for the target scene based on the scene input data of the target scene to obtain the target task processing model.

[0220] Optionally, the model request includes model specification parameters; the determination module 1104 is further configured to search for the corresponding target task processing model from the model library based on the model specification parameters, wherein the model library stores multiple task processing models with different model specification parameters.

[0221] Optionally, the device further includes: a deployment module configured to deploy a target task processing model and, based on the target task processing model, construct a task processing interface to enable the terminal device to schedule the target task processing model to execute the target task.

[0222] The solutions implemented in the embodiments of this specification are adapted to user needs to obtain target task processing models, realize personalized model services, provide users with an efficient, flexible and easy-to-use model service method, and improve user experience.

[0223] The above is an illustrative scheme of an information processing device based on a task processing model according to this embodiment. It should be noted that the technical solution of this information processing device based on a task processing model belongs to the same concept as the technical solution of the information processing method based on a task processing model described above. For details not described in detail in the technical solution of the information processing device based on a task processing model, please refer to the description of the technical solution of the information processing method based on a task processing model described above.

[0224] Figure 12 A structural block diagram of a computing device 1200 according to an embodiment of this specification is shown. The computing device 1200 includes a memory 1210 and a processor 1220; the memory 1210 stores computer programs / instructions, and the processor 1220 executes the computer programs / instructions, which, when executed by the processor 1220, implement the steps of the aforementioned task processing model training method, task processing method, emotional speech generation method, or information processing method based on a task processing model.

[0225] In one or more embodiments of this specification, the computing device can be understood as an integrated smart terminal, including but not limited to a server, desktop computer, personal computer (PC), all-in-one model machine, mobile phone, tablet computer or other portable smart terminal, etc., and the computing device may have the model described in the above embodiments of this application pre-installed.

[0226] Specifically, this computing device can pre-install various types of models, including but not limited to models in natural language processing, visual processing, speech processing, code processing, and multimodal task processing, thus providing diverse model selection. In different product forms, this computing device can support one or more model usage methods, including but not limited to model training, model invocation, model fine-tuning, model deployment, model inference, and application. In some product forms, this computing device also supports model management, including but not limited to multi-type model management (supporting the management of discriminative, generative, and other model types), model version control (supporting the control of different model versions), and model evaluation (evaluating model performance and effectiveness based on model evaluation tools). In other product forms, this computing device can also create applications based on models, providing application programming interface (API) invocation capabilities. Models can be invoked into created applications through the API interface, and application management tools are provided for application management and monitoring.

[0227] Furthermore, the computing device may also include data management (supporting the creation and management of model tuning datasets), a training center (providing abundant training resources to help users learn and master artificial intelligence technology), and basic control capabilities (providing enterprise-level basic control capabilities to ensure the security and efficient operation of the system). Through the above functions, it provides a comprehensive and integrated device for artificial intelligence development, training, deployment, and application.

[0228] Figure 13 A structural block diagram of an electronic device 1300 according to an embodiment of this specification is shown. A memory 1310 and a processor 1320 are connected via a bus 1330. The memory 1310 stores computer programs / instructions, and the processor 1320 executes the computer programs / instructions. When executed by the processor 1320, the computer programs / instructions implement the steps of the task processing model training method, task processing method, emotional speech generation method, or information processing method based on the task processing model described above.

[0229] Specifically, the components of the electronic device 1300 include, but are not limited to, a memory 1310 and a processor 1320. The processor 1320 and the memory 1310 can be connected via a bus 1330. The electronic device 1300 may also include an access device 1340, which enables the electronic device 1300 to communicate with a database 1350 storing data via one or more networks 1360. Examples of such networks include a Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the Internet. Access device 1340 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 wireless local area network (WLAN) interface, a Wi-MAX (World Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0230] In one embodiment of this specification, the above-described components of the electronic device 1300 and Figure 13 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 13 The block diagram of the electronic device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0231] Electronic device 1300 can be any type of stationary or mobile electronic device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable electronic devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary electronic devices such as desktop computers or PCs. Electronic device 1300 can also be a mobile or stationary electronic device.

[0232] The above is an illustrative scheme of an electronic device according to this embodiment. It should be noted that the technical solution of this electronic device belongs to the same concept as the above-mentioned task processing model training method, task processing method, emotional speech generation method, and information processing method based on task processing model. For details not described in detail in the technical solution of the electronic device, please refer to the description of the above-mentioned task processing model training method, task processing method, emotional speech generation method, or information processing method based on task processing model.

[0233] An embodiment of this specification also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the task processing model training method, task processing method, emotional speech generation method, or information processing method based on the task processing model described above.

[0234] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium belongs to the same concept as the technical solutions of the task processing model training method, task processing method, emotional speech generation method, and information processing method based on the task processing model described above. Details not described in detail in the technical solution of the storage medium can be found in the descriptions of the technical solutions of the task processing model training method, task processing method, emotional speech generation method, or information processing method based on the task processing model.

[0235] An embodiment of this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described task processing model training method, task processing method, emotional speech generation method, or information processing method based on a task processing model.

[0236] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product belongs to the same concept as the above-mentioned task processing model training method, task processing method, emotional speech generation method, and information processing method based on task processing model. For details not described in detail in the technical solution of the computer program product, please refer to the description of the above-mentioned task processing model training method, task processing method, emotional speech generation method, or information processing method based on task processing model.

[0237] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0238] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0239] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0240] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0241] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A method for training a task processing model, comprising: Obtain the first and second sample sets; Using the first sample set, the initial processing unit and at least two initial encoding units in the task processing model are trained to obtain the target processing unit and at least two target encoding units. The at least two initial encoding units are used to encode non-textual modal data, and different initial encoding units encode data of different modalities respectively. The initial processing unit is used to perform feature fusion on the outputs of the at least two initial encoding units. Using the second sample set, the initial decoding unit in the task processing model is trained to obtain the target decoding unit; A trained task processing model is constructed based on the target processing unit, the target decoding unit, and the at least two target encoding units.

2. The method according to claim 1, wherein the at least two initial coding units include an initial speech coding unit and an initial image coding unit, and the first sample set includes a first sample subset for the initial speech coding unit and a second sample subset for the initial image coding unit; The step of training the initial processing unit and at least two initial encoding units in the task processing model using the first sample set to obtain the target processing unit and at least two target encoding units includes: Using the first sample subset, the initial speech coding unit is trained to obtain the target speech coding unit; Using the second sample subset, the initial image coding unit and the initial processing unit are trained to obtain the target processing unit and the target image coding unit.

3. The method according to claim 2, wherein the first sample subset includes a first sample speech, a first sample question for the first sample speech, and a first sample answer to the first sample question; The step of training the initial speech coding unit using the first sample subset to obtain the target speech coding unit includes: The first sample speech is input into the initial speech coding unit to obtain sample speech features; The sample speech features and the first sample question are input into the initial processing unit to obtain the first predicted answer; The initial speech coding unit is trained based on the first sample answer and the first predicted answer to obtain the target speech coding unit.

4. The method according to claim 2, wherein the second sample subset includes a first sample image, a second sample question for the first sample image, and a second sample answer to the second sample question; The step of training the initial image coding unit and the initial processing unit using the second sample subset to obtain the target processing unit and the target image coding unit includes: The first sample image is input into the initial image encoding unit to obtain the features of the first sample image; The first sample image features and the second sample question are input into the initial processing unit to obtain the second predicted answer; Based on the second sample answer and the second predicted answer, the initial image encoding unit and the initial processing unit are trained to obtain the target processing unit and the target image encoding unit.

5. The method according to claim 1, wherein the initial decoding unit comprises an initial speech decoding unit, the second sample set comprises at least one of a third sample subset and a fourth sample subset, the third sample subset is used to train the speech decoding capability of the initial speech decoding unit, and the fourth sample subset is used to train the emotional speech generation capability of the initial speech decoding unit. The step of training the initial decoding unit in the task processing model using the second sample set to obtain the target decoding unit includes: If the second sample set includes the third sample subset, the initial speech decoding unit in the task processing model is trained using the third sample subset to obtain the target speech decoding unit; If the second sample set includes the fourth sample subset, the initial speech decoding unit in the task processing model is trained using the fourth sample subset to obtain the target speech decoding unit. When the second sample set includes the third sample subset and the fourth sample subset, the initial speech decoding unit in the task processing model is trained using the third sample subset and the fourth sample subset to obtain the target speech decoding unit.

6. The method according to claim 5, wherein training the initial speech decoding unit in the task processing model using the third sample subset and the fourth sample subset to obtain the target speech decoding unit includes: Using the third sample subset, the initial speech decoding unit in the task processing model is trained to obtain a pre-trained decoding unit; The pre-trained decoding unit is trained using the fourth sample subset to obtain the target speech decoding unit.

7. The method according to claim 6, wherein the third sample subset includes a second sample image, a third sample question for the second sample image, and a second sample speech corresponding to the third sample question; The step of training the initial speech decoding unit in the task processing model using the third sample subset to obtain a pre-trained decoding unit includes: The second sample image is input into the target image encoding unit to obtain the features of the second sample image; The second sample image features and the third sample question are input into the target processing unit for feature fusion to obtain the speech features to be decoded. The speech features to be decoded are input into the initial speech decoding unit to obtain the first predicted speech; The initial speech decoding unit is trained based on the second sample speech and the first predicted speech to obtain the pre-trained decoding unit.

8. The method according to claim 6, wherein the fourth sample subset includes desired sample speech and undesired sample speech; The step of training the pre-trained decoding unit using the fourth sample subset to obtain the target speech decoding unit includes: The desired sample speech and the unwanted sample speech are input into the pre-trained decoding unit to obtain the second predicted speech; Calculate the preference loss value based on the second predicted speech, the expected sample speech, and the unexpected sample speech; Based on the preference loss value, the unit parameters of the pre-trained decoding unit are adjusted to obtain the target speech decoding unit.

9. A task processing method, comprising: Acquire task data for the target task, wherein the task data includes at least one of text data, voice data, and image data; The task data is input into the task processing model to obtain the task processing result, wherein the task processing model is trained based on the training method described in any one of claims 1 to 8.

10. A method for generating emotional speech, comprising: The receiver receives emotional voice generation data sent by the terminal device, wherein the emotional voice generation data includes at least one of text data, voice data, and image data; The emotional speech generation data is input into the task processing model to obtain the target emotional speech, wherein the task processing model is trained based on the training method described in any one of claims 1 to 8; The target emotional voice is fed back to the terminal device.

11. An information processing method based on a task processing model, applied to a task platform, comprising: Receive model requests sent by terminal devices; Based on the model request, a target task processing model is determined from a plurality of task processing models, wherein the plurality of task processing models are trained based on the training method described in any one of claims 1 to 8.

12. The method according to claim 11, wherein the model request includes a scene identifier of the target scene; The step of determining the target task processing model from multiple task processing models based on the model request includes: Based on the scene identifier of the target scene, a target task processing model suitable for the target scene is searched from the model library, wherein the model library stores multiple task processing models suitable for different task processing scenarios.

13. The method according to claim 11, wherein the model request includes scene input data of the target scene; The step of determining the target task processing model from multiple task processing models based on the model request includes: From multiple task processing models, determine the task processing model that is suitable for the target scenario; Based on the scene input data of the target scene, a task processing model adapted to the target scene is trained to obtain the target task processing model.

14. The method according to claim 11, wherein the model request includes model specification parameters; The step of determining the target task processing model from multiple task processing models based on the model request includes: Based on the model specification parameters, the corresponding target task processing model is searched from the model library, wherein the model library stores multiple task processing models with different model specification parameters.

15. The method according to any one of claims 11 to 14, wherein after determining the target task processing model from multiple task processing models based on the model request, the method further comprises: Deploy the target task processing model and, based on the target task processing model, construct a task processing interface so that the terminal device can schedule the target task processing model to execute the target task.

16. A task platform, comprising a request interface and a response unit; The request interface is configured to receive a model request sent by the terminal device, wherein The model request includes at least one of the following: the scene identifier of the target scene, the scene input data of the target scene, and the model specification parameters. The response unit is configured to determine a target task processing model from a plurality of task processing models based on the model request, wherein the plurality of task processing models are trained based on the training method described in any one of claims 1 to 8.

17. The task platform according to claim 16, further comprising a task processing interface, wherein the task processing interface is constructed based on the target task processing model; The task processing interface is used for the terminal device to schedule and execute target tasks.

18. A computing device, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 15.

19. An electronic device comprising: A memory and a processor, the memory and the processor being connected via a bus; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 15.

20. A computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 15.

21. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 15.

Images