Task processing method and device and electronic equipment

Abstract

The invention discloses a task processing method and device and electronic equipment, and belongs to the technical field of artificial intelligence. The task processing method comprises the steps of inputting task data of a first processing task into a task processing model, wherein the task processing model comprises a semantic extraction module and a base module; obtaining first fusion semantic feature information corresponding to the task data and the first task attribute information through a semantic extraction module; processing the first fused semantic feature information through a base module to obtain a task processing result of the first processing task; wherein the first task attribute information is task attribute information corresponding to the task type to which the first processing task belongs.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence technology, specifically relating to a task processing method, apparatus, and electronic device. Background Technology

[0002] With the rapid development of artificial intelligence technology, neural network models, especially multimodal large models, have significantly improved people's production and lives. As multimodal large model technology advances, its application scope continues to expand, enabling it to handle multiple processing tasks.

[0003] In related technologies, when using a multimodal large model to process a type of processing task, a pre-trained neural network model is typically inherited as the base model of the model structure. The model parameters of the base model are first fixed and trained using the training dataset corresponding to the processing task to adjust the model parameters of the encoding and mapping modules in the model structure, excluding the base module, so that the model can initially perform the processing task. Then, the model parameters of the base module are opened up, and the model is trained again using the training dataset corresponding to the processing task to adjust the model parameters of the base module in the model structure, so that the model can perform the corresponding task more accurately.

[0004] Therefore, following the above approach, the multimodal model can only accurately handle one type of processing task. If the multimodal model is directly used to handle other types of tasks, the accuracy of the task processing results will inevitably be poor. Summary of the Invention

[0005] The purpose of this application is to provide a task processing method, apparatus, and electronic device that can improve the accuracy of task processing results.

[0006] In a first aspect, embodiments of this application provide a task processing method, the method comprising:

[0007] The task data of the first processing task is input into the task processing model, which includes a semantic extraction module and a base module.

[0008] The semantic extraction module obtains first fused semantic feature information that corresponds to both the task data and the first task attribute information.

[0009] The base module processes the first fused semantic feature information to obtain the task processing result of the first processing task;

[0010] Among them, the first task attribute information is the task attribute information corresponding to the task type to which the first processing task belongs.

[0011] Fourthly, embodiments of this application provide a task processing apparatus, the apparatus comprising:

[0012] The input module is used to input the task data of the first processing task into the task processing model. The task processing model includes a semantic extraction module and a base module. The semantic extraction module is used to implement the semantic extraction function in the base module, and the model parameters of the speech module are adjusted.

[0013] The acquisition module is used to acquire the first fused semantic feature information corresponding to the task data and the first task attribute information through the semantic extraction module;

[0014] The processing module is used to process the first fused semantic feature information through the base module to obtain the task processing result of the first processing task;

[0015] Among them, the first task attribute information is the task attribute information corresponding to the task type to which the first processing task belongs.

[0016] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, wherein the memory stores a program or instructions executable on the processor, and the program or instructions, when executed by the processor, implement the steps of the task processing method as described in the first aspect.

[0017] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the task processing method as described in the first aspect.

[0018] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the task processing method as described in the first aspect.

[0019] In a sixth aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the task processing method as described in the first aspect.

[0020] In this embodiment of the application, the task data of the first processing task is input into the task processing model, which includes a semantic extraction module and a base module. The semantic extraction module obtains the first fused semantic feature information that corresponds to both the task data and the first task attribute information. The base module processes the first fused semantic feature information to obtain the task processing result of the first processing task. The first task attribute information is the task attribute information corresponding to the task type to which the first processing task belongs. Thus, by adding a semantic extraction module to the task processing model, it is possible to extract fused semantic feature information that corresponds to both the task data of the first processing task and the task attribute information corresponding to the task type to which the first processing task belongs. This allows the base module to process the fused semantic feature information that corresponds to both the task data and the first task attribute information to obtain the task processing result of the first task. Consequently, when processing various types of processing tasks, the task processing model only needs to input the task data of that type of processing task into the task processing model, and the semantic extraction module can extract the fused semantic feature information that corresponds to both the task data of that type of processing task and the task attribute information corresponding to the corresponding task type. Thus, the base module can process the fused semantic feature information that corresponds to both the task data of that type of processing task and the task attribute information corresponding to the corresponding task type to obtain the corresponding task processing result. Therefore, the task processing model can accurately process different types of tasks, thereby effectively improving the accuracy of the task processing results. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the structure of a multimodal audio processing model provided in some embodiments of this application;

[0022] Figure 2 This is a flowchart illustrating a task processing method provided in some embodiments of this application;

[0023] Figure 3 These are schematic diagrams illustrating the structure of a task processing model provided in some embodiments of this application;

[0024] Figure 4 This is a flowchart illustrating a task processing method provided in some embodiments of this application;

[0025] Figure 5 This is a flowchart illustrating a task processing method provided in some embodiments of this application;

[0026] Figure 6A This is a flowchart illustrating a task processing method provided in some embodiments of this application;

[0027] Figure 6B This is a flowchart illustrating a task processing method provided in some embodiments of this application;

[0028] Figure 6C This is a flowchart illustrating a task processing method provided in some embodiments of this application;

[0029] Figure 6D This is a flowchart illustrating a task processing method provided in some embodiments of this application;

[0030] Figure 6E This is a flowchart illustrating a task processing method provided in some embodiments of this application;

[0031] Figure 7A These are schematic diagrams of task processing models provided in some embodiments of this application;

[0032] Figure 7B These are schematic diagrams of task processing models provided in some embodiments of this application;

[0033] Figure 8 These are schematic diagrams illustrating the output results of the task processing model provided in some embodiments of this application;

[0034] Figure 9 This is a flowchart illustrating a task processing model provided in some embodiments of this application;

[0035] Figure 10 These are schematic diagrams of the structure of a task processing device provided in some embodiments of this application;

[0036] Figure 11 These are schematic diagrams of the structure of a task processing device provided in some embodiments of this application;

[0037] Figure 12 These are schematic diagrams of the structure of electronic devices provided in some embodiments of this application;

[0038] Figure 13 These are schematic diagrams of the hardware structure of electronic devices provided in some embodiments of this application. Detailed Implementation

[0039] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0040] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0041] The terms "at least one," "at least one of," etc., used in the specification and claims of this application refer to any one, any two, or a combination of two or more of the included items. For example, at least one of a, b, and c can mean: "a," "b," "c," "a and b," "a and c," "b and c," and "a, b, and c," where a, b, and c can be single or multiple. Similarly, "at least two" refers to two or more items, and its meaning is similar to that of "at least one."

[0042] The task processing model training method provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0043] The task processing method provided in this application can be applied to multimodal dialogue scenarios.

[0044] The current mainstream multimodal audio processing model has the following structure: Figure 1 As shown, the system includes an encoding module 101, a mapping module 102, and a Large Language Model (LLM) model 103, where the text LLM model 103 can serve as the base model for a multimodal audio processing model. The audio signal is encoded into a vector in the feature space by the encoding module 101, then mapped by the mapping module 102 to the same size as the embedding vector of the text LLM model 103 as the base model, and finally fed into the LLM model 103. During training, audio-text data pairs are first constructed, the model parameters of the LLM model are frozen, and the model parameters of either the encoding module 101 or the mapping module 102 are adjusted to align the feature vector space at the audio modality token level with the semantic space of the text token embedding.

[0045] Table 1: Different tasks, same audio, different outputs

[0046]

[0047] However, this approach presents several challenges. First, due to the different information densities of audio and text data (audio data is generally longer), achieving token-level semantic space alignment is extremely difficult. Second, for multiple audio understanding subtasks, a single audio file can correspond to different tasks, as shown in Table 1. For instance, a single audio file can be used for speech recognition, translation, question answering, and other tasks. Specifically, for example... Figure 1 For an audio file (content: "What's the weather like today?"), training is performed on multiple tasks. The output of the audio file after passing through the encoding module 101 and the mapping module 102 represents the features of the audio signal. These features can be used for tasks such as speech recognition, translation, or question answering. That is, the output of the audio features is the same for different tasks. Therefore, the output of the mapping module 102 is related to the input audio and has no obvious correlation with the task attribute information prompt.

[0048] The same audio can correspond to different model outputs, which is achieved by following text prompts. This distinction can only be made within the text-based LLM model, so this approach typically requires fine-tuning the LLM model's parameters. For example, training samples from speech recognition tasks can be used to fine-tune the LLM model's parameters, enabling the multimodal audio processing model to handle speech recognition tasks. However, even after fine-tuning the LLM model's parameters using speech recognition training samples, the multimodal audio processing model will still inevitably exhibit poor accuracy in other speech processing tasks, such as speech translation and question answering.

[0049] To this end, this application provides a task processing method, in which task data of a first processing task is input into a task processing model, the task processing model including a semantic extraction module and a base module; the semantic extraction module obtains first fused semantic feature information corresponding to both the task data and the first task attribute information; the base module processes the first fused semantic feature information to obtain the task processing result of the first processing task; wherein, the first task attribute information is the task attribute information corresponding to the task type to which the first processing task belongs. Thus, by adding a semantic extraction module to the task processing model, it is possible to extract fused semantic feature information that corresponds to both the task data of the first processing task and the task attribute information corresponding to the task type to which the first processing task belongs. This allows the base module to process the fused semantic feature information that corresponds to both the task data and the first task attribute information to obtain the task processing result of the first task. Consequently, when processing various types of processing tasks, the task processing model only needs to input the task data of that type of processing task into the task processing model, and the semantic extraction module can extract the fused semantic feature information that corresponds to both the task data of that type of processing task and the task attribute information corresponding to the corresponding task type. Thus, the base module can process the fused semantic feature information that corresponds to both the task data of that type of processing task and the task attribute information corresponding to the corresponding task type to obtain the corresponding task processing result. Therefore, the task processing model can accurately process different types of tasks, thereby effectively improving the accuracy of the task processing results.

[0050] The execution subject of the task processing method provided in this application embodiment can be a task processing model training device. Exemplarily, the task processing model training device can be an electronic device, or a functional component or entity within that electronic device. The following will executor an electronic device as an example to illustrate the task processing model training method provided in this application embodiment.

[0051] Figure 2 This is a flowchart illustrating the task processing method provided in the embodiments of this application, as shown below. Figure 2 As shown, the task processing method provided in this application embodiment may include the following steps 201 to 203.

[0052] Step 201: The electronic device inputs the task data of the first processing task into the task processing model.

[0053] In some embodiments of this application, the task processing model described above may include a semantic extraction module and a base module.

[0054] In some embodiments of this application, the task processing model described above can process at least one processing task, and the task processing model described above can be a multimodal model.

[0055] In some embodiments of this application, the above-mentioned at least one processing task may include image processing task, voice processing task, and other tasks. The specific task may be determined according to actual needs, and no specific limitation is made here.

[0056] In some embodiments of this application, the above-mentioned image processing tasks may include image recognition tasks, text-to-image tasks, and other types of image processing tasks. The specific tasks can be determined according to actual needs, and no specific limitations are made here.

[0057] In some embodiments of this application, the above-mentioned speech processing tasks may include speech recognition tasks, speech translation tasks, speech question-and-answer tasks, and may also include other types of speech processing tasks. The specific tasks can be determined according to actual needs, and this embodiment does not impose specific limitations here.

[0058] In some embodiments of this application, the first processing task described above can be any one of the at least one processing task described above.

[0059] In some embodiments of this application, such as Figure 3 As shown, the task processing model 300 includes a semantic extraction module 301 and a base module 302.

[0060] In some embodiments of this application, the base module 302 described above may adopt a textual LLM model.

[0061] Step 202: The electronic device obtains the first fused semantic feature information corresponding to both the task data and the first task attribute information through the semantic extraction module.

[0062] In some embodiments of this application, the first fusion semantic feature information described above can be used to characterize the fusion semantics of the task data and the first task attribute information described above.

[0063] It should be noted that the explanation of step 202 above can be found in the relevant description in step 205 below. To avoid repetition, this embodiment will not repeat the description here.

[0064] Step 203: The electronic device processes the first fused semantic feature information through the base module to obtain the task processing result of the first processing task.

[0065] In some embodiments of this application, the aforementioned first task attribute information may be task attribute information corresponding to the task type to which the aforementioned first processing task belongs. In other words, the aforementioned first task attribute information is used to characterize the task type of the aforementioned first processing task.

[0066] In some embodiments of this application, the aforementioned first task attribute information may be the text prompt word corresponding to the task type to which the aforementioned first processing task belongs.

[0067] For example, if the task type of the first processing task is a speech recognition task and the task data is the audio of "How is the weather today?", the first task attribute information, i.e. the text prompt corresponding to the speech recognition task, can be "recognize the input audio into text". The text prompt "recognize the input audio into text" can indicate that the task type of the first processing task is a speech recognition task.

[0068] In the task processing method provided in this application embodiment, the task data of the first processing task is input into the task processing model, which includes a semantic extraction module and a base module; the semantic extraction module obtains first fused semantic feature information that corresponds to both the task data and the first task attribute information; the base module processes the first fused semantic feature information to obtain the task processing result of the first processing task; wherein, the first task attribute information is the task attribute information corresponding to the task type to which the first processing task belongs. Thus, by adding a semantic extraction module to the task processing model, it is possible to extract fused semantic feature information that corresponds to both the task data of the first processing task and the task attribute information corresponding to the task type to which the first processing task belongs. This allows the base module to process the fused semantic feature information that corresponds to both the task data and the first task attribute information to obtain the task processing result of the first task. Consequently, when processing various types of processing tasks, the task processing model only needs to input the task data of that type of processing task into the task processing model, and the semantic extraction module can extract the fused semantic feature information that corresponds to both the task data of that type of processing task and the task attribute information corresponding to the corresponding task type. Thus, the base module can process the fused semantic feature information that corresponds to both the task data of that type of processing task and the task attribute information corresponding to the corresponding task type to obtain the corresponding task processing result. Therefore, the task processing model can accurately process different types of tasks, thereby effectively improving the accuracy of the task processing results.

[0069] In some embodiments of this application, combined with Figure 2 ,like Figure 4 As shown, prior to step 201 above, the task processing method provided in this application embodiment may further include the following steps 204 to 207:

[0070] Step 204: The electronic device inputs the first training data into the task processing model.

[0071] In some embodiments of this application, the first training data may include training data corresponding to at least one of the processing tasks.

[0072] Step 205: The electronic device obtains second fused semantic feature information that corresponds to both the first training sample and the attribute information of the second task through the semantic extraction module.

[0073] In some embodiments of this application, the first training data may include the first training samples.

[0074] In some embodiments of this application, the aforementioned second task attribute information may be task attribute information corresponding to the task type to which the first training sample belongs. In other words, the aforementioned second task attribute information is used to characterize the task type to which the first training sample belongs.

[0075] In some embodiments of this application, the aforementioned second task attribute information may be the text prompt word corresponding to the task type to which the aforementioned first training sample belongs.

[0076] For example, if the task type of the first training sample is a speech recognition task, that is, if the first training sample is a training sample for a speech recognition task, the second task attribute information, that is, the text prompt corresponding to the speech recognition task, can be "recognize the input audio into text". The text prompt "recognize the input audio into text" can indicate that the task type of the first training sample is a speech recognition task.

[0077] In some embodiments of this application, combined with Figure 4 ,like Figure 5 As shown, step 205 above can be achieved through the following steps 205a and 205b:

[0078] Step 205a: The electronic device obtains the first concatenated feature information corresponding to both the first training sample and the attribute information of the second task through the semantic extraction module.

[0079] In some embodiments of this application, the first splicing feature information is used to characterize the fusion information that corresponds to both the first training sample and the second task attribute information.

[0080] Step 205b: The electronic device extracts semantic features from the first spliced ​​feature information through the semantic extraction module to obtain the second fused semantic feature information.

[0081] In some embodiments of this application, the second fusion semantic feature information is used to characterize the fusion semantics of the first training sample and the second task attribute information.

[0082] In some embodiments of this application, the electronic device can extract semantic features from the first spliced ​​feature information through the convolutional layer in the semantic extraction module 301 to obtain the second fused semantic feature information.

[0083] Thus, the electronic device obtains first concatenated feature information corresponding to both the first training sample and the attribute information of the second task through the semantic extraction module; and extracts semantic features from the first concatenated feature information through the semantic extraction module to obtain fused semantic feature information, which can accurately extract the fused semantics of the first training sample and the attribute information of the second task.

[0084] In some embodiments of this application, such as Figure 3 As shown, the task processing model 300 further includes a mapping module 303. Before step 202a, the task processing model training method provided in this application embodiment may further include the following step 208. Step 205a can be implemented through the following step 205a1:

[0085] Step 208: The electronic device obtains the mapping sample feature information corresponding to the first training sample through the mapping module.

[0086] In some embodiments of this application, the mapping module is used to map the first training sample to the same dimension as the second task attribute information, that is, the mapped sample feature information has the same dimension as the second task attribute information.

[0087] Step 205a1: The electronic device uses the semantic extraction module to concatenate the mapping sample feature information and the second task attribute information corresponding to the first training sample to obtain the first concatenated feature information.

[0088] In some embodiments of this application, the electronic device can extract features from the second task attribute information using the semantic extraction module to obtain first task attribute feature information, and then concatenate the mapping sample feature information and the first task attribute feature information to obtain first concatenated feature information.

[0089] In some embodiments of this application, the dimension of the first splicing feature information is the sum of the dimension of the mapping sample feature information and the dimension of the first task attribute feature information.

[0090] Thus, the electronic device obtains the mapping sample feature information corresponding to the first training sample through the mapping module, and can map the first training sample to the mapping sample feature information of the same dimension as the second task attribute information. This enables the semantic extraction module to concatenate the mapping sample feature information corresponding to the first training sample and the second task attribute information to obtain the first concatenated feature information.

[0091] In some embodiments of this application, the first training sample mentioned above can be an audio sample, such as... Figure 3As shown, the mapping module 303 may include an encoding unit 3031 and a mapping unit 3032. Step 208 can be implemented through the following steps 208a and 208b:

[0092] Step 208a: The electronic device extracts audio features from the first training sample through the encoding unit to obtain the audio feature information of the audio sample.

[0093] In some embodiments of this application, the encoding unit 3031 is used to encode the first training sample of the audio signal into features of the audio space, that is, to extract audio features from the first training sample to obtain the audio feature information of the audio sample.

[0094] Step 208b: The electronic device maps the audio feature information through the mapping unit to obtain the mapped sample feature information.

[0095] In some embodiments of this application, the mapping unit 3032 is used to map audio feature information into mapped sample feature information with the same dimension as the text feature information. In other words, the dimension of the mapped sample feature information is the same as the dimension of the text feature information.

[0096] Thus, the electronic device extracts audio features from the first training sample through the encoding unit to obtain audio feature information of the audio sample; through the mapping unit, it maps the audio feature information to obtain mapped sample feature information, which can map the audio feature information into mapped sample feature information with the same dimension as the text feature information.

[0097] Step 206: The electronic device processes the first fused semantic feature information through the base module and outputs the first model inference result corresponding to the first training sample.

[0098] In some embodiments of this application, the electronic device can identify the semantics represented by the first fused semantic feature information through the base module 202 and output the first model inference result.

[0099] Step 207: The electronic device adjusts the model parameters of the semantic extraction module based on the reasoning results of the first model and the first training data.

[0100] Thus, the electronic device inputs the first training data into a task processing model, which includes a semantic extraction module and a base module. The semantic extraction module acquires second fused semantic feature information corresponding to both the first training samples and the attribute information of the second task. The first training data includes the first training samples. The base module processes the second fused semantic feature information and outputs a first model inference result corresponding to the first training samples. The first model inference result and the first training data are used to adjust the model parameters of the semantic extraction module. Since the semantic extraction module can be used to implement semantic extraction, the electronic device only needs to adjust the model parameters of the semantic extraction module to complete the training of the task processing model, without needing to adjust the model parameters of the base module in the task processing model. This effectively avoids the problem of knowledge loss caused by adjusting the model parameters of the base module, thereby effectively improving the accuracy of the model inference results.

[0101] In some embodiments of this application, step 207 described above can be implemented by the following steps 207a to 207d:

[0102] Step 207a: Freeze the model parameters of the electronic device base module and the model parameters of the semantic extraction module, and adjust the model parameters of the mapping module based on the inference results of the first model and the sample labels in the first training data.

[0103] In some embodiments of this application, the electronic device can adjust the model parameters of the mapping module based on the difference between the inference result of the first model and the sample labels in the first training data.

[0104] Step 207b: After the electronic device adjusts the model parameters of the mapping module, it obtains the third fused semantic feature information corresponding to both the first training sample and the attribute information of the second task through the semantic extraction module.

[0105] It should be noted that the explanation of step 207b above can be found in the relevant description in step 205 above. To avoid repetition, this embodiment will not repeat the description here.

[0106] Step 207c: The electronic device processes the semantic feature information of the third unit through the base module to obtain the second model inference result corresponding to the first training sample.

[0107] It should be noted that the explanation of step 207c above can be found in the relevant description in step 206 above. To avoid repetition, this embodiment will not repeat it here.

[0108] Step 207d: Freeze the model parameters of the electronic device base module and the model parameters of the mapping module, and adjust the model parameters of the semantic extraction module based on the second model inference results and the first training data.

[0109] In some embodiments of this application, the electronic device can adjust the model parameters of the mapping module based on the difference between the second model inference result and the sample labels in the first training data.

[0110] In this way, by freezing the model parameters of the base module and the semantic extraction module, and based on the first model inference result and the sample labels in the first training data, the electronic device can adjust the model parameters of the mapping module. After adjusting the model parameters of the mapping module, the semantic extraction module can obtain the third fused semantic feature information corresponding to both the first training sample and the attribute information of the second task. By processing the third fused semantic feature information through the base module, the second model inference result corresponding to the first training sample can be obtained. By freezing the model parameters of the base module and the mapping module, and adjusting the model parameters of the semantic extraction module based on the second model inference result and the first training data, the training of the task processing model can be completed without adjusting the model parameters of the base module in the task processing model. This effectively avoids the problem of knowledge forgetting caused by adjusting the model parameters of the base module, and thus effectively improves the accuracy of the model inference results.

[0111] In some embodiments of this application, the first training data may include the training samples and sample labels, and the training samples may include at least one audio sample. The task processing model training method provided in this application may further include the following steps 209 to 211:

[0112] Step 209: The electronic device acquires the first text feature information corresponding to the first audio sample.

[0113] In some embodiments of this application, the first audio sample may be any one of the at least one audio sample.

[0114] In some embodiments of this application, the electronic device can obtain the first text feature information corresponding to the first audio sample through the base module 301.

[0115] Step 210: The electronic device processes the first text feature information and the second task attribute information through the base module to obtain at least one reasoning result corresponding to the first audio sample.

[0116] In some embodiments of this application, the electronic device can extract features from the second task attribute information using a base module to obtain first task attribute feature information. Then, the base module concatenates the first text feature information and the second task attribute feature information to obtain second concatenated feature information. Next, the base module extracts semantic features from the second concatenated feature information to obtain fused semantic feature information corresponding to the first text feature information and the second task attribute information. Finally, the base module identifies the semantics represented by the fused semantic feature information corresponding to the first text feature information and the second task attribute information, and outputs at least one inference result corresponding to the first audio sample, along with the probability value of each inference result.

[0117] In some embodiments of this application, the dimension of the second splicing feature information is the sum of the dimension of the first text feature information and the dimension of the first task attribute feature information.

[0118] Step 211: The electronic device determines the N inference results with the highest probability of at least one inference result as the sample label corresponding to the first audio sample.

[0119] In some embodiments of this application, N is a positive integer.

[0120] In some embodiments of this application, the electronic device can sort the above-mentioned at least one inference result in descending order of probability value, determine the N inference results with the highest probability, and determine the N inference results as the sample labels corresponding to the first audio sample.

[0121] In this way, the electronic device obtains the first text feature information corresponding to the first audio sample; through the base module, it processes the first text feature information and the second task attribute information to obtain at least one inference result corresponding to the first audio sample; and determines the N inference results with the highest probability of at least one inference result as the sample labels corresponding to the first audio sample. This allows the inference result with the highest accuracy to be used as the sample label. Furthermore, when adjusting the model parameters of the semantic task processing model extraction module based on the sample label, the accuracy of the model parameters of the semantic extraction module can be improved, thereby improving the accuracy of the inference results of the task processing model.

[0122] In some embodiments of this application, prior to step 207 above, the task processing model training method provided in this application may further include steps 212 to 214, and step 207 above can be implemented through step 207b below:

[0123] Step 212: The electronic device obtains the fourth fused semantic feature information, which corresponds to both the second text feature information and the task attribute information, through the fusion submodule.

[0124] In some embodiments of this application, the second text feature information is the text feature information corresponding to the second audio sample, the second audio sample is any audio sample in a subset of audio samples, and the subset of audio samples includes M audio samples from the at least one audio sample, where M is a positive integer.

[0125] In some embodiments of this application, the electronic device can map the second audio sample into the second text feature information through a mapping module.

[0126] It should be noted that the explanation of step 212 above can be found in the relevant description in step 205 above. To avoid repetition, this embodiment will not repeat the description here.

[0127] Step 213: The electronic device processes the fourth fused semantic feature information through the base module to obtain the third model inference result corresponding to the second audio sample.

[0128] It should be noted that the explanation of step 213 above can be found in the relevant description in step 206 above. To avoid repetition, this embodiment will not repeat the description here.

[0129] Step 214: The electronic device determines the N model inference results with the highest probability from the first model inference results and the third model inference results corresponding to the second audio sample.

[0130] In some embodiments of this application, the first model inference result may include multiple model inference results corresponding to the first training sample, and the third model inference result may include multiple model inference results corresponding to the second audio sample.

[0131] In some embodiments of this application, the electronic device can output multiple model inference results corresponding to the first training sample and the probability value of each model inference result through the base module.

[0132] In some embodiments of this application, the electronic device can output multiple model inference results corresponding to the second audio sample and the probability value of each model inference result through the base module.

[0133] In some embodiments of this application, the electronic device sorts the first model inference results and the third model inference results in descending order of probability value, and determines the N model inference results with the highest probability.

[0134] Step 207b: The electronic device adjusts the model parameters of the semantic extraction module based on N model inference results, N probability values ​​corresponding to the N model inference results, N inference results corresponding to the second audio sample, and the probability values ​​corresponding to the N inference results.

[0135] In some embodiments of this application, the N inference results corresponding to the second audio sample are the sample labels corresponding to the second audio sample.

[0136] In some embodiments of this application, the electronic device can adjust the model parameters of the semantic extraction module based on the above N model inference results, the N probability values ​​corresponding to the N model inference results, and the difference between the sample label corresponding to the second audio sample and the probability value corresponding to the sample label.

[0137] Thus, the electronic device obtains fourth fused semantic feature information corresponding to both the second text feature information and the task attribute information through the fusion submodule; processes the third fused semantic feature information through the base module to obtain the third model inference result corresponding to the second audio sample; determines the N model inference results with the highest probability from the first and third model inference results corresponding to the second audio sample; and adjusts the model parameters of the semantic extraction module based on the N model inference results, the N probability values ​​corresponding to the N model inference results, the N inference results corresponding to the second audio sample, and the probability values ​​corresponding to the N inference results. This allows the model parameters of the semantic task processing model extraction module to be adjusted based on the inference results with the highest accuracy and their probability values, thereby improving the accuracy of the model parameters of the semantic extraction module and thus improving the accuracy of the inference results of the task processing model.

[0138] The task processing method provided in this application embodiment will be described below with reference to specific implementation methods. Figure 6A This is a flowchart illustrating the task processing method provided in the embodiments of this application. Taking audio processing by a task processing model as an example, the method may include the following steps 61 to 64.

[0139] Step 61: Train the speech recognition task.

[0140] For example, such as Figure 6B As shown, the training process for a speech recognition task may include the following steps 611 to 617:

[0141] Step 611: Select a text-based base model (LLM). This can be selected from open-source models, such as Alibaba's Qwen series models. Alternatively, you can train your own base model. This invention does not depend on the selection of the text-based base model.

[0142] For example, the text base model LLM is the base module mentioned above.

[0143] Step 612: Select an encoder to encode the audio WAV signal into audio spatial features. SenseVoice or Whisper series encoders can be selected. This invention does not depend on the selection of the encoder.

[0144] For example, the Encoder audio encoder is the encoding unit described above.

[0145] Step 613: Determine the mapping layer, Proj layer. Its purpose is to map the output feature dimension of the WAV signal after passing through the Encoder to the same size as the embedding space of the base model. It typically consists of several fully connected layers. Sometimes, to reduce inference overhead, the features may be downsampled. For any task, the input to the Proj layer is the audio features extracted by the Encoder, and the output is the audio features mapped to the same size as the LLM.

[0146] For example, the Proj layer is the mapping unit described above.

[0147] Step 614, Construction Figure 1 Model Structure. This model structure is the same as the LLM model, except that the input of the text LLM model is replaced by the input of the Proj layer. For example, the input of this Proj has a similar feature space to the text token, and can represent text-like features.

[0148] Step 615: Construct audio-text (recognized text) speech recognition data pairs. This can be done by collecting open-source datasets or by recording your own data and annotating it manually or using tools. Similarly, if training for other tasks, such as translation, construct audio-text (translated text) translation data pairs.

[0149] Step 616: Select the prompt for the speech recognition task, such as: <Please recognize the following audio as text:>. Other instructions can also be selected. This instruction simply guides the different responses to the audio under different tasks.

[0150] For example, the prompt for a speech recognition task is the task attribute information of the speech recognition task.

[0151] Step 617: Train the model. Freeze the text LLM model parameters, train it using speech recognition data, and adjust the parameters of the Proj module or encoder module.

[0152] It should be noted that the specific implementation process of step 617 above can refer to step 207a above. To avoid repetition, this embodiment will not repeat it here.

[0153] Step 62: Insert the Fusion Transformer module to train the speech recognition task.

[0154] For example, the Fusion Transformer module mentioned above is the semantic extraction module mentioned above.

[0155] For example, such as Figure 7A As shown, based on the module trained in step 61, a Fusion Transformer module is added. Using speech recognition data, the parameters of the Fusion Transformer module are adjusted, all other parameters are frozen, and the speech recognition task is trained again.

[0156] For example, such as Figure 6C As shown, step 62 can be achieved through steps 621 to 626 as follows:

[0157] Step 621, in Figure 1 Building upon this foundation, a Fusion Transformer module was added between the Proj layer and the LLM model. As shown in Table 2, the input to the Fusion Transformer module is the output of the Proj layer, which consists of task-independent audio features. Since the input to the Fusion Transformer module also includes prompts from different tasks, the task prompts and audio features are fused at this layer, then adapted and fed into the LLM model. In other words, the output of the Fusion Transformer module is audio features bound to the task. Clearly, compared to Table 1, inserting the Fusion Transformer module makes training instruction following for different tasks much easier.

[0158] Table 2: Different tasks, same audio, different outputs

[0159]

[0160] Step 622: Connect the embeddings corresponding to the audio features output from the Proj layer to the Fusion Transformer module, and also connect the embeddings corresponding to the text prompt instructions to the Fusion Transformer module. For example, different audio subtasks are further processed in the Fusion Transformer module to complete instruction following and the mapping from audio modalities to the text LLM model embedding space. Unlike... Figure 1This design achieves token-level alignment, enabling alignment at the input WAV level. Because the Fusion Transformer module uses a multi-layered transformer, it has the ability to learn contextual information from both audio and text. This design removes the unreasonable constraint of token-level alignment, making it easier to align audio modalities to text modalities and avoiding the problem of audio-text length mismatch. Furthermore, in the Fusion Transformer module, task instruction following is achieved through the injection of different text prompts corresponding to different tasks. The entire process described above does not involve parameter tuning of the LLM model; all alignment is completed before the LLM model. This truly achieves zero-parameter tuning of the text LLM base model, aligning multi-audio understanding tasks and avoiding the industry-wide problem of catastrophic knowledge loss in text LLM models.

[0161] Step 623: Connect the text response to the Fusion Transformer module. To support multi-turn dialogue during the conversation, this implementation uses the following connection method: the text response is also sent to the Fusion Transformer module. This allows the Fusion Transformer module to receive input and output information from the multi-turn dialogue, thus supporting multi-turn dialogue with voice input.

[0162] Step 624: The text in the question and the text in the answer are directly connected to the LLM model. To support the injection of text information at any position, the text is not sent to the LLM model through the Fusion Transformer module, but directly to the LLM model, thus not changing the original text input and output method. This supports the injection of text information at the beginning and middle of the dialogue, and supports multi-turn dialogue logic with alternating voice and text input. The addition of the audio modality does not change the input and output form of the original text-based model. The model block diagram for multi-turn dialogue is as follows: Figure 7B As shown.

[0163] Step 625: Initialize the Fusion Transformer module randomly or in other ways, and inherit the parameters obtained from training in step 61.

[0164] Step 626: Continue training the model using speech recognition data, adjusting only the parameters of the Fusion Transformer module while freezing the parameters of other modules, until the model's loss on the validation set no longer decreases significantly. For example, the cross-entropy loss no longer decreases significantly within the range of 0.01.

[0165] It should be noted that the specific implementation process of step 626 above can refer to step 207d above. To avoid repetition, this embodiment will not repeat it here.

[0166] Step 63: Construct self-aligned question-and-answer data.

[0167] For example, this embodiment proposes a self-aligned data generation and training method. During the alignment process, it was found that the construction of the dialogue training data has a significant impact on the alignment effect. In dialogue or knowledge-based question-answering scenarios, different models can produce different responses to the same query. More specifically, if the text of model A's response is used to train a multimodal large model with model B as the base model, the knowledge capability of the trained multimodal large model's responses is relatively low, provided that the LLM parameters of model B are frozen. Based on this, this embodiment provides the following data construction and training steps to construct self-aligned question-answering data.

[0168] For example, the self-aligned question-and-answer data mentioned above is the first training sample mentioned above.

[0169] For example, such as Figure 6D As shown, step 63 above can be accomplished through steps 631 to 636 as follows:

[0170] Step 631: Using a large language model, construct text queries for different topics, which can be single-turn or multi-turn queries. For example, topics include, but are not limited to, history, law, entertainment, economics, mathematics, etc.

[0171] Step 632: Select different text prompts, such as restrictions on response style, word count, and conversational guidance. It's recommended to consider diversity in the prompts for better model alignment. For example, Table 3 shows that for the same query, different prompts result in different response styles from the text model. In dialogue scenarios, the conversational nature and proactiveness of voice interaction need to be considered.

[0172] Table 3. Different prompts and corresponding response examples

[0173]

[0174] Step 633: Generate text responses using the base LLM model. Assuming the text base model in this implementation is model B, the text prompt from step 632 and the text query from step 631 can be fed into model B. The output answer of model B is used as the text prompt-text query-text answer data pair for basic training data. The generated answer should not be do-sampled; the answer should be the token corresponding to the highest probability output by model B. In summary, the answer data generated using the text base model LLM is used for training the multimodal model.

[0175] Step 634: All the text queries mentioned above are synthesized into corresponding audio files using the TTS model, forming text prompt-audio query-text answer data pairs.

[0176] Step 635: In multi-turn dialogues, certain queries can be randomly selected for audio replacement to construct training data that supports the ability to input text and audio alternately.

[0177] Step 636: The general approach is to align the data pairs constructed in step 633. This embodiment proposes a novel probability density alignment method. This embodiment aligns not only the answer text with the highest probability, but also the probability density of the tokens predicted by the base model LLM. For example... Figure 8 As shown in (a) of the example, using the text answer "The Three Kingdoms consist of the three kingdoms of Wei, Shu, and Wu," data constructed using a typical one-hot encoding method is presented. This method aligns tokens using a one-hot encoding, where the aligned model predicts a token probability of 1 and other token probabilities of 0. However, this probability is not the true probability density of the text model; this method only aligns the token ID corresponding to the highest probability in the text LLM model. Since aligning the probability density corresponding to the next token is found to be more meaningful, it is desirable for the model to align the highest probability with other probabilities. However, in practice, storing the probability density data for each predicted token would be too large and unnecessary. For example, the top 10 predicted probabilities and their corresponding tokens are retained as training data. In multimodal alignment dialogue tasks, this implementation aligns the top 10 predicted probabilities. Of course, the top n probabilities can also be selected; this implementation does not limit this. Figure 8As shown in (b) above, taking "The Three Kingdoms consisted of the three kingdoms of Wei, Shu, and Wu" as an example, the probability data for the top two tokens (only two tokens are shown for simplicity in the illustration) is given. Each time the next token is calculated, the probability is calculated... Figure 8 In (b), the cross-entropy of the two tokens in the horizontal direction, the first two are likely the probabilities of the real LLM model, not one-hot data.

[0178] Step 64: Train a hybrid task involving dialogue and speech recognition.

[0179] For example, such as Figure 6E As shown, step 64 above can be achieved through steps 641 to 644 as follows:

[0180] Step 641: Prepare the data pairs for the dialogue task and the speech recognition task.

[0181] For example, the above data pair is the first training sample and the sample label corresponding to the first training sample.

[0182] Step 642: Mix dialogue data and speech recognition data. Since the speech recognition task has already been aligned in steps 61 and 62, a dialogue task is added in step 64. The dialogue task requires only a very small amount of data to complete the alignment of the multimodal dialogue task.

[0183] Step 643: Adjust the parameters of the Fusion Transformer module and freeze all other parameters.

[0184] Step 644: For dialogue tasks, the loss function needs to calculate the predicted probability and the cross-entropy loss of the top 10 most probable values ​​in the training data.

[0185] Step 645: Train the model until the loss no longer decreases significantly.

[0186] Below, in conjunction with Figure 9 The given examples illustrate the improved task processing model training method implemented in this application. For example... Figure 9 As shown, the method includes the following steps 901 to 904:

[0187] Step 901: Initialize model structure parameters.

[0188] This example uses the Alibaba Qwen2.5 intstruct 7B model as the text-based LLM model and the Whisper middle300M model as the encoder. By default, the Whisper encoder generates 50 tokens per second. To reduce training and inference costs, for the Proj layer, the encoder output is first passed through a one-dimensional average pooling layer with stride=2 and kernel size=2 to reduce the input signal sampling rate, generating 25 tokens per second. Since the encoder output feature size differs from the text LLM embedding size, the pooling layer output needs to be mapped to the same size as the embedding. Therefore, two fully connected layers can be used to ensure consistent output sizes. For the FusionTransformer module, this implementation uses three transformer layers. These three transformer layers have the exact same size as the transformer layers in the Qwen2.5 intstruct 7B model, and the attention mask is also identical. The FusionTransformer module is initialized using the parameters of the first three layers of Qwen2.5.

[0189] Step 902: Obtain model training data.

[0190] We collected 200,000 hours of open-source speech recognition data from the internet and some of our own speech recognition data for speech recognition training. For dialogue or question-and-answer chat data, we used a large third-party model to generate multi-turn dialogues for multiple sub-scenarios. We selected 50,000 multi-turn dialogues, each with no more than 10 turns. The queries from these 50,000 multi-turn dialogues were fed into the responses of Qwen2.5's intstruct 7B as training data; see the data construction method for details. The text queries from these 50,000 multi-turn dialogues were used to create audio queries using a TTS model for training the voice modal input. In total, approximately 2,000 hours of voice data were used for audio queries.

[0191] Step 903: Train the model.

[0192] Firstly, according to Figure 1 The model structure uses speech recognition data to train the parameters of the Proj and encoder modules, with a constant learning rate of lr=5e-4, 8 L40s machines with 4 cards each, a batch size of 256, 2000 warm-up steps, and training for one epoch.

[0193] Based on the above training, add a Fusion Transformer module and continue training the Fusion Transformer module using speech recognition data, with the same settings as above.

[0194] Then, add question-and-answer or casual conversation data to train a hybrid task combining speech recognition and question-and-answer casual conversation. Freeze all parameters except the fusion parameter. Since the model already has multimodal speech recognition capabilities, we only need to align the question-and-answer task while maintaining its speech recognition capabilities. Because only a small amount of question-and-answer casual conversation data is needed to complete the alignment, a larger ratio of speech recognition to question-and-answer casual conversation data yields better results, such as a 100:1 data ratio. Parameter settings are the same as above, thus requiring only a short training time of 1000 steps for the model to achieve alignment capabilities.

[0195] Step 904: Output the model inference results.

[0196] For example, the model trained using the above method was evaluated on speech recognition and question-and-answer chat tasks, and the results were compared with those of the open-source minicpm-o 2.6. Table 4 shows the CER performance of speech recognition on some open-source test sets. It can be seen that the speech recognition metrics of this solution are better than those of the open-source minicpm-o 2.6.

[0197] Table 5 shows the performance of the question-answering task on the self-built question-answering test set QA500. The test set consists of 500 self-built audio queries covering knowledge questions and answers in various fields. The scoring is out of 5. The reference answers and the model responses used for testing are fed into a large model for scoring. It can be seen that the frozen text-based model preserves knowledge much better than the scheme that adjusts the text-based model (minicpm adjusts the text-based model parameters). Furthermore, compared to the original text-based QA500 model, the scoring results of this implementation are very close to those of the text model, demonstrating the effectiveness of this scheme for aligning multimodal audio tasks.

[0198] Table 4. Comparison of Speech Recognition Experiment Results

[0199]

[0200] Table 5. Comparison of the results of the question-and-answer casual conversation experiment

[0201]

[0202] This implementation provides a low-cost solution for multimodal understanding tasks. By completely freezing the parameters of the text-based model and designing the model structure, self-aligned data generation method, and training method, it effectively solves the problem of knowledge loss in multimodal scenarios. It also supports the problem of text knowledge injection in multi-turn dialogue scenarios, preserving the original text input and output format, and can bring an improvement in user experience to dialogue systems in multimodal dialogue scenarios.

[0203] It should be noted that each of the above method embodiments, or various possible implementations of each method embodiment, can be executed individually or in combination of any two or more. The specific implementation can be determined according to actual usage requirements, and this application embodiment does not impose any restrictions on this.

[0204] The task processing model training method provided in this application can be executed by a task processing model training device. This application uses the example of a task processing model training device executing the task processing model training method to illustrate the task processing model training device provided in this application.

[0205] Figure 10 This is a schematic diagram of the structure of a task processing device 1000 provided in an embodiment of this application. The task processing model training device 1000 includes: an input unit 1001, an acquisition unit 1002, and a processing unit 1003.

[0206] The input unit 1001 is used to input the task data of the first processing task into the task processing model, which includes a semantic extraction module and a base module.

[0207] The acquisition unit 1002 is used to acquire, through the semantic extraction module, first fused semantic feature information that corresponds to both the task data and the first task attribute information;

[0208] Processing unit 1003 is used to process the first fused semantic feature information through the base module to obtain the task processing result of the first processing task;

[0209] The first task attribute information is the task attribute information corresponding to the task type to which the first processing task belongs.

[0210] In some embodiments of this application, the input unit 1001 is further configured to:

[0211] Before inputting the task data of the first processing task into the task processing model, the first training data is input into the task processing model.

[0212] The acquisition unit 1002 is further configured to acquire, through the semantic extraction module, second fused semantic feature information that corresponds to both the first training sample and the second task attribute information, wherein the first training data includes the first training sample and the second task attribute information is the task attribute information corresponding to the task type to which the task to which the first training sample belongs.

[0213] Combination Figure 10 ,like Figure 11 As shown, the device 1000 further includes: an output unit 1004, used to process the second fused semantic feature information through the base module and output the first model inference result corresponding to the first training sample;

[0214] The adjustment unit 1005 is used to adjust the model parameters of the semantic extraction module based on the inference results of the first model and the first training data.

[0215] In some embodiments of this application, the acquisition unit 1002 is specifically used for:

[0216] The semantic extraction module obtains first concatenated feature information that corresponds to both the first training sample and the attribute information of the second task.

[0217] The semantic extraction module extracts semantic features from the first spliced ​​feature information to obtain the second fused semantic feature information.

[0218] In some embodiments of this application, the task processing model further includes a mapping module, and the acquisition module is further configured to:

[0219] Before obtaining the first concatenated feature information corresponding to both the first training sample and the second task attribute information through the semantic extraction module, the mapping module obtains the mapping sample feature information corresponding to the first training sample.

[0220] The acquisition unit 1002 is specifically used for:

[0221] The semantic extraction module concatenates the mapping sample feature information and the second task attribute information to obtain the first concatenated feature information.

[0222] In some embodiments of this application, the acquisition unit 1002 is specifically used for:

[0223] Freeze the model parameters of the base module and the model parameters of the semantic extraction module, and adjust the model parameters of the mapping module based on the first model inference result and the sample labels in the first training data.

[0224] After adjusting the model parameters of the mapping module, the semantic extraction module obtains third fused semantic feature information that corresponds to both the first training sample and the second task attribute information.

[0225] The base module processes the third fused semantic feature information to obtain the second model inference result corresponding to the first training sample;

[0226] The model parameters of the base module and the model parameters of the mapping module are frozen, and the model parameters of the semantic extraction module are adjusted based on the second model inference result and the first training data.

[0227] In some embodiments of this application, the first training sample is an audio sample, the mapping module includes an encoding unit and a mapping unit, and the acquisition unit 1002 is specifically used for:

[0228] The encoding unit extracts audio features from the first training sample to obtain the audio feature information of the audio sample.

[0229] The mapping unit maps the audio feature information to obtain the mapped sample feature information.

[0230] In some embodiments of this application, the first training data includes the training samples and sample labels, the training samples include at least one audio sample, and the acquisition unit 1002 is further configured to:

[0231] Obtain the first text feature information corresponding to the first audio sample;

[0232] The base module processes the first text feature information and the second task attribute information to obtain at least one inference result corresponding to the first audio sample.

[0233] The N inference results with the highest probability of the at least one inference result are determined as the sample labels corresponding to the first audio sample;

[0234] Wherein, the first audio sample is any one of the at least one audio samples, and N is a positive integer.

[0235] In some embodiments of this application, the adjustment unit 1005 is further configured to:

[0236] Based on the inference results of at least one first model and the first training dataset, before adjusting the model parameters of the semantic extraction module, the fusion submodule obtains fourth fused semantic feature information that corresponds to both the second text feature information and the second task attribute information.

[0237] The base module processes the fourth fused semantic feature information to obtain the third model inference result corresponding to the second audio sample;

[0238] From the first model inference result and the third model inference result corresponding to the second audio sample, determine the N model inference results with the highest probability;

[0239] The adjustment unit 1005 is specifically used for:

[0240] Based on the N model inference results, the N probability values ​​corresponding to the N model inference results, the N inference results corresponding to the second audio sample, and the probability values ​​corresponding to the N inference results, the model parameters of the semantic extraction module are adjusted.

[0241] Wherein, the second text feature information is the text feature information corresponding to the second audio sample, the second audio sample is any audio sample in the audio sample subset, and the audio sample subset includes M audio samples from the at least one audio sample, where M is a positive integer.

[0242] The task processing apparatus provided in this application embodiment inputs the task data of a first processing task into a task processing model. The task processing model includes a semantic extraction module and a base module. The semantic extraction module obtains first fused semantic feature information that corresponds to both the task data and the first task attribute information. The base module processes the first fused semantic feature information to obtain the task processing result of the first processing task. The first task attribute information is the task attribute information corresponding to the task type to which the first processing task belongs. Thus, by adding a semantic extraction module to the task processing model, it is possible to extract fused semantic feature information that corresponds to both the task data of the first processing task and the task attribute information corresponding to the task type to which the first processing task belongs. This allows the base module to process the fused semantic feature information that corresponds to both the task data and the first task attribute information to obtain the task processing result of the first task. Consequently, when processing various types of processing tasks, the task processing model only needs to input the task data of that type of processing task into the task processing model, and the semantic extraction module can extract the fused semantic feature information that corresponds to both the task data of that type of processing task and the task attribute information corresponding to the corresponding task type. Thus, the base module can process the fused semantic feature information that corresponds to both the task data of that type of processing task and the task attribute information corresponding to the corresponding task type to obtain the corresponding task processing result. Therefore, the task processing model can accurately process different types of tasks, thereby effectively improving the accuracy of the task processing results.

[0243] The task processing device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device, augmented reality / virtual reality device, robot, wearable device, super mobile personal computer, netbook, or personal digital assistant, etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the specific implementation.

[0244] The task processing device in this application embodiment can be a device with an operating system. The operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system.

[0245] The task processing model training apparatus provided in this application embodiment can implement the various processes implemented in the various embodiments of the above-described task processing model training method. To avoid repetition, it will not be described again here.

[0246] The task processing apparatus provided in this application embodiment can implement the various processes implemented in the various embodiments of the above-described task processing method. To avoid repetition, it will not be described again here.

[0247] Optionally, such as Figure 12 As shown, this application embodiment also provides an electronic device 1200, including a processor 1201 and a memory 1202. The memory 1202 stores a program or instructions that can run on the processor 1201. When the program or instructions are executed by the processor 1201, they implement the various steps of the above-described task processing model training method or task processing method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0248] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0249] Figure 13 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0250] The electronic device 1300 includes, but is not limited to, components such as: radio frequency unit 1301, network module 1302, audio output unit 1303, input unit 1304, sensor 1305, display unit 1306, user input unit 1307, interface unit 1308, memory 1309, and processor 1310.

[0251] Those skilled in the art will understand that the electronic device 1300 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 1310 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 13 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0252] The processor 1310 is used to input the task data of the first processing task into the task processing model, which includes a semantic extraction module and a base module.

[0253] The semantic extraction module obtains first fused semantic feature information that corresponds to both the task data and the first task attribute information.

[0254] The base module processes the first fused semantic feature information to obtain the task processing result of the first processing task;

[0255] The first task attribute information is the task attribute information corresponding to the task type to which the first processing task belongs.

[0256] In some embodiments of this application, the processor 1310 is further configured to:

[0257] Before inputting the task data of the first processing task into the task processing model, the first training data is input into the task processing model.

[0258] The semantic extraction module obtains second fused semantic feature information that corresponds to both the first training sample and the second task attribute information. The first training data includes the first training sample, and the second task attribute is the task attribute information corresponding to the task type to which the first training sample belongs.

[0259] The base module processes the second fused semantic feature information and outputs the first model inference result corresponding to the first training sample.

[0260] Based on the inference results of the first model and the first training data, the model parameters of the semantic extraction module are adjusted.

[0261] In some embodiments of this application, the processor 1310 is specifically used for:

[0262] The semantic extraction module obtains first concatenated feature information that corresponds to both the first training sample and the attribute information of the second task.

[0263] The semantic extraction module extracts semantic features from the first spliced ​​feature information to obtain the second fused semantic feature information.

[0264] In some embodiments of this application, the task processing model further includes a mapping module, and the acquisition module is further configured to:

[0265] Before obtaining the first concatenated feature information corresponding to both the first training sample and the second task attribute information through the semantic extraction module, the mapping module obtains the mapping sample feature information corresponding to the first training sample.

[0266] The processor 1310 is specifically used for:

[0267] The semantic extraction module concatenates the mapping sample feature information and the second task attribute information to obtain the first concatenated feature information.

[0268] In some embodiments of this application, the processor 1310 is specifically used for:

[0269] Freeze the model parameters of the base module and the model parameters of the semantic extraction module, and adjust the model parameters of the mapping module based on the first model inference result and the sample labels in the first training data.

[0270] After adjusting the model parameters of the mapping module, the semantic extraction module obtains third fused semantic feature information that corresponds to both the first training sample and the second task attribute information.

[0271] The base module processes the third fused semantic feature information to obtain the second model inference result corresponding to the first training sample;

[0272] The model parameters of the base module and the model parameters of the mapping module are frozen, and the model parameters of the semantic extraction module are adjusted based on the second model inference result and the first training data.

[0273] In some embodiments of this application, the first training sample is an audio sample, the mapping module includes an encoding unit and a mapping unit, and the processor 1310 is specifically used for:

[0274] The encoding unit extracts audio features from the first training sample to obtain the audio feature information of the audio sample.

[0275] The mapping unit maps the audio feature information to obtain the mapped sample feature information.

[0276] In some embodiments of this application, the first training data includes the training samples and sample labels, the training samples include at least one audio sample, and the processor 1310 is further configured to:

[0277] Obtain the first text feature information corresponding to the first audio sample;

[0278] The base module processes the first text feature information and the second task attribute information to obtain at least one inference result corresponding to the first audio sample.

[0279] The N inference results with the highest probability of the at least one inference result are determined as the sample labels corresponding to the first audio sample;

[0280] Wherein, the first audio sample is any one of the at least one audio samples, and N is a positive integer.

[0281] In some embodiments of this application, the processor 1310 is further configured to:

[0282] Based on the inference results of at least one first model and the first training dataset, before adjusting the model parameters of the semantic extraction module, the fusion submodule obtains fourth fused semantic feature information that corresponds to both the second text feature information and the second task attribute information.

[0283] The base module processes the fourth fused semantic feature information to obtain the third model inference result corresponding to the second audio sample;

[0284] From the first model inference result and the third model inference result corresponding to the second audio sample, determine the N model inference results with the highest probability;

[0285] The processor 1310 is specifically used for:

[0286] Based on the N model inference results, the N probability values ​​corresponding to the N model inference results, the N inference results corresponding to the second audio sample, and the probability values ​​corresponding to the N inference results, the model parameters of the semantic extraction module are adjusted.

[0287] Wherein, the second text feature information is the text feature information corresponding to the second audio sample, the second audio sample is any audio sample in the audio sample subset, and the audio sample subset includes M audio samples from the at least one audio sample, where M is a positive integer.

[0288] The electronic device provided in this application embodiment inputs task data of a first processing task into a task processing model. The task processing model includes a semantic extraction module and a base module. The semantic extraction module is used to implement the semantic extraction function in the base module, and the model parameters for semantic extraction are adjusted. The semantic extraction module obtains first fused semantic feature information that corresponds to both the task data and the first task attribute information. The base module processes the first fused semantic feature information to obtain the task processing result of the first processing task. The first task attribute information is the task attribute information corresponding to the task type to which the first processing task belongs. Thus, by adding a semantic extraction module to the task processing model, it is possible to extract fused semantic feature information that corresponds to both the task data of the first processing task and the task attribute information corresponding to the task type to which the first processing task belongs. This allows the base module to process the fused semantic feature information that corresponds to both the task data and the first task attribute information to obtain the task processing result of the first task. Consequently, when processing various types of processing tasks, the task processing model only needs to input the task data of that type of processing task into the task processing model, and the semantic extraction module can extract the fused semantic feature information that corresponds to both the task data of that type of processing task and the task attribute information corresponding to the corresponding task type. Thus, the base module can process the fused semantic feature information that corresponds to both the task data of that type of processing task and the task attribute information corresponding to the corresponding task type to obtain the corresponding task processing result. Therefore, the task processing model can accurately process different types of tasks, thereby effectively improving the accuracy of the task processing results.

[0289] It should be understood that, in this embodiment, the input unit 1304 may include a graphics processing unit (GPU) 13041 and a microphone 13042. The GPU 13041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 1306 may include a display panel 13061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 1307 includes a touch panel 13071 and at least one of other input devices 13072. The touch panel 13071 is also called a touch screen. The touch panel 13071 may include a touch detection device and a touch controller. Other input devices 13072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be described in detail here.

[0290] The memory 1309 can be used to store software programs and various data. The memory 1309 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 1309 may include volatile memory or non-volatile memory, or the memory 1009 may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 1309 in this embodiment includes, but is not limited to, these and any other suitable types of memory.

[0291] Processor 1310 may include one or more processing units; optionally, processor 1310 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 1310.

[0292] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described task processing method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here.

[0293] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0294] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described task processing method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0295] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0296] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the task processing method embodiments described above, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0297] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0298] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0299] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A task processing method, characterized in that, The method includes: The task data of the first processing task is input into the task processing model, which includes a semantic extraction module and a base module. The semantic extraction module obtains first fused semantic feature information that corresponds to both the task data and the first task attribute information. The base module processes the first fused semantic feature information to obtain the task processing result of the first processing task; The first task attribute information is the task attribute information corresponding to the task type to which the first task belongs.

2. The method according to claim 1, characterized in that, Before inputting the task data of the first processing task into the task processing model, the method further includes: Input the first training data into the task processing model; The semantic extraction module obtains second fused semantic feature information that corresponds to both the first training sample and the second task attribute information. The first training data includes the first training sample, and the second task attribute is the task attribute information corresponding to the task type to which the first training sample belongs. The base module processes the second fused semantic feature information and outputs the first model inference result corresponding to the first training sample. Based on the inference results of the first model and the first training data, the model parameters of the semantic extraction module are adjusted.

3. The method according to claim 2, characterized in that, The step of obtaining second fused semantic feature information corresponding to both the first training sample and the second task attribute information in the first training dataset through the semantic extraction module includes: The semantic extraction module obtains first concatenated feature information that corresponds to both the first training sample and the attribute information of the second task. The semantic extraction module extracts semantic features from the first spliced ​​feature information to obtain the second fused semantic feature information.

4. The method according to claim 3, characterized in that, The task processing model further includes a mapping module. Before obtaining the first concatenated feature information corresponding to both the first training sample and the second task attribute information through the semantic extraction module, the method further includes: The mapping module obtains the mapping sample feature information corresponding to the first training sample. The step of obtaining first concatenated feature information corresponding to both the first training sample and the second task attribute information through the semantic extraction module includes: The semantic extraction module concatenates the mapping sample feature information and the second task attribute information to obtain the first concatenated feature information.

5. The method according to claim 4, characterized in that, The step of adjusting the model parameters of the semantic extraction module based on the inference results of the first model and the first training data includes: Freeze the model parameters of the base module and the model parameters of the semantic extraction module, and adjust the model parameters of the mapping module based on the first model inference result and the sample labels in the first training data. After adjusting the model parameters of the mapping module, the semantic extraction module obtains third fused semantic feature information that corresponds to both the first training sample and the second task attribute information. The base module processes the third fused semantic feature information to obtain the second model inference result corresponding to the first training sample; The model parameters of the base module and the model parameters of the mapping module are frozen, and the model parameters of the semantic extraction module are adjusted based on the second model inference result and the first training data.

6. A task processing device, characterized in that, The device includes: An input unit is used to input the task data of the first processing task into the task processing model, the task processing model including a semantic extraction module and a base module; The acquisition unit is used to acquire, through the semantic extraction module, first fused semantic feature information that corresponds to both the task data and the first task attribute information; The processing unit is used to process the first fused semantic feature information through the base module to obtain the task processing result of the first processing task; The first task attribute information is the task attribute information corresponding to the task type to which the first processing task belongs.

7. The apparatus according to claim 6, characterized in that, The input module is also used for: Before inputting the task data of the first processing task into the task processing model, the first training data is input into the task processing model. The acquisition unit is further configured to acquire, through the semantic extraction module, second fused semantic feature information that corresponds to both the first training sample and the second task attribute information, wherein the first training data includes the first training sample and the second task attribute is the task attribute information corresponding to the task type to which the first training sample belongs. The device further includes: an output unit, used to process the second fused semantic feature information through the base module, and output the first model inference result corresponding to the first training sample; The adjustment unit is used to adjust the model parameters of the semantic extraction module based on the inference results of the first model and the first training data.

8. The apparatus according to claim 7, characterized in that, The acquisition unit is specifically used for: The semantic extraction module obtains first concatenated feature information that corresponds to both the first training sample and the attribute information of the second task. The semantic extraction module extracts semantic features from the first spliced ​​feature information to obtain the second fused semantic feature information.

9. The apparatus according to claim 8, characterized in that, The task processing model further includes a mapping module, and the acquisition unit is further used for: Before obtaining the first concatenated feature information corresponding to both the first training sample and the second task attribute information through the semantic extraction module, the mapping module obtains the mapping sample feature information corresponding to the first training sample. The acquisition unit is specifically used for: The semantic extraction module concatenates the mapping sample feature information and the second task attribute information to obtain the first concatenated feature information.

10. The apparatus according to claim 9, characterized in that, The acquisition unit is specifically used for: Freeze the model parameters of the base module and the model parameters of the semantic extraction module, and adjust the model parameters of the mapping module based on the first model inference result and the sample labels in the first training data. After adjusting the model parameters of the mapping module, the semantic extraction module obtains third fused semantic feature information that corresponds to both the first training sample and the second task attribute information. The base module processes the third fused semantic feature information to obtain the second model inference result corresponding to the first training sample; The model parameters of the base module and the model parameters of the mapping module are frozen, and the model parameters of the semantic extraction module are adjusted based on the second model inference result and the first training data.

11. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a program or instructions that can run on the processor, the program or instructions being executed by the processor to implement the steps of the task processing method as described in any one of claims 1 to 5.

Images