Task processing method, automatic question answering method and multimedia data identification model training method

Abstract

Provided in the embodiments of the present description are a task processing method, an automatic question answering method and a multimedia data identification model training method, the task processing method comprising: acquiring target multimedia data and task processing data for the target multimedia data; inputting the target multimedia data into a multimedia data identification model to obtain multimedia features corresponding to the target multimedia data, the multimedia data identification model being trained on the basis of a plurality of sample multimedia text pairs and sample tasks corresponding to the sample multimedia text pairs, and the sample multimedia text pairs comprising sample multimedia data and sample text data corresponding to the sample multimedia data; and inputting the multimedia features and the task processing data into a neural network model for natural language processing so as to obtain a processing result corresponding to the task processing data. Combining the multimedia data identification model and the neural network model for natural language processing to process complex multimedia tasks improves the universality of multimedia task processing.

Description

Task processing, automatic question answering, and multimedia data recognition model training methods

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on July 27, 2023, with application number 202310935405.3 and application name “Task processing, automatic question answering and multimedia data recognition model training method”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The embodiments of this specification relate to the field of computer technology, and in particular to a task processing method. Background Art

[0003] With the development of computer technology, text communication is gradually unable to meet the actual needs of users. Users mainly use multimedia data to express their emotions, tone, environment and other information in natural conversations. Therefore, multimedia signals that record rich knowledge have gradually become a common and important means of communication.

[0004] Currently, conversational robots are often combined with the input / output interfaces of multimedia data processing models to support multimedia conversations and multimedia data processing tasks. However, these solutions rely on invoking external software interfaces, which is limited by the capabilities of the external software. Furthermore, each task is implemented using a separate external model, making it incapable of handling complex multimedia understanding tasks. Therefore, a highly universal and general multimedia data processing solution is urgently needed.

[0005] Summary of the Invention

[0006] In view of this, embodiments of this specification provide a task processing method. One or more embodiments of this specification also relate to an automatic question-answering method, a multimedia data recognition model training method, a task processing device, an automatic question-answering device, a multimedia data recognition model training device, a computing device, a computer-readable storage medium, and a computer program to address technical deficiencies in the prior art.

[0007] According to a first aspect of an embodiment of this specification, a task processing method is provided, including:

[0008] Acquiring target multimedia data and task processing data for the target multimedia data;

[0009] Inputting target multimedia data into a multimedia data recognition model to obtain multimedia features corresponding to the target multimedia data, wherein the multimedia data recognition model is trained based on multiple sets of sample multimedia text pairs and sample tasks corresponding to the sample multimedia text pairs, wherein the sample multimedia text pairs include sample multimedia data and sample text data corresponding to the sample multimedia data;

[0010] The multimedia features and task processing data are input into the neural network model for natural language processing to obtain the processing results corresponding to the task processing data.

[0011] According to a second aspect of the embodiments of this specification, an automatic question-answering method is provided, comprising:

[0012] receiving target multimedia data and question data corresponding to the target multimedia data sent by a user through a client;

[0013] Inputting target multimedia data into a multimedia data recognition model to obtain multimedia features corresponding to the target multimedia data, wherein the multimedia data recognition model is trained based on multiple sets of sample multimedia text pairs and sample tasks corresponding to the sample multimedia text pairs, wherein the sample multimedia text pairs include sample multimedia data and sample text data corresponding to the sample multimedia data;

[0014] The multimedia features and question data are input into the neural network model for natural language processing to obtain the answer results corresponding to the question data.

[0015] According to a third aspect of the embodiments of this specification, a multimedia data recognition model training method is provided, which is applied to a cloud-side device, including:

[0016] Acquire multiple groups of sample multimedia text pairs, wherein the sample multimedia text pairs carry sample tasks;

[0017] According to the target sample task, extracting a first sample multimedia text pair from the multiple groups of sample multimedia text pairs, wherein the first sample multimedia text pair is any one of the multiple groups of sample multimedia text pairs corresponding to the target sample task;

[0018] Inputting the first sample multimedia data into the multimedia data recognition model to obtain a first sample multimedia feature corresponding to the first sample multimedia data;

[0019] Inputting the first sample multimedia feature and the first sample text data into a neural network model for natural language processing to obtain a first prediction result;

[0020] Training a multimedia data recognition model based on the first sample text data and the first prediction result, and obtaining model parameters of the trained multimedia data recognition model;

[0021] Send the model parameters of the trained multimedia data recognition model to the end-side device.

[0022] According to a fourth aspect of the embodiments of this specification, there is provided a task processing device, including:

[0023] a first acquisition module configured to acquire target multimedia data and task processing data for the target multimedia data;

[0024] a first input module configured to input target multimedia data into a multimedia data recognition model to obtain multimedia features corresponding to the target multimedia data, wherein the multimedia data recognition model is trained based on a plurality of sets of sample multimedia text pairs and sample tasks corresponding to the sample multimedia text pairs, wherein the sample multimedia text pairs include the sample multimedia data and sample text data corresponding to the sample multimedia data;

[0025] The second input module is configured to input multimedia features and task processing data into a neural network model for natural language processing to obtain processing results corresponding to the task processing data.

[0026] According to a fifth aspect of the embodiments of this specification, an automatic question-answering device is provided, comprising:

[0027] A receiving module configured to receive target multimedia data and question data corresponding to the target multimedia data sent by a user through a client;

[0028] a third input module configured to input the target multimedia data into a multimedia data recognition model to obtain multimedia features corresponding to the target multimedia data, wherein the multimedia data recognition model is trained based on a plurality of sets of sample multimedia text pairs and sample tasks corresponding to the sample multimedia text pairs, the sample multimedia text pairs including the sample multimedia data and sample text data corresponding to the sample multimedia data;

[0029] The fourth input module is configured to input multimedia features and question data into a neural network model for natural language processing to obtain a response result corresponding to the question data.

[0030] According to a sixth aspect of the embodiments of this specification, a multimedia data recognition model training device is provided, which is applied to a cloud-side device, including:

[0031] A second acquisition module is configured to acquire multiple groups of sample multimedia text pairs, wherein the sample multimedia text pairs carry sample tasks;

[0032] an extraction module configured to extract a first sample multimedia text pair from the plurality of sample multimedia text pairs according to the target sample task, wherein the first sample multimedia text pair is any one of the plurality of sample multimedia text pairs corresponding to the target sample task;

[0033] a fifth input module configured to input the first sample multimedia data into the multimedia data recognition model to obtain a first sample multimedia feature corresponding to the first sample multimedia data;

[0034] a sixth input module, configured to input the first sample multimedia feature and the first sample text data into a neural network model for natural language processing to obtain a first prediction result;

[0035] A first training module is configured to train a multimedia data recognition model based on the first sample text data and the first prediction result, and obtain model parameters of the trained multimedia data recognition model;

[0036] The sending module is configured to send the model parameters of the trained multimedia data recognition model to the terminal device.

[0037] According to a seventh aspect of the embodiments of this specification, a computing device is provided, including:

[0038] memory and processor;

[0039] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the method provided in the first aspect, the second aspect, or the third aspect are implemented.

[0040] According to an eighth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions. When the instructions are executed by a processor, the steps of the method provided in the first aspect, the second aspect, or the third aspect are implemented.

[0041] According to a ninth aspect of the embodiments of this specification, a computer program is provided, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the method provided in the first aspect, the second aspect, or the third aspect above.

[0042] An embodiment of the present specification provides a task processing method, which obtains target multimedia data and task processing data for the target multimedia data; inputs the target multimedia data into a multimedia data recognition model to obtain multimedia features corresponding to the target multimedia data, wherein the multimedia data recognition model is trained based on multiple sets of sample multimedia text pairs and sample tasks corresponding to the sample multimedia text pairs, wherein the sample multimedia text pairs include sample multimedia data and sample text data corresponding to the sample multimedia data; and inputs the multimedia features and task processing data into a neural network model for natural language processing to obtain processing results corresponding to the task processing data. By utilizing multiple sets of sample multimedia text pairs and their corresponding sample tasks for training, a multimedia data recognition model capable of complex multimedia understanding is obtained, thereby combining the multimedia data recognition model with the neural network model for natural language processing to achieve processing of complex multimedia tasks and improve the universality of multimedia task processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] FIG1 is an architecture diagram of a task processing system provided by one embodiment of this specification;

[0044] FIG2 is an architecture diagram of another task processing system provided by one embodiment of this specification;

[0045] FIG3 is a flowchart of a task processing method provided by one embodiment of this specification;

[0046] FIG4 is a flowchart of training an audio recognition model in a task processing method provided by one embodiment of this specification;

[0047] FIG5 is a flowchart of a training process of a neural network model for natural language processing in a task processing method provided by one embodiment of this specification;

[0048] FIG6 is a flow chart of an automatic question-answering method provided by one embodiment of this specification;

[0049] FIG7 is a flowchart of a multimedia data recognition model training method provided by one embodiment of this specification;

[0050] FIG8 is a flowchart of a task processing method according to an embodiment of the present disclosure;

[0051] FIG9 is a schematic diagram of an automatic question-and-answer interface provided by one embodiment of this specification;

[0052] FIG10 is a schematic diagram of the structure of a task processing device provided by one embodiment of this specification;

[0053] FIG11 is a schematic diagram of the structure of an automatic question-answering device provided by one embodiment of this specification;

[0054] FIG12 is a schematic structural diagram of a multimedia data recognition model training device provided by one embodiment of this specification;

[0055] FIG13 is a structural block diagram of a computing device provided in one embodiment of this specification. DETAILED DESCRIPTION

[0056] The following description sets forth many specific details to facilitate a thorough understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the scope of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0057] The terms used in one or more embodiments of this specification are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of this specification. The singular forms "a," "the," and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0058] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0059] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0060] In one or more embodiments of this specification, a large model refers to a deep learning model with large-scale model parameters, typically containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even more than ten trillion model parameters. A large model can also be called a cornerstone model / foundation model. It is pre-trained on a large-scale unlabeled corpus to produce a pre-trained model with more than 100 million parameters. This model can adapt to a wide range of downstream tasks and has good generalization capabilities, such as a large language model (LLM) and a multi-modal pre-training model.

[0061] When large models are used in practice, only a small number of samples are needed to fine-tune the pre-trained model and it can be applied to different tasks. Large models can be widely used in natural language processing (NLP), computer vision and other fields. Specifically, they can be applied to computer vision tasks such as visual question answering (VQA), image caption (IC), and image generation, as well as natural language processing tasks such as text-based sentiment classification, text summary generation, and machine translation. The main application scenarios of large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.

[0062] With the advancement of computer technology, large models have begun to shine. Their remarkable capabilities in language understanding, generation, interaction, and reasoning have led to their widespread application in natural language processing fields such as conversation, translation, and code generation. However, existing large models only support written language and lack the ability to perceive multimedia signals such as audio, video, and images. However, multimedia signals are extremely common and important. Take audio signals, for example. On the one hand, they are ubiquitous in the real world and include various types such as human voices, background sounds, and music, recording a wealth of knowledge. On the other hand, users primarily use spoken language in natural conversations, and information such as their emotions, tone of voice, and their surroundings is often conveyed through audio. Therefore, how to enable users to more naturally interact with large models using multimedia data has become a research priority.

[0063] Currently, conversational robots are often combined with the input / output interfaces of multimedia data processing models to support multimedia conversation and data processing tasks. However, these solutions rely on invoking external software interfaces, limiting their effectiveness to the capabilities of the external software. Furthermore, each task is implemented in isolation through independent external models, making it difficult to complete more complex multimedia understanding and conversation tasks. Furthermore, the conversational robots also lack intelligence.

[0064] In order to solve the above-mentioned solution, the embodiments of this specification construct a general multimedia data recognition model, which is connected to the neural network model for natural language processing, so that the neural network model for natural language processing can support multiple types of multimedia data input, and realize the understanding tasks of various types of multimedia data, including video description, image style recognition, audio description, music style recognition, speech recognition, ambient sound recognition, background sound recognition and other tasks, and realize intelligent question-answering tasks based on multimedia understanding, thereby expanding the cognitive ability of the neural network model for natural language processing, so that the neural network model for natural language processing can better understand the content, background, emotion and status of user input, and improve user interactivity.

[0065] Specifically, target multimedia data and task processing data for the target multimedia data are obtained; the target multimedia data is input into a multimedia data recognition model to obtain multimedia features corresponding to the target multimedia data, wherein the multimedia data recognition model is trained based on multiple sets of sample multimedia text pairs and sample tasks corresponding to the sample multimedia text pairs, wherein the sample multimedia text pairs include sample multimedia data and sample text data corresponding to the sample multimedia data; the multimedia features and task processing data are input into a neural network model for natural language processing to obtain processing results corresponding to the task processing data. By training with multiple sets of sample multimedia text pairs and their corresponding sample tasks, a multimedia data recognition model capable of complex multimedia understanding is obtained, thereby combining the multimedia data recognition model with the neural network model for natural language processing to achieve processing of complex multimedia tasks and improve the universality of multimedia task processing.

[0066] In this specification, a task processing method is provided. This specification also involves an automatic question-answering method, a multimedia data recognition model training method, a task processing device, an automatic question-answering device, a multimedia data recognition model training device, a computing device, a computer-readable storage medium and a computer program, which are described in detail one by one in the following embodiments.

[0067] Referring to FIG1 , FIG1 shows an architecture diagram of a task processing system provided by an embodiment of this specification. The task processing system may include a client 100 and a server 200;

[0068] The client 100 is configured to send target multimedia data and task processing data corresponding to the target multimedia data to the server 200;

[0069] The server 200 is configured to input target multimedia data into a multimedia data recognition model to obtain multimedia features corresponding to the target multimedia data, wherein the multimedia data recognition model is trained based on multiple sets of sample multimedia text pairs and sample tasks corresponding to the sample multimedia text pairs, wherein the sample multimedia text pairs include sample multimedia data and sample text data corresponding to the sample multimedia data; input the multimedia features and task processing data into a neural network model for natural language processing to obtain processing results corresponding to the task processing data; and send the processing results corresponding to the task processing data to the client 100;

[0070] The client 100 is further configured to receive the processing result corresponding to the task processing data sent by the server 200 .

[0071] By applying the solution of the embodiments of this specification, a multimedia data recognition model capable of complex multimedia understanding is obtained by training with multiple groups of sample multimedia text pairs and their corresponding sample tasks, thereby combining the multimedia data recognition model with a neural network model for natural language processing to realize the processing of complex multimedia tasks and improve the universality of multimedia task processing.

[0072] Referring to FIG. 2 , FIG. 2 shows an architecture diagram of another task processing system provided by one embodiment of this specification. The task processing system may include multiple clients 100 and a server 200. The clients 100 may be referred to as end-side devices, and the server 200 may be referred to as cloud-side devices. Multiple clients 100 may establish communication connections via the server 200. In a task processing scenario, the server 200 is used to provide multimedia task processing services between the multiple clients 100. The multiple clients 100 may act as either senders or receivers, respectively, and communicate via the server 200.

[0073] The user can interact with the server 200 through the client 100 to receive data sent by other clients 100, or send data to other clients 100, etc. In the task processing scenario, the user can publish a data stream to the server 200 through the client 100, and the server 200 generates a task processing result based on the data stream and pushes the task processing result to other clients that have established communication. Among them, the client 100 and the server 200 are connected through the network. The network provides a medium for the communication link between the client 100 and the server 200. The network can include various connection types, such as wired, wireless communication links or optical fiber cables, etc. The data transmitted by the client 100 may need to be encoded, transcoded, compressed, etc. before being published to the server 200.

[0074] The client 100 can be a browser, an APP (Application), or a web application such as an H5 (HyperText Markup Language 5, version 5) application, a light application (also known as a mini-program, a lightweight application), or a cloud application. The client 100 can be developed based on the software development kit (SDK) of the corresponding service provided by the server 200, such as a real-time communication (RTC) SDK. The client 100 can be deployed in an electronic device and needs to rely on the device to run or certain APPs in the device to run. For example, the electronic device can have a display screen and support information browsing, such as a personal mobile terminal such as a mobile phone, tablet computer, personal computer, etc. Various other types of applications can also be configured in the electronic device, such as human-computer dialogue applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0075] The server 200 may include servers that provide various services, such as servers that provide communication services to multiple clients, servers that provide background training to support models used on clients, and servers that process data sent by clients. It should be noted that the server 200 can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. The server can also be a server in a distributed system, or a server that is integrated with a blockchain. The server can also be a cloud server for basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0076] It is worth noting that the task processing methods provided in the embodiments of this specification are generally executed by the server. However, in other embodiments of this specification, the client may also have similar functions to the server and thus execute the task processing methods provided in the embodiments of this specification. In other embodiments, the task processing methods provided in the embodiments of this specification may also be executed jointly by the client and the server.

[0077] Referring to FIG3 , FIG3 shows a flowchart of a task processing method provided by an embodiment of this specification, which specifically includes the following steps:

[0078] Step 302: Acquire target multimedia data and task processing data for the target multimedia data.

[0079] In one or more embodiments of this specification, target multimedia data and task processing data for the target multimedia data may be acquired, thereby performing task processing based on the target multimedia data and the task processing data to obtain a task processing result.

[0080] Specifically, the target multimedia data includes but is not limited to target audio, target video, target image, and the like. The target multimedia data can be data in different scenarios, such as a conference video in a meeting scenario, rain sounds in a natural scenario, and the like. The target multimedia data can be different types of multimedia data. For example, the target audio can be human voice, background sound, music, and the like; the target video can be a person video, a landscape video, a music video, and the like; and the target image can be a person image, a landscape image, and the like. Task processing data for target multimedia data can be understood as questions about the target multimedia data, such as the speaker's gender, the music style, and the like. The specific selection is based on the actual situation, and the embodiments of this specification do not impose any restrictions on this.

[0081] In practical applications, there are multiple ways to obtain target multimedia data and task processing data for the target multimedia data. The specific method to be selected depends on the actual situation, and the embodiments of this specification do not impose any restrictions on this. In one possible implementation of this specification, the target multimedia data and task processing data for the target multimedia data can be received from a user. In another possible implementation of this specification, the target multimedia data and task processing data for the target multimedia data can be read from other data acquisition devices or databases.

[0082] It should be noted that the target multimedia data to be acquired may be one or more, and one target multimedia data may correspond to one task processing data or multiple task processing data.

[0083] For example, taking the target multimedia data as target audio, if one target audio corresponds to one task processing data, the task can be understood as a single-audio, single-round dialogue task, that is, one round of question and answer for one audio. For example, input audio 1 and question 1, and obtain answer 1 to complete the dialogue task. If one target audio corresponds to multiple task processing data, the task can be understood as a single-audio, multi-round dialogue task, that is, multiple rounds of question and answer for one audio. For example, input audio 1 and question 1, obtain answer 1, then input question 2, and obtain answer 2 to complete the dialogue task. If multiple target audios correspond to multiple task processing data, the task can be understood as a multi-audio, multi-round dialogue task, that is, multiple rounds of question and answer for multiple audios. For example, input audio 1 and question 1, obtain answer 1, then input audio 2 and question 2, and obtain answer 2 to complete the dialogue task.

[0084] Step 304: Input the target multimedia data into a multimedia data recognition model to obtain multimedia features corresponding to the target multimedia data, wherein the multimedia data recognition model is trained based on multiple groups of sample multimedia text pairs and sample tasks corresponding to the sample multimedia text pairs, and the sample multimedia text pairs include sample multimedia data and sample text data corresponding to the sample multimedia data.

[0085] In one or more embodiments of this specification, after obtaining target multimedia data and task processing data for the target multimedia data, the target multimedia data may be further input into a multimedia data recognition model to obtain multimedia features corresponding to the target multimedia data.

[0086] Specifically, multimedia data recognition models are machine learning models, which can be understood as trained programs that can discover patterns and make predictions in new data. These models are represented as mathematical functions that receive input data, make predictions based on the input data, and then provide outputs in response. Before inputting target multimedia data into the multimedia data recognition model, a pre-trained multimedia data recognition model can be obtained, or the model can be trained using multiple sets of sample multimedia text pairs.

[0087] In an optional embodiment of the present specification, before inputting the target multimedia data into the multimedia data recognition model and obtaining multimedia features corresponding to the target multimedia data, the following steps may be further included:

[0088] Acquire multiple groups of sample multimedia text pairs, wherein the sample multimedia text pairs carry sample tasks;

[0089] According to the target sample task, extracting a first sample multimedia text pair from the multiple groups of sample multimedia text pairs, wherein the first sample multimedia text pair is any one of the multiple groups of sample multimedia text pairs corresponding to the target sample task;

[0090] Inputting the first sample multimedia data into the multimedia data recognition model to obtain a first sample multimedia feature corresponding to the first sample multimedia data;

[0091] Inputting the first sample multimedia feature and the first sample text data into a neural network model for natural language processing to obtain a first prediction result;

[0092] Calculating a first loss value according to the first sample text data and the first prediction result;

[0093] Adjust the model parameters of the multimedia data recognition model according to the first loss value, and return to execute the step of extracting the first sample multimedia text pair from multiple groups of sample multimedia text pairs according to the target sample task until the first preset stopping condition is reached to obtain a trained multimedia data recognition model.

[0094] Specifically, the multimedia data recognition model is trained using supervised training. The sample text data in the multiple sets of sample multimedia text pairs are the true labels of the multimedia data recognition process, that is, the multimedia data recognition training targets, which are used to guide the training process of the multimedia data recognition model. The sample multimedia text pairs are used to train the multimedia data recognition model. The sample multimedia text pairs include sample multimedia data and sample text data. The sample multimedia data can be multimedia data of different scenes and types. The sample text data can be descriptive information of the sample multimedia data, such as a title or introduction, or it can be content information in the sample multimedia data, such as lyrics or speech content. The sample task refers to the sample processing task corresponding to the sample multimedia text pair. The sample task includes at least one of a sample multimedia understanding task and a sample multimedia recognition task.

[0095] In actual applications, there are many ways to obtain multiple groups of sample multimedia text pairs, which can be selected based on actual conditions. The embodiments of this specification do not impose any restrictions on this. In one possible implementation of this specification, a large number of sample multimedia text pairs carrying sample tasks can be received from a user. In another possible implementation of this specification, a large number of sample multimedia text pairs carrying sample tasks can be read from other data acquisition devices or databases. In yet another possible implementation of this specification, a large amount of original multimedia data and original text data corresponding to the original multimedia data can be obtained to construct multiple groups of sample multimedia text pairs.

[0096] It should be noted that in the process of training the multimedia data recognition model, in order to prevent the multimodality of the sample multimedia data from having a negative impact on the text processing capability of the neural network model used for natural language processing, when adjusting the model parameters according to the first loss value, the model parameters of the neural network model used for natural language processing can be fixed and not trained, and only the model parameters of the multimedia data recognition model can be trained.

[0097] In one possible implementation of this specification, the first preset stopping condition includes the first loss value being less than or equal to a first preset threshold value. After calculating the first loss value based on the first sample text data and the first prediction result, the first loss value is compared with the first preset threshold value.

[0098] Specifically, if the first loss value is greater than the first preset threshold, it means that the difference between the first sample text data and the first prediction result is large, and the multimedia data recognition model has poor prediction ability for the first prediction result. At this time, the model parameters of the multimedia data recognition model can be adjusted, and the step of extracting the first sample multimedia text pair from multiple groups of sample multimedia text pairs according to the target sample task is returned to be executed, and the multimedia data recognition model is continued to be trained until the first loss value is less than or equal to the first preset threshold, indicating that the difference between the first sample text data and the first prediction result is small, and the first preset stop condition is met, and a multimedia data recognition model that has completed training is obtained.

[0099] In another possible implementation of the present specification, in addition to comparing the first loss value and the first preset threshold, it is also possible to determine whether the current multimedia data recognition model is trained in combination with a preset number of iterations.

[0100] Specifically, if the first loss value is greater than the first preset threshold, the model parameters of the multimedia data recognition model are adjusted, and the step of extracting the first sample multimedia text pair from multiple groups of sample multimedia text pairs according to the target sample task is returned to the execution, and the multimedia data recognition model is continued to be trained until the first preset number of iterations is reached, and the iteration is stopped to obtain a multimedia data recognition model that has completed the training. The first preset threshold and the first preset number of iterations are selected according to the actual situation, and the embodiments of this specification do not impose any restrictions on this.

[0101] In practical applications, there are many functions for calculating the first loss value, such as i-k ,...,u i-1 To predict the current text word u i , the first loss value is calculated using the negative logarithmic maximum likelihood loss function, that is, the first loss value can be calculated by the following formula (1):

[0102] Among them, L(u) is the first loss value, P is the probability of the text word appearing, u i-k ,...,u i-1 is the first K text words, u i is the current text word, Θ is the model parameter.

[0103] Using the solution of the embodiments of this specification, a first loss value is calculated based on the first sample text data and the first prediction result, and the first loss value is compared with a first preset stopping condition. If the first preset stopping condition is not met, the multimedia data recognition model is trained continuously until the first preset stopping condition is met, thereby completing the training and obtaining the multimedia data recognition model. By continuously adjusting the model parameters of the multimedia data recognition model, the resulting multimedia data recognition model can be made more accurate.

[0104] In practical applications, based on the target sample task, before extracting the first sample multimedia text pair from multiple groups of sample multimedia text pairs, the target sample task can be determined. There are many ways to determine the target sample task, which can be selected based on actual conditions, and this specification does not impose any limitation on this.

[0105] In a possible implementation of this specification, a sample multimedia understanding task or a sample multimedia recognition task may be directly used as a target sample task.

[0106] In another possible implementation of the present specification, since the multimedia data types processed by the multimedia data recognition model are relatively complex, including complex acoustic scenes including human speech, using only one sample task to train the multimedia data recognition model may cause the multimedia data recognition model to perform well on the scene understanding task but perform poorly on the multimedia data recognition task. Therefore, the multimedia data recognition model can be trained in two stages using a course learning method. The two stages correspond to different target sample tasks, that is, the sample tasks include a sample multimedia understanding task and a sample multimedia recognition task. Before extracting the first sample multimedia text pair from multiple groups of sample multimedia text pairs according to the target sample task, the following steps may also be included:

[0107] When the first loss value does not satisfy the task switching condition, determining the target sample task to be a sample multimedia understanding task;

[0108] When the first loss value meets the task switching condition, the target sample tasks are determined to be the sample multimedia understanding task and the sample multimedia recognition task.

[0109] Specifically, the sample multimedia understanding task can be called a multimedia data title automatic generation task, such as the audio title automatic generation task (AAC, Automatic Audio Captioning). The sample text data in the sample multimedia text pair used in the sample multimedia understanding task is a simple summary of the entire scene information. The task goal of the sample multimedia understanding task is to generate a summary of the entire scene information. For example, "someone is talking, and there are birds singing in the background." The sample multimedia recognition task can be called a multimedia data recognition task, such as the audio recognition task (ASR, Automatic Speech Recognition). The sample text data in the sample multimedia text pair used in the sample multimedia recognition task not only includes a summary of the entire scene information, but also includes the natural language text information in the sample multimedia data. For example, converting the speech of a human voice into the natural language text information corresponding to its syllables and words. The task goal of the sample multimedia recognition task is to generate a summary of the entire scene information and the natural language text information corresponding to the multimedia data. For example, "someone is saying: "Have you eaten?", and there are birds singing in the background."

[0110] It should be noted that during the first round of training, the target sample task can be set by default as the sample multimedia understanding task. During the training process, it is determined whether the first loss value meets the task switching condition. If it does not meet the condition, it indicates that the multimedia data recognition model has not yet converged, and the sample multimedia understanding task will continue to be executed. If it does meet the condition, it indicates that the multimedia data recognition model has converged and can stably process the sample multimedia understanding task. At this time, the sample multimedia recognition task can be added to the sample multimedia understanding task to carry out the second stage of learning and training. Among them, the task switching condition is that the first loss value is less than the task switching loss value. The task switching loss value is selected according to the actual situation and is not limited in this embodiment of the present specification.

[0111] By applying the solution of the embodiments of this specification, it is determined whether the first loss value satisfies the task switching condition. If the first loss value does not satisfy the task switching condition, the target sample task is determined to be the sample multimedia understanding task; if the first loss value does satisfy the task switching condition, the target sample task is determined to be both the sample multimedia understanding task and the sample multimedia recognition task. By dividing the target sample task into two-stage learning tasks, the multimedia data recognition model can capture complex acoustic scenes at multiple granularities, allowing the multimedia data recognition model to handle a wider range of tasks and improving its universality.

[0112] In an optional embodiment of the present specification, the obtaining of multiple groups of sample multimedia text pairs may include the following steps:

[0113] Acquiring a plurality of original multimedia data and original text data corresponding to the original multimedia data, wherein the original multimedia data includes at least one of ambient sound data, human voice data, and music data;

[0114] Performing data enhancement on original multimedia data and original text data to obtain enhanced multimedia data and enhanced text data;

[0115] Based on the enhanced multimedia data and enhanced text data, multiple groups of sample multimedia-text pairs are constructed.

[0116] In actual applications, there are many ways to obtain multiple original multimedia data and the original text data corresponding to the original multimedia data. The specific method is selected according to the actual situation. The embodiments of this specification do not impose any restrictions on this. In one possible implementation of this specification, various types of original multimedia data and original text data corresponding to the original multimedia data input by the user can be received. In another possible implementation of this specification, various types of original multimedia data and original text data corresponding to the original multimedia data can be collected from the Internet and various public data sources.

[0117] It should be noted that after the original multimedia data and the original text data corresponding to the original multimedia data are obtained, the original multimedia data and the original text data may be processed according to the data type of the original multimedia data.

[0118] Specifically, for example, if the original multimedia data is raw audio data, and the original audio data is ambient sound data, the original text data corresponding to the ambient sound data is primarily the audio title or a description of the audio uploaded by the user to a website, which is typically quite cluttered and contains a lot of noise. In one possible implementation of this specification, an audio-text matching model can be used to determine a matching index between the original audio data and the original text data, and the original audio data can be filtered based on the matching index.

[0119] If the original audio data is human voice data: the original text data corresponding to the human voice data can be divided into two categories, one is the speaker's meta-information, including emotions, language, accent, etc., and the other is the content information in the human voice. In one possible implementation of this specification, since the original text data usually does not have a fixed format, in order to make it easier for the multimedia data recognition model to match the neural network model used for natural language processing, this meta-information and content information can be combined into natural language to form descriptive text data for the human voice data. In another possible implementation of this specification, since the human voice data is mostly recorded in a recording studio with better conditions, the background sound is relatively clean and does not quite conform to the actual user's speaking environment. Therefore, the human voice data can be artificially noised and reverberated to make it more in line with the actual situation.

[0120] If the original audio data is music data: the text information corresponding to the music data includes label information such as music style, playing instruments, singing voice, and lyrics, which is similar to vocal data. Therefore, the original text data corresponding to the music data can also be combined into natural language to form descriptive text data for the music data.

[0121] Furthermore, since the original multimedia data and original text data directly obtained may be relatively simple, the original multimedia data and original text data can be enhanced to obtain enhanced multimedia data and enhanced text data, and multiple groups of sample multimedia text pairs can be constructed based on the enhanced multimedia data and enhanced text data.

[0122] In actual applications, there are many ways to perform data enhancement on original multimedia data and original text data to obtain enhanced multimedia data and enhanced text data. The specific selection depends on the actual situation, and the embodiments of this specification do not impose any restrictions on this. Taking the original multimedia data as original audio data as an example, in one possible implementation method of this specification, the music data and the human voice data can be audio mixed, and the original text data corresponding to the two can be spliced ​​to construct enhanced audio data with both music and human voice and its corresponding enhanced text data. In another possible implementation method of this specification, the multimedia data recognition model can be used to identify the ambient sound data, determine the content text of the human voice in the ambient sound data, and construct enhanced audio data with both ambient sound data and human voice data and its corresponding enhanced text data.

[0123] By applying the solution of the embodiments of this specification, a plurality of original multimedia data and original text data corresponding to the original multimedia data are obtained, wherein the original multimedia data includes at least one of ambient sound data, human voice data, and music data; data enhancement is performed on the original multimedia data and original text data to obtain enhanced multimedia data and enhanced text data; and multiple groups of sample multimedia text pairs are constructed based on the enhanced multimedia data and enhanced text data. When constructing the sample multimedia text pairs, all types of sample multimedia data are considered, so the trained multimedia data recognition model can support different types of multimedia data input. By performing data enhancement on the original multimedia data and original text data, the situation where the multimedia data recognition model only performs well on one type of data due to a single data source is avoided, making the multimedia data recognition model more accurate and more universal.

[0124] It's worth noting that once the multimedia data recognition model is trained, it can be combined with a neural network model used for natural language processing to enable intelligent question-answering based on multimedia data input. The multimedia data recognition model can also be used independently as a general multimedia data pre-training model and fine-tuned for use in various multimedia data processing tasks.

[0125] Referring to Figure 4, Figure 4 shows a training flow chart of an audio recognition model in a task processing method provided by an embodiment of the present specification. As shown in Figure 4, multiple groups of sample audio text pairs are obtained, wherein the sample audio text pairs carry sample tasks; according to the target sample task, any sample audio text pair is extracted from the multiple groups of sample audio text pairs; the sample audio data in the sample audio text pair is input into the audio recognition model to obtain sample audio features corresponding to the sample audio data; feature extraction is performed on the sample text data "A dog barks while a man speaks" in the sample audio text pair to obtain sample text features; the sample audio features are input into the linear layer, the feature dimensions of the sample audio features are aligned with the feature dimensions of the sample text features, and the sample audio features after dimension conversion are obtained; the sample audio features and the sample text features after dimension conversion are input into a neural network model for natural language processing to obtain a prediction result; a loss value is calculated based on the sample text data "A dog barks while a man speaks" and the prediction result; the model parameters of the audio recognition model are adjusted according to the loss value, and the step of extracting any sample audio text pair from the multiple groups of sample audio text pairs according to the target sample task is returned to execute until the first preset stop condition is reached, thereby obtaining a trained audio recognition model.

[0126] Step 306: Input the multimedia features and task processing data into the neural network model for natural language processing to obtain a processing result corresponding to the task processing data.

[0127] In one or more embodiments of the present specification, target multimedia data and task processing data for the target multimedia data are obtained, the target multimedia data are input into a multimedia data recognition model, and multimedia features corresponding to the target multimedia data are obtained. Then, the multimedia features and the task processing data can be further input into a neural network model for natural language processing to obtain processing results corresponding to the task processing data.

[0128] Specifically, the neural network model used for natural language processing, also known as the deep learning model for natural language, can be understood as a large model. The processing result corresponding to the task processing data is the answer to the task processing data.

[0129] By applying the solution of the embodiments of this specification, a multimedia data recognition model for understanding all types of multimedia, including human voice, environmental sound, music, etc., is constructed by leveraging the powerful language understanding and generation capabilities of the neural network model for natural language processing. This combines the multimedia data recognition model with the neural network model for natural language processing to achieve the processing of complex multimedia tasks and improve the universality of multimedia task processing.

[0130] In practical applications, there are multiple ways to input multimedia features and task processing data into a neural network model for natural language processing to obtain processing results corresponding to the task processing data. The specific method to be selected depends on the actual situation, and the embodiments of this specification do not impose any restrictions on this. In one possible implementation of this specification, multimedia features and task processing data can be input into a neural network model for natural language processing, and feature extraction is performed on the task processing data in the neural network model for natural language processing to obtain text features. The neural network model for natural language processing further processes the multimedia features and text features to obtain processing results corresponding to the task processing data.

[0131] In another possible implementation of the present specification, before inputting the task processing data into the neural network model for natural language processing, feature extraction may be performed on the task processing data to obtain text features, and then the text features and multimedia features may be input into the neural network model for natural language processing. That is, the above-mentioned inputting multimedia features and task processing data into the neural network model for natural language processing to obtain processing results corresponding to the task processing data may include the following steps:

[0132] Perform feature extraction on the task processing data to obtain text features corresponding to the task processing data;

[0133] The text features and multimedia features are input into the neural network model for natural language processing to obtain the processing results corresponding to the task processing data.

[0134] In practical applications, there are many ways to extract features from task processing data and obtain text features corresponding to the task processing data. The specific method is selected according to the actual situation, and the embodiments of this specification do not impose any restrictions on this. In one possible implementation of this specification, the text features corresponding to the task processing data can be generated using one-hot encoding. In another possible implementation of this specification, the task processing data can be input into an encoder to obtain the text features corresponding to the task processing data.

[0135] Furthermore, after obtaining the text features corresponding to the task processing data, the text features and multimedia features can be concatenated and input into a neural network model for natural language processing to obtain the processing results corresponding to the task processing data. Alternatively, the feature dimensions of the multimedia features can be transformed, and the text features and the converted multimedia features can be concatenated and input into a neural network model for natural language processing to obtain the processing results corresponding to the task processing data.

[0136] By applying the solution of the embodiments of this specification, feature extraction is performed on the task processing data to obtain text features corresponding to the task processing data; the text features and multimedia features are input into the neural network model used for natural language processing to obtain processing results corresponding to the task processing data, thereby reducing the data processing amount of the neural network model used for natural language processing and improving task processing efficiency.

[0137] In one possible implementation of this specification, before inputting text features and multimedia features into the neural network model for natural language processing and obtaining processing results corresponding to task processing data, the following steps may also be included:

[0138] According to the text features, the feature dimensions of the multimedia features are converted to obtain the converted multimedia features;

[0139] Input text features and multimedia features into the neural network model for natural language processing to obtain processing results corresponding to the task processing data, including:

[0140] The text features and the converted multimedia features are input into the neural network model for natural language processing to obtain the processing results corresponding to the task processing data.

[0141] It should be noted that the feature dimensions of the multimedia features output by the multimedia data recognition model may differ from the feature dimensions of the text features corresponding to the task processing data. When the multimedia features and text features are spliced ​​together, the different feature dimensions may cause the neural network model used for natural language processing to fail. Therefore, in the embodiments of this specification, after obtaining the multimedia features, the feature dimensions of the multimedia features can be converted based on the text features to obtain the converted multimedia features. For example, if the feature dimension of the text feature is 512, the feature dimension of the converted multimedia features will also be 512.

[0142] In practical applications, a linear layer can be set up in the multimedia data recognition model to transform the feature dimensions of multimedia features to align them with the dimensions of the neural network model used for natural language processing, that is, the dimensions of text features. During training, the model parameters of the linear layer and the multimedia data recognition model are jointly adjusted based on the first loss value.

[0143] Using the solutions of the embodiments of this specification, the feature dimensions of multimedia features are converted based on text features to obtain converted multimedia features. The text features and the converted multimedia features are then input into a neural network model for natural language processing to obtain processing results corresponding to the task processing data. By aligning the multimedia features with the text semantic features, it is ensured that the multimedia features and text features can be successfully spliced ​​and input into the neural network model for natural language processing.

[0144] In an optional embodiment of the present specification, the above-mentioned inputting multimedia features and task processing data into the neural network model for natural language processing and obtaining the processing results corresponding to the task processing data may further include the following steps:

[0145] Obtain multiple sets of sample multimedia text pairs;

[0146] Extracting a second sample multimedia text pair from the plurality of groups of sample multimedia text pairs, wherein the second sample multimedia text pair is any one of the plurality of groups of sample multimedia text pairs;

[0147] Inputting the second sample multimedia data into the multimedia data recognition model to obtain second sample multimedia features corresponding to the second sample multimedia data;

[0148] constructing second sample prompt data according to the second sample multimedia feature and the second sample multimedia text pair;

[0149] inputting the second sample prompt data into a neural network model for natural language processing to obtain a second prediction result;

[0150] Calculating a second loss value based on the second sample text data and the second prediction result;

[0151] Adjust the model parameters of the multimedia data recognition model and the neural network model for natural language processing according to the second loss value, and return to the step of extracting a second sample multimedia text pair from the multiple groups of sample multimedia text pairs until the second preset stopping condition is reached, thereby obtaining a trained neural network model for natural language processing.

[0152] Specifically, the second sample multimedia text pair includes second sample multimedia data and second sample text data. The second sample prompt data includes second sample multimedia features, second sample text features corresponding to the second sample text data, and second sample processing features of the second sample processing data. The second sample processing data can be understood as the sample problem corresponding to the second sample multimedia text pair, and the number of second sample processing data can be one or more. When the second sample multimedia data is input into the multimedia data recognition model, the multimedia data recognition model is a multimedia data recognition model trained based on multiple groups of sample multimedia text pairs and sample tasks corresponding to the sample multimedia text pairs. The training method of the neural network model for natural language processing is supervised training, and the sample text data in the multiple groups of sample multimedia text pairs are the real labels of the natural language processing process, that is, the training targets of the neural network model for natural language processing, which are used to guide the training process of the neural network model for natural language processing.

[0153] It is worth noting that in order to make the sample text data in the sample multimedia text pairs more matched with the sample processing data, in a possible implementation method of this specification, after obtaining multiple groups of sample multimedia text pairs, the sample text data in the sample multimedia text pairs can be adjusted according to the sample processing tasks to obtain updated multiple groups of sample multimedia text pairs, and further extract the second sample multimedia text pair from the updated multiple groups of sample multimedia text pairs.

[0154] In one possible implementation of this specification, the second preset stop condition includes the second loss value being less than or equal to a second preset threshold. After calculating the second loss value based on the second sample text data and the second prediction result, the second loss value is compared with the second preset threshold.

[0155] Specifically, if the second loss value is greater than the second preset threshold, it means that the difference between the second sample text data and the second prediction result is large, and the neural network model for natural language processing has poor prediction ability for the second prediction result. At this time, the model parameters of the multimedia data recognition model and the neural network model for natural language processing can be adjusted, and the step of extracting the second sample multimedia text pair from multiple groups of sample multimedia text pairs can be returned to execute, and the multimedia data recognition model and the neural network model for natural language processing can be continued to be trained until the second loss value is less than or equal to the second preset threshold, indicating that the difference between the second sample text data and the second prediction result is small, and the second preset stop condition is met, and a trained neural network model for natural language processing is obtained.

[0156] In another possible implementation of the present specification, in addition to comparing the size relationship between the second loss value and the second preset threshold, it is also possible to determine whether the current neural network model for natural language processing is trained in combination with a preset number of iterations.

[0157] Specifically, if the second loss value is greater than the second preset threshold, the model parameters of the multimedia data recognition model and the neural network model for natural language processing are adjusted, and the step of extracting the second sample multimedia text pair from multiple groups of sample multimedia text pairs is returned to execute, and the multimedia data recognition model and the neural network model for natural language processing are continued to be trained until the second preset number of iterations is reached, and the iteration is stopped to obtain a trained neural network model for natural language processing, wherein the second preset threshold and the second preset number of iterations are selected according to actual conditions, and the embodiments of this specification do not impose any restrictions on this.

[0158] It should be noted that, since the model parameters of the trained multimedia data recognition model are also updated during the training process of the neural network model for natural language processing, when the target multimedia data is input into the multimedia data recognition model and the multimedia features corresponding to the target multimedia data are obtained, the target multimedia data can be input into the updated multimedia data recognition model, that is, the training of the multimedia data recognition model can be divided into two stages, the first stage is the separate training of the multimedia data recognition model, and the second stage is the joint training with the neural network model for natural language processing.

[0159] In practical applications, there are many functions for calculating the second loss value, such as predicting the current text word based on the previous K text words and using the negative logarithmic maximum likelihood loss function to calculate the second loss value. That is, the second loss value can be calculated using the above formula (1).

[0160] Applying the solution of the embodiments of this specification, a second loss value is calculated based on the second sample text data and the second prediction result, and the second loss value is compared with a second preset stopping condition. If the second preset stopping condition is not met, the multimedia data recognition model and the neural network model for natural language processing are continuously trained until the second preset stopping condition is met, thereby completing the training and obtaining the neural network model for natural language processing. By continuously adjusting the model parameters of the neural network model for natural language processing, the resulting neural network model for natural language processing can be made more accurate.

[0161] In an optional embodiment of the present specification, the step of constructing the second sample prompt data based on the second sample multimedia feature and the second sample multimedia text pair may include the following steps:

[0162] Analyzing the second sample multimedia text pair to determine second sample processed data corresponding to the second sample multimedia text pair;

[0163] Performing feature extraction on the second sample text data to obtain second sample text features;

[0164] Second sample prompt data is constructed according to the second sample processed data, the second sample multimedia features, and the second sample text features.

[0165] It should be noted that after determining the second sample processing data corresponding to the second sample multimedia text pair, feature extraction can be performed on the second sample processing data to obtain second sample processing features. Second sample prompt data can then be constructed based on the second sample processing features, the second sample multimedia features, and the second sample text features. The implementation of "performing feature extraction on the second sample text data to obtain second sample text features" is similar to the implementation of "performing feature extraction on the task processing data to obtain text features corresponding to the task processing data" described above, and will not be further elaborated in this embodiment of the specification.

[0166] In actual applications, there are many ways to analyze the second sample multimedia text pair and determine the second sample processing data corresponding to the second sample multimedia text pair. The specific selection depends on the actual situation, and the embodiments of this specification do not impose any restrictions on this. In one possible implementation of this specification, the second sample processing data corresponding to the second sample multimedia text pair can be randomly generated according to the type of the second sample multimedia data. For example, if the second sample multimedia data is human voice data, the second sample processing data is what the speaker said. In another possible implementation of this specification, the second sample multimedia text pair can be analyzed according to the sample processing task to determine the second sample processing data corresponding to the second sample multimedia text pair.

[0167] Furthermore, in order to align the response of the neural network model used for natural language processing with the user's understanding, in an embodiment of this specification, a preset prompt template can be obtained, and the second sample processing features, the second sample multimedia features and the second sample text features can be filled into the preset prompt template to construct the second sample prompt data.

[0168] For example, the preset prompt template can use "\n\nHuman:" and "\n\nAssistant:" to distinguish questions and answers, and "Audio:![audio]" to represent the input of raw audio data. In practical applications, the second sample audio feature can be inserted into the text representation based on the position of "![audio]", making the second sample prompt data more flexible. After the second sample processing feature is inserted into "\n\nHuman:", and the second sample text feature is inserted into "\n\nAssistant:", the second sample prompt data is obtained.

[0169] It is worth noting that there are multiple template formats for the preset prompt template, and the selection is made according to the actual situation. The embodiments of this specification do not impose any limitation on this.

[0170] In one possible implementation of this specification, the user adds the target multimedia data and presses Enter. The robot responds with "Received." The user then sends the task processing data, and the robot determines the corresponding processing result. For example, if the target multimedia data is audio, the corresponding preset prompt template is "\n\nHuman: Audio 1:! [audio]\n\nAssistant: Received.\n\nHuman: Question 1\n\nAssistant: Answer 1."

[0171] In another possible implementation of this specification, the user adds target multimedia data and enters task processing data. The robot then directly determines the processing result corresponding to the task processing data based on the target multimedia data. For example, if the target multimedia data is audio, the corresponding preset prompt template for this method is "\n\nHuman:Audio 1:![audio]\nQuestion 1\n\nAssistant:Answer 1".

[0172] Furthermore, based on the user's question-and-answer habits, in addition to the above-mentioned single-audio single-round dialogue prompt template, the preset prompt template in the embodiment of this specification can also be divided into a single-audio multi-round dialogue prompt template "\n\nHuman:Audio 1:![audio]\n\nAssistant:Received.\n\nHuman:Question 1\n\nAssistant:Answer 1\n\nHuman:Question 2\n\nAssistant:Answer 2..." and a multi-audio multi-round dialogue prompt template "\n\nHuman:Audio 1:![audio1]\n\nAssistant:Received.\n\nHuman:Audio 2:![audio2]\n\nAssistant:Received.\n\nHuman:Question 1\n\nAssistant:Answer 1\n\nHuman:Question 2\n\nAssistant:Answer 2...", and the specific selection is made according to actual circumstances, and the embodiment of this specification does not impose any limitation on this.

[0173] By applying the solution of the embodiments of this specification, the second sample multimedia text pair is analyzed to determine the second sample processing data corresponding to the second sample multimedia text pair; feature extraction is performed on the second sample text data to obtain second sample text features; and second sample prompt data is constructed based on the second sample processing data, the second sample multimedia features, and the second sample text features, so that the neural network model used for natural language processing can be trained based on the second sample prompt data, thereby ensuring the accuracy of the training process.

[0174] In an optional embodiment of the present specification, in order to enable the neural network model for natural language processing to cover more multimedia data processing tasks, in the embodiment of the present specification, second sample processing data corresponding to the second sample multimedia text pair can be determined based on the sample processing task. That is, the above-mentioned analysis of the second sample multimedia text pair to determine the second sample processing data corresponding to the second sample multimedia text pair can include the following steps:

[0175] Obtaining a sample processing task, wherein the sample processing task includes at least one of a question-answering task, a strategy generation task, a poetry creation task, and a recommendation task;

[0176] According to the sample processing task, the second sample multimedia text pair is analyzed to determine the second sample processing data corresponding to the second sample multimedia text pair.

[0177] Specifically, the question-answering task can be understood as a factual question-answering task, which refers to factual question-answering based on the content of multimedia data. For example, based on the audio of human voice, questions and answers are asked about the speaker's speech emotions, speech content, etc.; based on the audio of music, questions and answers are asked about the style and instruments of the music audio; based on the audio of ambient sound, questions and answers are asked about the entity that makes the sound, etc. The strategy generation task can be understood as an event response task, which refers to giving corresponding measures or response plans in response to the relevant instructions input by the user based on the scene or event recorded by the multimedia data. For example, in response to the sound of a water leak, the user asks about precautions and gives response plans such as turning off the faucet. The poetry creation task refers to creating poems related to the content of the multimedia data in response to the user's input instructions for writing poems based on the content of the multimedia data. The recommendation task refers to making relevant recommendations based on the facts of the multimedia data. For example, based on the songs input by the user, relevant style songs are recommended; based on the human voice, the speaker's region is judged and relevant recommendations based on the region are made, etc.

[0178] In practical applications, there are multiple ways to obtain sample processing tasks, and the specific method to be selected depends on the actual situation. This specification does not impose any restrictions on this. In one possible implementation of this specification, the sample processing tasks can be configured by relevant personnel. In another possible implementation of this specification, at least one of the following tasks can be randomly selected as the sample processing task: a question-answering task, a strategy generation task, a poetry creation task, or a recommendation task.

[0179] For example, assuming that the sample processing task is a question-answering task, the second sample processing data is "What is the speaker's speaking emotion"; assuming that the sample processing task is a strategy generation task, the second sample processing data is "What is the processing strategy"; assuming that the sample processing task is a poetry creation task, the second sample processing data is "Please write a poem"; assuming that the sample processing task is a recommendation task, the second sample processing data is "What are the songs of related styles?"

[0180] Applying the solutions of the embodiments of this specification, a sample processing task is obtained, where the sample processing task includes at least one of a question-answering task, a strategy generation task, a poetry composition task, and a recommendation task. Based on the sample processing task, a second sample multimedia text pair is analyzed to determine second sample processing data corresponding to the second sample multimedia text pair. By considering multiple complex multimedia understanding tasks during the training process, the task processing process is more consistent with real-world user usage, enabling more complex multimedia understanding and corresponding dialogues.

[0181] Refer to Figure 5, which shows a training flowchart of a neural network model for natural language processing in a task processing method provided in an embodiment of this specification. As shown in Figure 5, multiple groups of sample audio-text pairs are obtained, wherein the sample audio-text pairs include sample audio data and sample text data. Extract any one sample audio-text pair from multiple groups of sample audio-text pairs, input sample audio data from the sample audio-text pair into an audio recognition model to obtain sample audio features corresponding to the sample audio data; perform feature extraction on the sample text data "A dog barks while a man speaks" from the sample audio-text pair to obtain sample text features; input the sample audio features into a linear layer, align feature dimensions of the sample audio features with feature dimensions of the sample text features, and obtain dimension-converted sample audio features; analyze the sample audio-text pair according to a sample processing task to determine sample processing data; perform feature extraction on the sample processing data to obtain sample processing features; construct sample prompt data based on the dimension-converted sample audio features, sample processing features, and sample text features; input the sample prompt data into a neural network model for natural language processing to obtain a prediction result; calculate a loss value based on the sample text data "A dog barks while a man speaks" and the prediction result; adjust model parameters of the audio recognition model and the neural network model for natural language processing based on the loss value, and return to the step of extracting any one sample audio-text pair from the multiple groups of sample audio-text pairs until a second preset stop condition is reached, thereby obtaining a trained neural network model for natural language processing.

[0182] Referring to FIG6 , FIG6 shows a flow chart of an automatic question-answering method provided by an embodiment of this specification, which specifically includes the following steps:

[0183] Step 602: Receive target multimedia data and question data corresponding to the target multimedia data sent by the user through the client.

[0184] Step 604: Input the target multimedia data into a multimedia data recognition model to obtain multimedia features corresponding to the target multimedia data, wherein the multimedia data recognition model is trained based on multiple groups of sample multimedia text pairs and sample tasks corresponding to the sample multimedia text pairs, and the sample multimedia text pairs include sample multimedia data and sample text data corresponding to the sample multimedia data.

[0185] Step 606: Input the multimedia features and question data into a neural network model for natural language processing to obtain a response result corresponding to the question data.

[0186] It should be noted that the question data is also the task processing data, and the answer result corresponding to the question data is also the processing result corresponding to the task processing data. The implementation of steps 602 to 606 is the same as the implementation of steps 302 to 306 above, and will not be repeated in this embodiment.

[0187] By applying the solution of the embodiments of this specification, since the multimedia data recognition model is constructed with the powerful language understanding and generation capabilities of the neural network model used for natural language processing, a multimedia data recognition model for understanding all types of multimedia including human voice, environmental sound, music, etc. is constructed. By combining the multimedia data recognition model with the neural network model used for natural language processing, complex multimedia tasks can be directly processed and the response results corresponding to the question data can be accurately obtained.

[0188] In an optional embodiment of the present specification, after inputting the multimedia features and question data into the neural network model for natural language processing and obtaining a response result corresponding to the question data, the following steps may also be included:

[0189] Send the answer result corresponding to the question data to the client, so that the client can display the answer result;

[0190] Receive the post-processing request sent by the user based on the reply result, process the post-processing request, and obtain the reply result corresponding to the post-processing request.

[0191] Specifically, a post-processing request refers to a question-and-answer request sent by a user based on a response result after receiving the response result. The data requested for processing in the post-processing request can be new question data sent for the target multimedia data, or it can be question data for new target multimedia data. The specific selection depends on the actual situation and is not limited in this embodiment of the specification.

[0192] In actual applications, there are many ways for the client to display the data relationship identification results to the user, and the specific selection is based on the actual situation. The embodiments of this specification do not impose any restrictions on this. In one possible implementation of this specification, the client can only display the reply result to the user. In another possible implementation of this specification, the client can display the question data and the reply result to the user at the same time. After the client displays the reply result to the user, the user can process the data according to the reply result, or send a post-processing request based on the reply result displayed by the client.

[0193] It should be noted that the implementation method of processing the post-processing request and obtaining the reply result corresponding to the post-processing request is the same as the implementation method of the automatic question-answering method, and will not be repeated in the embodiments of this specification.

[0194] By applying the solution of the embodiment of this specification, the reply result corresponding to the question data is sent to the client so that the client can display the reply result; the post-processing request sent by the user based on the reply result is received, and the post-processing request is processed to obtain the reply result corresponding to the post-processing request, thereby increasing the interaction with the user and improving user satisfaction.

[0195] Referring to FIG. 7 , FIG. 7 shows a flowchart of a multimedia data recognition model training method provided by one embodiment of this specification. The multimedia data recognition model training method is applied to a cloud-side device and specifically includes the following steps:

[0196] Step 702: Acquire multiple groups of sample multimedia text pairs, wherein the sample multimedia text pairs carry sample tasks.

[0197] Step 704: extracting a first sample multimedia text pair from the plurality of sample multimedia text pairs according to the target sample task, wherein the first sample multimedia text pair is any one of the plurality of sample multimedia text pairs corresponding to the target sample task.

[0198] Step 706: Input the first sample multimedia data into the multimedia data recognition model to obtain first sample multimedia features corresponding to the first sample multimedia data.

[0199] Step 708: Input the first sample multimedia features and the first sample text data into a neural network model for natural language processing to obtain a first prediction result.

[0200] Step 710: Train a multimedia data recognition model based on the first sample text data and the first prediction result, and obtain model parameters of the trained multimedia data recognition model.

[0201] Step 712: Send the model parameters of the trained multimedia data recognition model to the end-side device.

[0202] It should be noted that the implementation of steps 702 to 712 is the same as the implementation of steps 302 to 306 described above, and will not be described in detail in this embodiment of the specification.

[0203] In actual applications, after the cloud-side device sends the model parameters of the trained multimedia data recognition model to the terminal-side device, the terminal-side device can locally build a multimedia data recognition model based on the model parameters of the multimedia data recognition model, thereby using the multimedia data recognition model locally to perform multimedia task processing.

[0204] Using the solution of the embodiments of this specification, a first loss value is calculated based on the first sample text data and the first prediction result, and the first loss value is compared with a first preset stopping condition. If the first preset stopping condition is not met, the multimedia data recognition model is trained continuously until the first preset stopping condition is met, thereby completing the training and obtaining the multimedia data recognition model. By continuously adjusting the model parameters of the multimedia data recognition model, the resulting multimedia data recognition model can be made more accurate.

[0205] 8 , which shows a flowchart of a task processing method according to an embodiment of the present disclosure, specifically including:

[0206] Construct sample audio-text pairs: collect various types of original audio data and related original text data from the Internet and various public data sources, process the original audio data and original text data according to the audio type of the original audio data, and construct sample audio-text pairs. Specifically, determine the matching index of audio and text through the audio-text matching model, filter the ambient sound data according to the matching index; add noise and reverberation to the human voice data; combine the meta-information and content information in the human voice data into natural language; and also combine the original text data corresponding to the music data into natural language. Furthermore, data enhancement can be performed on the original audio data and original text data, such as mixing the music data and the human voice data, and splicing the corresponding original text data of the two to construct audio data with both music and human voice and its corresponding text data;

[0207] Pre-training - alignment to text modality: A pre-trained audio recognition model is selected as the basic skeleton and parameter starting point for the audio recognition model in this solution. The target audio is input into the audio recognition model for feature encoding, and the encoded audio features are transformed through a linear layer to align to the dimension of the text features, and then spliced ​​front and back with the text features, and input together into the neural network model for natural language processing. Specifically, the training process adopts a course learning method to train the audio recognition model in two stages. The two stages correspond to different target sample tasks. The first stage corresponds to the sample audio understanding task, and the second stage corresponds to the sample audio understanding task and the sample audio recognition task. The training process fixes the model parameters of the neural network model used for natural language processing, and uses the audio recognition model as an encoder to align the text modality;

[0208] Constructing sample prompt data: Sample prompt data can be called sample audio instruction data and is supervised fine-tuning (SFT) data, such as "\n\nHuman:Audio 1:![audio]\n\nAssistant:Received.\n\nHuman:Question 1\n\nAssistant:Answer 1...". During the construction of sample prompt data, sample audio-text pairs can be analyzed based on sample processing tasks to determine corresponding sample processing data. The sample processing tasks include at least one of question-answering tasks, strategy generation tasks, poetry creation tasks, and recommendation tasks.

[0209] Supervised fine-tuning training: Using a pre-trained audio recognition model as a starting point, and using sample prompt data, fine-tune the pre-trained audio recognition model and the neural network model used for natural language processing together to align with user understanding.

[0210] By applying the solution of the embodiments of this specification, all types of audio data are taken into account when constructing sample audio-text pairs, so the trained audio recognition model can support different types of audio input. By leveraging the powerful language understanding, generation, interaction and reasoning capabilities of the neural network model used for natural language processing, and adopting pre-training and supervised fine-tuning training methods, the trained audio recognition model is aligned with the neural network model used for natural language processing to achieve various audio understanding-related functions, and does not rely on calling external software interfaces. Since the audio recognition model can support more complex mixed inputs of various types of audio, and a variety of complex sample processing tasks are taken into account during training, it can be more in line with the actual user usage and complete more complex audio understanding and corresponding dialogue tasks.

[0211] See Figure 9, which shows a schematic diagram of an automatic question-and-answer interface provided by one embodiment of this specification. The automatic question-and-answer interface is divided into a question-handling request input interface and a response result display interface. The question-handling request input interface includes a request information input box, an "OK" control, and a "Cancel" control. The response result display interface includes a response result display box.

[0212] The user enters the target audio and question data for the target audio through the request information input box displayed on the client and clicks the "OK" control. The server receives the target audio and question data sent by the client, inputs the target audio into the audio recognition model, and obtains the audio features corresponding to the target audio. The audio recognition model is trained based on multiple sets of sample audio-text pairs and sample tasks corresponding to the sample audio-text pairs. The sample audio-text pairs include sample audio data and sample text data corresponding to the sample audio data. The audio features and question data are input into the neural network model used for natural language processing, and the response result corresponding to the question data is obtained and sent to the client. The client displays the response result in the response result display box.

[0213] In actual applications, users can operate controls by clicking, double-clicking, touching, hovering the mouse, sliding, long pressing, voice control, or shaking, etc. The specific selection is based on the actual situation, and the embodiments of this specification do not impose any restrictions on this.

[0214] Corresponding to the above-mentioned task processing method embodiment, this specification also provides a task processing device embodiment. FIG10 shows a schematic diagram of the structure of a task processing device provided in one embodiment of this specification. As shown in FIG10 , the device includes:

[0215] A first acquisition module 1002 is configured to acquire target multimedia data and task processing data for the target multimedia data;

[0216] A first input module 1004 is configured to input target multimedia data into a multimedia data recognition model to obtain multimedia features corresponding to the target multimedia data, wherein the multimedia data recognition model is trained based on multiple sets of sample multimedia text pairs and sample tasks corresponding to the sample multimedia text pairs, wherein the sample multimedia text pairs include sample multimedia data and sample text data corresponding to the sample multimedia data;

[0217] The second input module 1006 is configured to input multimedia features and task processing data into the neural network model for natural language processing to obtain processing results corresponding to the task processing data.

[0218] Optionally, the second input module 1006 is further configured to perform feature extraction on the task processing data to obtain text features corresponding to the task processing data; input the text features and multimedia features into a neural network model for natural language processing to obtain processing results corresponding to the task processing data.

[0219] Optionally, the device also includes: a conversion module, configured to convert the feature dimensions of the multimedia features according to the text features to obtain the converted multimedia features; a second input module 1006, further configured to input the text features and the converted multimedia features into a neural network model for natural language processing to obtain the processing results corresponding to the task processing data.

[0220] Optionally, the device also includes: a second training module, configured to obtain multiple groups of sample multimedia text pairs, wherein the sample multimedia text pairs carry sample tasks; extract a first sample multimedia text pair from the multiple groups of sample multimedia text pairs according to the target sample task, wherein the first sample multimedia text pair is any one of the multiple groups of sample multimedia text pairs corresponding to the target sample task; input the first sample multimedia data into the multimedia data recognition model to obtain a first sample multimedia feature corresponding to the first sample multimedia data; input the first sample multimedia feature and the first sample text data into a neural network model for natural language processing to obtain a first prediction result; calculate a first loss value based on the first sample text data and the first prediction result; adjust the model parameters of the multimedia data recognition model based on the first loss value, and return to execute the step of extracting the first sample multimedia text pair from the multiple groups of sample multimedia text pairs according to the target sample task, until the first preset stop condition is reached, and a trained multimedia data recognition model is obtained.

[0221] Optionally, the sample tasks include a sample multimedia understanding task and a sample multimedia recognition task; the second training module is further configured to determine that the target sample task is a sample multimedia understanding task when the first loss value does not meet the task switching condition; and to determine that the target sample task is a sample multimedia understanding task and a sample multimedia recognition task when the first loss value meets the task switching condition.

[0222] Optionally, the device also includes: a third training module, configured to obtain multiple groups of sample multimedia text pairs; extract a second sample multimedia text pair from the multiple groups of sample multimedia text pairs, wherein the second sample multimedia text pair is any one of the multiple groups of sample multimedia text pairs; input the second sample multimedia data into the multimedia data recognition model to obtain a second sample multimedia feature corresponding to the second sample multimedia data; construct second sample prompt data based on the second sample multimedia feature and the second sample multimedia text pair; input the second sample prompt data into the neural network model for natural language processing to obtain a second prediction result; calculate a second loss value based on the second sample text data and the second prediction result; adjust the model parameters of the multimedia data recognition model and the neural network model for natural language processing based on the second loss value, and return to execute the step of extracting the second sample multimedia text pair from the multiple groups of sample multimedia text pairs until the second preset stop condition is reached to obtain a trained neural network model for natural language processing.

[0223] Optionally, the third training module is further configured to analyze the second sample multimedia text pair to determine the second sample processed data corresponding to the second sample multimedia text pair; perform feature extraction on the second sample text data to obtain second sample text features; and construct second sample prompt data based on the second sample processed data, the second sample multimedia features and the second sample text features.

[0224] Optionally, the third training module is further configured to obtain sample processing tasks, wherein the sample processing tasks include at least one of question-answering tasks, strategy generation tasks, poetry creation tasks, and recommendation tasks; according to the sample processing tasks, the second sample multimedia text pair is analyzed to determine the second sample processing data corresponding to the second sample multimedia text pair.

[0225] Optionally, the device also includes: a construction module, configured to obtain multiple original multimedia data and original text data corresponding to the original multimedia data, wherein the original multimedia data includes at least one of ambient sound data, human voice data, and music data; perform data enhancement on the original multimedia data and the original text data to obtain enhanced multimedia data and enhanced text data; and construct multiple groups of sample multimedia text pairs based on the enhanced multimedia data and the enhanced text data.

[0226] By applying the solution of the embodiments of this specification, a multimedia data recognition model for understanding all types of multimedia, including human voice, environmental sound, music, etc., is constructed by leveraging the powerful language understanding and generation capabilities of the neural network model for natural language processing. This combines the multimedia data recognition model with the neural network model for natural language processing to achieve the processing of complex multimedia tasks and improve the universality of multimedia task processing.

[0227] The above is a schematic scheme of a task processing device of this embodiment. It should be noted that the technical scheme of the task processing device and the technical scheme of the task processing method described above are of the same concept. For details not described in detail in the technical scheme of the task processing device, please refer to the description of the technical scheme of the task processing method described above.

[0228] Corresponding to the above-mentioned automatic question-answering method embodiment, this specification also provides an automatic question-answering device embodiment. FIG11 shows a schematic structural diagram of an automatic question-answering device provided in one embodiment of this specification. As shown in FIG11 , the device includes:

[0229] The receiving module 1102 is configured to receive target multimedia data and question data corresponding to the target multimedia data sent by the user through the client;

[0230] The third input module 1104 is configured to input the target multimedia data into a multimedia data recognition model to obtain multimedia features corresponding to the target multimedia data, wherein the multimedia data recognition model is trained based on multiple sets of sample multimedia text pairs and sample tasks corresponding to the sample multimedia text pairs, wherein the sample multimedia text pairs include the sample multimedia data and sample text data corresponding to the sample multimedia data;

[0231] The fourth input module 1106 is configured to input the multimedia features and question data into the neural network model for natural language processing to obtain a response result corresponding to the question data.

[0232] Optionally, the device also includes: a processing module, configured to send a reply result corresponding to the question data to the client, so that the client displays the reply result; receive a post-processing request sent by the user based on the reply result, and process the post-processing request to obtain a reply result corresponding to the post-processing request.

[0233] By applying the solution of the embodiments of this specification, since the multimedia data recognition model is constructed with the powerful language understanding and generation capabilities of the neural network model used for natural language processing, a multimedia data recognition model for understanding all types of multimedia including human voice, environmental sound, music, etc. is constructed. By combining the multimedia data recognition model with the neural network model used for natural language processing, complex multimedia tasks can be directly processed and the response results corresponding to the question data can be accurately obtained.

[0234] The above is a schematic diagram of an automatic question-answering device according to this embodiment. It should be noted that the technical solution of this automatic question-answering device and the technical solution of the automatic question-answering method described above are based on the same concept. For details not described in detail in the technical solution of the automatic question-answering device, please refer to the description of the technical solution of the automatic question-answering method described above.

[0235] Corresponding to the above-mentioned multimedia data recognition model training method embodiment, this specification also provides a multimedia data recognition model training device embodiment. Figure 12 shows a schematic diagram of the structure of a multimedia data recognition model training device provided by one embodiment of this specification. As shown in Figure 12, the device is applied to a cloud-side device and includes:

[0236] The second acquisition module 1202 is configured to acquire multiple groups of sample multimedia text pairs, wherein the sample multimedia text pairs carry sample tasks;

[0237] An extraction module 1204 is configured to extract a first sample multimedia text pair from the plurality of sample multimedia text pairs according to the target sample task, wherein the first sample multimedia text pair is any one of the plurality of sample multimedia text pairs corresponding to the target sample task;

[0238] The fifth input module 1206 is configured to input the first sample multimedia data into the multimedia data recognition model to obtain a first sample multimedia feature corresponding to the first sample multimedia data;

[0239] A sixth input module 1208 is configured to input the first sample multimedia feature and the first sample text data into a neural network model for natural language processing to obtain a first prediction result;

[0240] A first training module 1210 is configured to train a multimedia data recognition model based on the first sample text data and the first prediction result, and obtain model parameters of the trained multimedia data recognition model;

[0241] The sending module 1212 is configured to send the model parameters of the trained multimedia data recognition model to the terminal device.

[0242] Using the solution of the embodiments of this specification, a first loss value is calculated based on the first sample text data and the first prediction result, and the first loss value is compared with a first preset stopping condition. If the first preset stopping condition is not met, the multimedia data recognition model is trained continuously until the first preset stopping condition is met, thereby completing the training and obtaining the multimedia data recognition model. By continuously adjusting the model parameters of the multimedia data recognition model, the resulting multimedia data recognition model can be made more accurate.

[0243] The above is a schematic diagram of a multimedia data recognition model training device according to this embodiment. It should be noted that the technical solution of this multimedia data recognition model training device and the technical solution of the multimedia data recognition model training method described above are based on the same concept. For details not described in detail in the technical solution of the multimedia data recognition model training device, please refer to the description of the technical solution of the multimedia data recognition model training method described above.

[0244] Figure 13 shows a block diagram of a computing device according to one embodiment of the present disclosure. Components of the computing device 1300 include, but are not limited to, a memory 1310 and a processor 1320. The processor 1320 is connected to the memory 1310 via a bus 1330, and a database 1350 is used to store data.

[0245] The computing device 1300 also includes an access device 1340 that enables the computing device 1300 to communicate via one or more networks 1360. Examples of such networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 1340 may include one or more of any type of network interface (e.g., a network interface card (NIC)) whether wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a World Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0246] In one embodiment of the present specification, the aforementioned components of computing device 1300 and other components not shown in FIG13 may also be connected to each other, for example, via a bus. It should be understood that the computing device structure block diagram shown in FIG13 is for illustrative purposes only and does not limit the scope of this specification. Those skilled in the art may add or replace other components as needed.

[0247] Computing device 1300 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, personal digital assistant, laptop computer, notebook computer, netbook computer, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smartwatch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or personal computer (PC). Computing device 1300 may also be a mobile or stationary server.

[0248] Among them, the processor 1320 is used to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above-mentioned task processing method or automatic question-answering method or multimedia data recognition model training method.

[0249] The above is a schematic diagram of a computing device according to this embodiment. It should be noted that the technical solution of this computing device is based on the same concept as the technical solutions of the aforementioned task processing method, automatic question-answering method, and multimedia data recognition model training method. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solutions of the aforementioned task processing method, automatic question-answering method, or multimedia data recognition model training method.

[0250] An embodiment of the present specification also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the above-mentioned task processing method, automatic question-answering method, or multimedia data recognition model training method.

[0251] The above is a schematic diagram of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium is based on the same concept as the technical solutions of the aforementioned task processing method, automatic question-answering method, and multimedia data recognition model training method. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solutions of the aforementioned task processing method, automatic question-answering method, or multimedia data recognition model training method.

[0252] An embodiment of the present specification further provides a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned task processing method or automatic question-answering method or multimedia data recognition model training method.

[0253] The above is an illustrative solution of a computer program according to this embodiment. It should be noted that the technical solution of this computer program is based on the same concept as the technical solutions of the aforementioned task processing method, automatic question-answering method, and multimedia data recognition model training method. For details not described in detail in the technical solution of the computer program, please refer to the description of the technical solutions of the aforementioned task processing method, automatic question-answering method, or multimedia data recognition model training method.

[0254] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0255] The computer instructions include computer program code, which may be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium may include any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunications signal, and a software distribution medium.

[0256] It should be noted that for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0257] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0258] The preferred embodiments disclosed above are intended only to help illustrate this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific embodiments described. Obviously, many modifications and variations can be made based on the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that those skilled in the art can better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A task processing method, comprising: Acquire target multimedia data and task processing data for the target multimedia data; Inputting the target multimedia data into a multimedia data recognition model to obtain multimedia features corresponding to the target multimedia data, wherein the multimedia data recognition model is trained based on multiple groups of sample multimedia text pairs and sample tasks corresponding to the sample multimedia text pairs, and the sample multimedia text pairs include sample multimedia data and sample text data corresponding to the sample multimedia data; The multimedia features and the task processing data are input into a neural network model for natural language processing to obtain a processing result corresponding to the task processing data.

2. According to the method of claim 1, the step of inputting the multimedia features and the task processing data into a neural network model for natural language processing to obtain a processing result corresponding to the task processing data comprises: Performing feature extraction on the task processing data to obtain text features corresponding to the task processing data; The text features and the multimedia features are input into a neural network model for natural language processing to obtain a processing result corresponding to the task processing data.

3. The method according to claim 2, before inputting the text features and the multimedia features into a neural network model for natural language processing and obtaining the processing results corresponding to the task processing data, further comprises: According to the text feature, the feature dimension of the multimedia feature is converted to obtain the converted multimedia feature; The step of inputting the text features and the multimedia features into a neural network model for natural language processing to obtain a processing result corresponding to the task processing data includes: The text features and the converted multimedia features are input into a neural network model for natural language processing to obtain a processing result corresponding to the task processing data.

4. The method according to claim 1, before inputting the target multimedia data into a multimedia data recognition model to obtain multimedia features corresponding to the target multimedia data, further comprising: Acquire multiple groups of sample multimedia text pairs, wherein the sample multimedia text pairs carry sample tasks; According to the target sample task, extracting a first sample multimedia text pair from the multiple groups of sample multimedia text pairs, wherein the first sample multimedia text pair is any one of the multiple groups of sample multimedia text pairs corresponding to the target sample task; Inputting the first sample multimedia data into a multimedia data recognition model to obtain a first sample multimedia feature corresponding to the first sample multimedia data; Inputting the first sample multimedia feature and the first sample text data into a neural network model for natural language processing to obtain a first prediction result; Calculating a first loss value according to the first sample text data and the first prediction result; adjusting the model parameters of the multimedia data recognition model according to the first loss value, and returning to the step of performing the target sample task and extracting the first sample multimedia text pair from the multiple groups of sample multimedia text pairs. The steps are repeated until a first preset stop condition is reached to obtain a trained multimedia data recognition model.

5. According to the method of claim 4, the sample tasks include a sample multimedia understanding task and a sample multimedia recognition task; Before extracting the first sample multimedia text pair from the plurality of groups of sample multimedia text pairs according to the target sample task, the method further comprises: When the first loss value does not satisfy the task switching condition, determining that the target sample task is the sample multimedia understanding task; When the first loss value satisfies the task switching condition, the target sample task is determined to be the sample multimedia understanding task and the sample multimedia recognition task.

6. The method according to claim 1, before inputting the multimedia features and the task processing data into a neural network model for natural language processing and obtaining a processing result corresponding to the task processing data, further comprises: Obtain multiple groups of sample multimedia text pairs; Extracting a second sample multimedia text pair from the plurality of groups of sample multimedia text pairs, wherein the second sample multimedia text pair is any one of the plurality of groups of sample multimedia text pairs; Inputting the second sample multimedia data into the multimedia data recognition model to obtain second sample multimedia features corresponding to the second sample multimedia data; constructing second sample prompt data according to the second sample multimedia feature and the second sample multimedia text pair; Inputting the second sample prompt data into a neural network model for natural language processing to obtain a second prediction result; Calculating a second loss value according to the second sample text data and the second prediction result; The model parameters of the multimedia data recognition model and the neural network model for natural language processing are adjusted according to the second loss value, and the step of extracting a second sample multimedia text pair from the multiple groups of sample multimedia text pairs is returned to be executed until a second preset stopping condition is reached, thereby obtaining a trained neural network model for natural language processing.

7. The method according to claim 6, wherein constructing the second sample prompt data according to the second sample multimedia feature and the second sample multimedia text pair comprises: Analyzing the second sample multimedia text pair to determine second sample processed data corresponding to the second sample multimedia text pair; Performing feature extraction on the second sample text data to obtain second sample text features; Second sample prompt data is constructed according to the second sample processing data, the second sample multimedia features and the second sample text features.

8. The method according to claim 7, wherein analyzing the second sample multimedia text pair to determine the second sample processed data corresponding to the second sample multimedia text pair comprises: Acquire a sample processing task, wherein the sample processing task includes at least one of a question-answering task, a strategy generation task, a poetry creation task, and a recommendation task; According to the sample processing task, the second sample multimedia text pair is analyzed to determine the second sample processing data corresponding to the second sample multimedia text pair.

9. According to the method of claim 4 or 6, the step of obtaining multiple groups of sample multimedia text pairs comprises: Acquire a plurality of original multimedia data and original text data corresponding to the original multimedia data, wherein the original multimedia data includes at least one of ambient sound data, human voice data, and music data; Performing data enhancement on the original multimedia data and the original text data to obtain enhanced multimedia data and enhanced text data; A plurality of groups of sample multimedia-text pairs are constructed based on the enhanced multimedia data and the enhanced text data.

10. An automatic question-answering method, comprising: Receiving target multimedia data and question data for the target multimedia data sent by a user through a client; Inputting the target multimedia data into a multimedia data recognition model to obtain multimedia features corresponding to the target multimedia data, wherein the multimedia data recognition model is trained based on multiple groups of sample multimedia text pairs and sample tasks corresponding to the sample multimedia text pairs, and the sample multimedia text pairs include sample multimedia data and sample text data corresponding to the sample multimedia data; The multimedia features and the question data are input into a neural network model for natural language processing to obtain a response result corresponding to the question data.

11. The method according to claim 10, after inputting the multimedia features and the question data into a neural network model for natural language processing and obtaining a response result corresponding to the question data, further comprising: Sending a reply result corresponding to the question data to the client, so that the client displays the reply result; Receive a post-processing request sent by the user based on the reply result, process the post-processing request, and obtain a reply result corresponding to the post-processing request.

12. A multimedia data recognition model training method, applied to a cloud-side device, comprising: Acquire multiple groups of sample multimedia text pairs, wherein the sample multimedia text pairs carry sample tasks; According to the target sample task, extracting a first sample multimedia text pair from the multiple groups of sample multimedia text pairs, wherein the first sample multimedia text pair is any one of the multiple groups of sample multimedia text pairs corresponding to the target sample task; Inputting the first sample multimedia data into a multimedia data recognition model to obtain a first sample multimedia feature corresponding to the first sample multimedia data; Inputting the first sample multimedia feature and the first sample text data into a neural network model for natural language processing to obtain a first prediction result; According to the first sample text data and the first prediction result, the multimedia data recognition model is trained to obtain model parameters of the trained multimedia data recognition model; The model parameters of the trained multimedia data recognition model are sent to the terminal device.

13. A computing device comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the method described in any one of claims 1 to 9 or any one of claims 10 to 11 or claim 12 are implemented.

14. A computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the method described in any one of claims 1 to 9 or any one of claims 10 to 11 or claim 12.

Images