Task execution and model training method and device, equipment and medium

Abstract

The invention provides a task execution and model training method and device, equipment and a medium, and relates to the field of artificial intelligence, and the method comprises the steps: obtaining input data and a language processing task associated with the input data; calling a multi-modal model to execute a language processing task on the input data to obtain a task execution result; wherein the multi-modal model is obtained by comparing and pre-training a multi-field and multi-language text audio pair; and displaying a task execution result. By calling the multi-modal model subjected to comparison and pre-training to perform task processing, the semantic relationship between the text and the audio can be more accurately understood, so that a task execution result with higher quality is provided. The multi-modal, multi-field and multi-language text audio pair is adopted to perform comparison pre-training on the multi-modal model, so that the model can learn universal cross-modal representation, the adaptability of the model in different fields and languages is improved, and the applicability of the model in diversified application scenes is enhanced.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular, to a task execution and model training method, apparatus, device, and medium. Background Art

[0002] With the rapid development of artificial intelligence technology, natural language processing (NLP) and audio processing have gradually become research hotspots in the field of multimodal machine learning. In practical applications, text and audio, as two important information carriers, often need to work together to complete cross-modal tasks. However, the single-modal models in related technologies are difficult to effectively capture the complex semantic relationships between text and audio, resulting in limited task performance. Summary of the Invention

[0003] This application aims to solve at least one of the technical problems in related technologies to some extent.

[0004] To this end, this application proposes a task execution and model training method, apparatus, device, and medium. By calling a multi-modal model pre-trained by contrast to perform task processing, it can more accurately understand the semantic relationship between text and audio, thereby providing a higher-quality task execution result. Among them, multi-modal, multi-domain, and multi-language text-audio pairs are used to perform contrast pre-training on the multi-modal model, enabling the model to learn general cross-modal representations, improving the adaptability of the model in different domains and languages, and enhancing its applicability in diverse application scenarios.

[0005] An embodiment of one aspect of this application proposes a task execution method, including:

[0006] Obtain input data and the language processing task associated with the input data;

[0007] Call a multi-modal model to perform the language processing task on the input data to obtain a task execution result; wherein, the multi-modal model is obtained by performing contrast pre-training using multi-domain and multi-language text-audio pairs;

[0008] Display the task execution result.

[0009] An embodiment of another aspect of this application proposes a training method for a multi-modal model, including:

[0010] Obtain a training set; wherein, the training set includes text-audio pairs in multiple domains and text-audio pairs in multiple languages, and the text-audio pairs include sample audio and sample text that are semantically related;

[0011] Encode the sample audio in N text-audio pairs using the audio encoding layer in the multi-modal model to obtain the sample audio features of the N text-audio pairs; where N is a set positive integer;

[0012] Encode the sample text in the N text-audio pairs respectively using the text encoding layer in the multi-modal model to obtain the sample text features of the N text-audio pairs;

[0013] Use the contrast projection layer in the multi-modal model to project the sample audio features and sample text features corresponding to the N text-audio pairs into the same semantic space to obtain the audio projection features and text projection features corresponding to the N text-audio pairs;

[0014] Perform contrastive pre-training on the multi-modal model based on the text projection features and audio projection features corresponding to the N text-audio pairs.

[0015] Another embodiment of this application proposes a task execution device, including:

[0016] An acquisition module, configured to acquire input data and a language processing task associated with the input data;

[0017] An execution module, configured to call a multi-modal model to execute the language processing task on the input data to obtain a task execution result; where the multi-modal model is obtained by performing contrastive pre-training using text-audio pairs in multiple domains and multiple languages;

[0018] A display module, configured to display the task execution result.

[0019] Another embodiment of this application proposes a training device for a multi-modal model, including:

[0020] An acquisition module, configured to acquire a training set; where the training set includes text-audio pairs in multiple domains and text-audio pairs in multiple languages, and the text-audio pairs include sample audio and sample text with related semantics;

[0021] A first encoding module, configured to encode the sample audio in N text-audio pairs using the audio encoding layer in the multi-modal model to obtain the sample audio features of the N text-audio pairs; where N is a set positive integer;

[0022] A second encoding module, configured to encode the sample text in the N text-audio pairs respectively using the text encoding layer in the multi-modal model to obtain the sample text features of the N text-audio pairs;

[0023] A projection module, configured to project the sample audio features and sample text features corresponding to the N text-audio pairs into the same semantic space by using the contrast projection layer in the multimodal model, so as to obtain the audio projection features and text projection features corresponding to the N text-audio pairs;

[0024] A pre-training module, configured to perform contrastive pre-training on the multimodal model based on the text projection features and audio projection features corresponding to the N text-audio pairs.

[0025] In another aspect, an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the method for task execution as described in the foregoing aspect is implemented, and / or the method for training the multimodal model as described in the foregoing other aspect is implemented.

[0026] In yet another aspect, an embodiment of the present application provides a non-transitory computer-readable storage medium, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method for task execution as described in the foregoing aspect is implemented, and / or when executed, the method for training the multimodal model as described in the foregoing other aspect is implemented.

[0027] In still another aspect, an embodiment of the present application provides a computer program product, on which a computer program is stored. When the program is executed by a processor, the method for task execution as described in the foregoing aspect is implemented, and / or when executed, the method for training the multimodal model as described in the foregoing other aspect is implemented.

[0028] The task execution and model training methods, devices, equipment, and media provided by the present application can more accurately understand the semantic relationship between text and audio by calling a multimodal model that has undergone contrastive pre-training, so as to provide higher-quality task execution results. Among them, multimodal, multi-domain, and multilingual text-audio pairs (including text and audio with highly related semantics) are used to perform contrastive pre-training (i.e., contrastive learning) on the multimodal model, enabling the model to learn general cross-modal representations. This not only improves the adaptability of the model in different domains and languages but also enhances its applicability in diverse application scenarios. Moreover, through contrastive pre-training, on the one hand, the model can perform well on unseen data or new tasks, especially in zero-shot or few-shot scenarios. This characteristic enables the model to better handle complex tasks. On the other hand, unlabeled multimodal data (text-audio pairs) can be fully utilized to pre-train the model, significantly reducing the dependence on labeled data and the cost of data collection and annotation.

[0029] Additional aspects and advantages of the present application will be given in part in the following description, become apparent in part from the following description, or be learned by practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The above-mentioned and / or additional aspects and advantages of the present application will become apparent and be readily understood from the following description of embodiments in conjunction with the accompanying drawings, where:

[0031] Figure 1 is a schematic flowchart of the first task execution method provided by an embodiment of the present application;

[0032] Figure 2 is a schematic flowchart of the second task execution method provided by an embodiment of the present application;

[0033] Figure 3 is a schematic flowchart of the third task execution method provided by an embodiment of the present application;

[0034] Figure 4 is a schematic flowchart of the fourth task execution method provided by an embodiment of the present application;

[0035] Figure 5 is a schematic flowchart of the fifth task execution method provided by an embodiment of the present application;

[0036] Figure 6 is a schematic flowchart of the sixth task execution method provided by an embodiment of the present application;

[0037] Figure 7 is a schematic flowchart of the seventh task execution method provided by an embodiment of the present application;

[0038] Figure 8 is a schematic flowchart of the eighth task execution method provided by an embodiment of the present application;

[0039] Figure 9 is a schematic flowchart of the first training method of the multimodal model provided by an embodiment of the present application;

[0040] Figure 10 is a schematic flowchart of the second training method of the multimodal model provided by an embodiment of the present application.

[0041] Figure 11 is a schematic diagram of a general contrastive language-audio pre-training framework provided by the present application;

[0042] Figure 12 is a schematic structural diagram of a task execution device provided by an embodiment of the present application;

[0043] Figure 13Schematic diagram of the structure of a training device for a multimodal model provided by an embodiment of the present application;

[0044] Figure 14 Schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0045] The embodiments of the present application will be described in detail below. Examples of the embodiments are shown in the drawings, where the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the drawings are exemplary and are intended to explain the present application and should not be construed as a limitation of the present application.

[0046] In recent years, the field of audio-text cross-modal learning has shown a rapid development trend. Driven by the contrastive learning framework (such as the Contrastive Language-Audio Pre-training (CLAP) model inspired by Contrastive Language-Image Pre-training (CLIP)), it is gradually moving from a single English voice retrieval task towards the construction stage of a general model for multiple languages and multiple tasks. Although the related technologies have shown remarkable results in sound event retrieval, they still face bottleneck problems that need to be broken through, such as weak speech understanding ability and limited multi-language support range. For example, the related technologies at least include the following disadvantages:

[0047] 1. Limitations of the contrastive language-audio pre-training method

[0048] In the related technologies, the mainstream contrastive language-audio pre-training methods mainly focus on the sound and music retrieval tasks in the English environment. Their model architectures and training data are optimized around English speech and music features, resulting in significant shortcomings in multi-language speech understanding ability. For example, in the face of non-English languages (such as Chinese, Spanish, etc.), the model cannot effectively parse the semantic information in the speech and is difficult to achieve cross-language audio content understanding and retrieval.

[0049] 2. Problem of domain fragmentation of existing audio encoders

[0050] In the design of audio encoders in related technologies, there is an obvious domain fragmentation phenomenon, specifically manifested as the lack of compatibility between voice / music encoders and speech encoders in terms of model structure, feature representation, and training objectives. For example, voice / music encoders usually extract features based on spectrograms or Mel-Frequency Cepstral Coefficients (MFCC for short), focusing on the time-frequency domain analysis of audio; while speech encoders rely on acoustic models to analyze the temporal dependence relationship of speech. This fragmentation makes it difficult for the same model to process voice, music, and speech data simultaneously, restricting the application scenarios of cross-domain audio tasks.

[0051] 3. Insufficient multilingual support capabilities

[0052] There are two major deficiencies in related technologies in terms of multilingual support: First, the language coverage is limited, and most models only support a small number of mainstream languages, making it difficult to meet the needs of global applications; Second, even within the supported languages, the model's speech understanding ability for non-English languages is still weak, manifested as high speech recognition error rates, inaccurate semantic parsing, etc. For example, in the task of multilingual speech command recognition, the recognition accuracy of the model for low-resource languages (such as Swahili and Urdu) is relatively low.

[0053] 4. Keyword recognition bottleneck in zero-shot classification tasks

[0054] In the zero-shot classification scenario, the model needs to directly recognize unseen audio categories based on pre-trained knowledge. However, related technologies perform poorly in keyword recognition accuracy, and the main problems include: insufficient model generalization ability, making it difficult to transfer the speech features learned in the pre-training stage to new categories; lack of robustness in feature representation, being sensitive to interference factors such as background noise and accent changes; and the cross-language semantic gap problem, resulting in the model being unable to effectively associate the semantic information of audio signals with target keywords. For example, in the cross-language audio event detection task, the keyword recognition accuracy of zero-shot classification is relatively low.

[0055] Therefore, in view of at least one of the above problems existing in related technologies, this application proposes a task execution and model training method, device, equipment, and medium.

[0056] The following describes the task execution and model training method, device, equipment, and medium of the embodiments of this application with reference to the accompanying drawings. Before specifically describing the embodiments of this application, for the convenience of understanding, first introduce the commonly used technical terms:

[0057] Contrastive Pretraining: Contrastive Pretraining is a self-supervised learning method that enables the model to learn the representation features of data by comparing positive samples (similar samples) and negative samples (dissimilar samples). This method is commonly used in the pre-training phase of unlabeled data to help the model extract high-quality feature representations and provide a better foundation for subsequent tasks (such as classification, retrieval, etc.).

[0058] Zero-shot Classification: Zero-shot Classification refers to the ability of a model to directly classify unseen classes without being trained on specific classes. This ability typically relies on the model's pre-trained knowledge (such as a language model's understanding of semantics) and prompts or descriptive information to complete the task.

[0059] Language Processing Tasks: Language Processing Tasks refer to a series of tasks that analyze, understand, generate, or interact with natural languages (such as Chinese, English, etc.).

[0060] Retrieval Task: Retrieval Task (Information Retrieval Task) refers to the process of quickly and accurately obtaining relevant information from a large-scale data collection according to user needs or query conditions.

[0061] Instruction Understanding Task: It refers to enabling a computer system to accurately parse instructions or requests given by users in natural language form through natural language processing technology.

[0062] Audio Classification Task: It refers to automatically classifying audio signals into predefined categories through machine learning or deep learning techniques.

[0063] Question Answering Task: It refers to enabling a computer system to understand questions posed by users in natural language form through natural language processing technology and return accurate and relevant answers.

[0064] Information Extraction Task: It refers to the process of automatically identifying and extracting structured information from unstructured or semi-structured data.

[0065] Voice Instruction: It refers to instructions or requests sent by users to a computer system, intelligent device, or application program in the form of natural language (spoken language).

[0066] Semantic Space: In natural language processing, Semantic Space refers to the representation of language symbols (such as words, sentences) and their meanings in a mathematical space.

[0067] Language Space: In natural language processing, Language Space refers to the set of language symbols (such as words, sentences) and their structured representations.

[0068] Figure 1 This is a schematic flowchart of the first task execution method provided by the embodiments of the present application.

[0069] It should be noted that the task execution method of the embodiments of the present application can be applied to a task execution device. In some possible embodiments, the task execution device can be configured in an electronic device so that the electronic device can execute the task execution function. Additionally, in some possible embodiments, the task execution device can also be software in the electronic device, etc.

[0070] Among them, the electronic device includes but is not limited to: a terminal, a personal computer (PC), etc. Among them, a terminal is an entity on the user side for receiving or transmitting signals, such as a mobile phone. A terminal can also be referred to as a terminal device, a user equipment (abbreviated as UE), a mobile station (abbreviated as MS), a mobile terminal device (abbreviated as MT), etc. A terminal can be an automobile with communication functions, a smart car, a mobile phone, a wearable device, a tablet computer (Pad), a computer with wireless transceiver functions, a virtual reality (abbreviated as VR) terminal, an augmented reality (abbreviated as AR) terminal, a wireless terminal in industrial control, a wireless terminal in self-driving, a wireless terminal in remote medical surgery, a wireless terminal in a smart grid, a wireless terminal in transportation safety, a wireless terminal in a smart city, a wireless terminal in a smart home, and so on. The embodiments of the present application do not limit the specific technologies and specific device forms adopted by the terminal.

[0071] As Figure 1 shown, the task execution method may include the following steps S101 to S103:

[0072] Step S101, obtain input data and a language processing task associated with the input data.

[0073] Among them, the input data includes but is not limited to: text, audio, etc. It should be noted that the input data can be unimodal or multimodal, and the present application does not limit this.

[0074] Among them, the language processing tasks include, but are not limited to: retrieval tasks, instruction understanding tasks, audio classification tasks, question and answer tasks, information extraction tasks, etc.

[0075] Among them, there is an association relationship between the input data and the language processing tasks. Among them, no restrictions are imposed on the association relationship. Exemplarily, the input data can be used as the basic information of the language processing task, or the input data drives the execution of the language processing task, or the input data is used as the constraint condition of the language processing task, etc. This application does not impose any restrictions on this.

[0076] Among them, when the input data includes text, no restrictions are imposed on the acquisition method of the text. Exemplarily, it can include, but is not limited to, the following acquisition methods:

[0077] 1. The text input by the user, or the text stored locally; among them, the input methods of the text include, but are not limited to, touch input (such as swiping, clicking, etc.), keyboard input, voice input, etc.

[0078] 2. The text obtained by web crawling technology;

[0079] 3. The text obtained after performing speech recognition on the audio;

[0080] For example, the text obtained by using a speech recognition algorithm to recognize the voice commands input by the user on the interaction interface of an electronic device (such as a terminal); or, the text obtained by using a speech recognition algorithm to perform speech recognition on the input video, etc.;

[0081] The text obtained after performing image recognition on the image;

[0082] For example, the text obtained by using an optical character recognition algorithm to recognize the text in the image, or, for a video, the text obtained by using an optical character recognition algorithm to extract the subtitles from the video image frames; or, the descriptive text of the video generated by using a video understanding model, etc.

[0083] 5. The new text generated based on multiple texts.

[0084] Among them, when the input data includes audio, no restrictions are imposed on the acquisition method of the audio. Exemplarily, it can include, but is not limited to, the following acquisition methods:

[0085] 1. The audio input by the user's voice, or the audio stored locally;

[0086] 2. The audio obtained by web crawling technology;

[0087] 3. The voice commands obtained through the interaction of the user on the interaction interface of an electronic device (such as a terminal);

[0088] 4. The audio obtained after extracting the audio from the video;

[0089] 5. The audio obtained by converting the text into audio;

[0090] 6. The new audio generated based on multiple audios.

[0091] Among them, there is no restriction on the acquisition method of the language processing task associated with the input data. Exemplarily, the language processing task can be a manually input language processing task, or a language processing task determined based on the application scenario, etc. Among them, the manual input methods include but are not limited to voice input, text input, gesture input, touch input, etc., and the present application does not limit this.

[0092] Step S102, call a multimodal model to perform a language processing task on the input data to obtain a task execution result; among them, the multimodal model is obtained by contrastive pre-training using text-audio pairs in multiple domains and multiple languages.

[0093] Among them, the multimodal model can be a model for simultaneously processing and fusing various modal data such as text and audio, and has language processing capabilities. For example, it can be a General Language-Audio Pretraining (GLAP) model, etc., and the present application does not limit this.

[0094] Among them, multiple domains include but are not limited to: the speech domain, the sound domain, the music domain, etc., and multiple languages include but are not limited to: English, Chinese, French, etc.

[0095] Among them, the text-audio pair refers to a data form in which text content is paired with the corresponding audio signal. It should be noted that the text and audio in the text-audio pair can be semantically related. Exemplarily, the semantically related audio and text can be audio and text with a semantic similarity greater than the first set threshold, where the first set threshold can be preset, and the present application does not limit its value.

[0096] Optionally, the text-audio pairs in multiple domains and multiple languages refer to those obtained by pairing text contents in different language forms with audios in different domains.

[0097] In the embodiments of the present application, text-audio pairs in multiple domains and multiple languages can be used to perform contrastive pre-training on a multimodal model, enabling the model to learn the semantic association between text and audio. Thus, using multimodal, multi-domain, and multilingual text-audio pairs (including text and audio with highly correlated semantics) to perform contrastive pre-training (i.e., contrastive learning) on the multimodal model enables the model to learn general cross-modal representations, which not only improves the adaptability of the model in different domains and languages but also enhances its applicability in diverse application scenarios. Moreover, through contrastive pre-training, on the one hand, the model can perform well on unseen data or new tasks, especially in zero-shot or few-shot scenarios, and this characteristic enables the model to better handle complex tasks. On the other hand, unlabeled multimodal data (text-audio pairs) can be fully utilized to pre-train the model, significantly reducing the dependence on labeled data and the cost of data collection and annotation.

[0098] In the embodiments of the present application, a multimodal model can be invoked to perform a language processing task associated with the input data on the input data to obtain a task execution result. Exemplarily, the input data can be input into the multimodal model, and the task execution result can be obtained in response to the output of the multimodal model.

[0099] It can be understood that the task execution result is strongly related to the language processing task.

[0100] Step S103, display the task execution result.

[0101] As an example, the task execution result can be displayed in an electronic device (such as a terminal).

[0102] In a possible implementation manner of the present application, the display manner of the task execution result can be determined according to the language processing task.

[0103] As an example, a correspondence relationship between the language processing task and the display manner can be established in advance and the above correspondence relationship can be saved. Furthermore, after determining the language processing task, the corresponding display manner can be used to display the task execution result of the language processing task. Exemplarily, when the language processing task is a retrieval task, the display manner of its task execution result is in text form; when the language processing task is an audio classification task, the display manner of its task execution result is in voice form, etc., which will not be elaborated here.

[0104] It should be noted that the above display manners corresponding to the language processing tasks are only exemplary, and there can be others in actual applications, and the present application does not limit this.

[0105] In a possible implementation manner of the embodiments of the present application, the input data can also be displayed.

[0106] In another possible implementation manner of the embodiments of the present application, it is also possible to display the complete processing link for performing a language processing task on the input data, or a partial processing link, etc., and the present application does not limit this.

[0107] The task execution method of the embodiments of the present application includes: obtaining input data and a language processing task associated with the input data; calling a multimodal model to perform a language processing task on the input data to obtain a task execution result, where the multimodal model is obtained by contrastive pre-training using text-audio pairs in multiple domains and multiple languages; and displaying the task execution result. Thus, by calling the contrastively pre-trained multimodal model for task processing, the semantic relationship between text and audio can be understood more accurately, thereby providing a higher-quality task execution result. Among them, using multimodal, multi-domain, and multi-language text-audio pairs (including text and audio with highly related semantics) to perform contrastive pre-training (i.e., contrastive learning) on the multimodal model enables the model to learn general cross-modal representations, which not only improves the adaptability of the model in different domains and languages but also enhances its applicability in diverse application scenarios. Moreover, through contrastive pre-training, on the one hand, the model can perform well on unseen data or new tasks, especially in zero-shot or few-shot scenarios, and this characteristic enables the model to better handle complex tasks. On the other hand, unlabeled multimodal data (text-audio pairs) can be fully utilized to pre-train the model, significantly reducing the dependence on labeled data and reducing the cost of data collection and annotation.

[0108] The embodiments of the present application provide another task execution method. Figure 2 It is a schematic flowchart of the second task execution method provided by the embodiments of the present application. It should be noted that this task execution method can be executed alone, or it can be executed in combination with any one of the embodiments or possible implementation manners in the present application, or it can also be executed in combination with any one of the technical solutions in the related art. The embodiments of the present application do not limit this.

[0109] As Figure 2 shown, this task execution method may include the following steps S201 to S203:

[0110] Step S201, obtain input data and the language processing task indicated by the input data.

[0111] Among them, it should be noted that the explanations of the acquisition methods of the input data, the language processing task, and the input data in step S101 also apply to this embodiment and will not be elaborated here.

[0112] In the embodiments of the present application, the input data can be used to indicate the corresponding language processing task.

[0113] To effectively and accurately determine the language processing task indicated by the input data, as a possible implementation, the corresponding language processing task can be determined according to the intent to which the input data belongs.

[0114] To accurately determine the intent to which the input data belongs, in a possible implementation, the input data can be matched with predefined keywords (such as "query", "retrieve", "complaint", etc.) and / or regular expressions to identify the intent to which the input data belongs. For example, assuming the predefined keyword includes "retrieve", if "retrieve" is included in the input data, then the intent to which the input data belongs is "retrieve"; for another example, assuming the predefined regular expression is "(today|tomorrow|the day after tomorrow|this week|next week)? What's the weather (like|how|condition|situation)?", and the input data is "What's the weather like in a certain place tomorrow", by matching the above input data with the predefined regular expression, it is determined that the intent to which the input data belongs is "query weather".

[0115] In another possible implementation, an intent recognition model can be used to perform intent recognition on the input data to obtain the intent to which the input data belongs.

[0116] Among them, the intent recognition model has an intent recognition function and can be a classification model (such as a Support Vector Machine (SVM) model, a random forest model, etc.) or a deep learning model (a Recurrent Neural Network (RNN) model, a Convolutional Neural Network (CNN) model, etc.), and this application does not limit this.

[0117] It should be noted that this application does not limit the way to obtain the intent to which the input data belongs. For example, the predefined regular expression can also be used as a preliminary screening tool for the intent to which the input data belongs, and then a classification model is used to perform classification processing on the screening results of the preliminary screening tool to obtain the intent to which the input data belongs.

[0118] It should be understood that the language processing task indicated by the input data can be explicit or implicit, and this application does not limit this. For example, when the input data is "Please help me retrieve the songs containing xx from the local songs", the language processing task indicated by the input data can be explicit, that is, the indicated language processing task is a "retrieval task"; for another example, when the input data is "very tired", the language processing task indicated by the input data can be implicit, and the intent to which the input data belongs can be determined through the context of the input data, so as to determine the language processing task indicated based on the intent of the input data.

[0119] In order to accurately determine the corresponding language processing task according to the intention to which the input data belongs, as a possible implementation, a corresponding relationship between the intention and the language processing task can be established in advance and this corresponding relationship can be saved. In subsequent applications, after determining the intention to which the input data belongs, the above corresponding relationship can be queried according to the intention to which the input data belongs to determine the corresponding language processing task.

[0120] Optionally, in some embodiments, the language processing task can also be indicated by the input object of the input data.

[0121] Among them, the input object is, for example, a user. As an example, in the case where the input object is a user, the language processing task associated with the input data can be indicated by means of manual input (such as voice, text, gestures (such as clicking, swiping, long pressing, etc.), interacting with controls in the touch human-computer interface, etc.).

[0122] It should be noted that the present application does not limit the acquisition method of the language processing task indicated by the input object.

[0123] Step S202: Invoke the multimodal model to perform the language processing task on the input data to obtain a task execution result; among them, the multimodal model is obtained by contrastive pre-training using text-audio pairs in multiple domains and multiple languages.

[0124] Step S203: Display the task execution result.

[0125] It should be noted that the execution process of steps S202 to S203 can refer to the execution process of any embodiment of the present application and will not be elaborated here.

[0126] In the task execution method of the embodiment of the present application, the language processing task is indicated by the input data, so that the language processing task called by the model is more targeted, and the accuracy and effectiveness of the task execution result are improved.

[0127] The embodiment of the present application provides another task execution method. Figure 3 It is a schematic flowchart of the third task execution method provided by the embodiment of the present application. It should be noted that this task execution method can be executed alone, or can be executed in combination with any one embodiment or possible implementation in the embodiments of the present application, or can also be executed in combination with any one technical solution in the related art. The embodiments of the present application do not limit this.

[0128] As Figure 3 shown, this task execution method can include the following steps S301 to S304:

[0129] Step S301: Obtain the input data and the language processing task associated with the input data.

[0130] It should be noted that the explanation of step S301 can be referred to the execution process of any embodiment of this application, and will not be elaborated here.

[0131] Step S302: Determine the encoding layer to be called from the multimodal model according to the language processing task.

[0132] It should be noted that the explanation of the multimodal model in step S102 also applies to this embodiment, and will not be elaborated here.

[0133] The multimodal model includes, but is not limited to, multilingual text encoding layers and multi-domain audio encoding layers, etc.

[0134] The multilingual text encoding layer, also known as the Multilingual Text Encoder, has the function of converting text data in different languages into feature representations.

[0135] The multi-domain audio encoding layer can be used to convert audio data in different domains into feature representations.

[0136] In the embodiment of this application, the encoding layer to be called is used to indicate the encoding layer in the multimodal model to be called, and can be at least one of the multilingual text encoding layer and the multi-domain audio encoding layer in the multimodal model. That is, the encoding layer to be called can be the multilingual text encoding layer in the multimodal model, or the encoding layer to be called can be the multi-domain audio encoding layer in the multimodal model, or the encoding layer to be called can include the multilingual text encoding layer and the multi-domain audio encoding layer in the multimodal model.

[0137] In the embodiment of this application, the encoding layer to be called can be determined from the multimodal model according to the language processing task.

[0138] As a possible implementation, a corresponding relationship between the language processing task and the called encoding layer can be established in advance and saved. Thus, when the language processing task is determined, the above corresponding relationship is queried to obtain the encoding layer corresponding to the language processing task, and this encoding layer is determined as the encoding layer to be called.

[0139] Step S303: Call the encoding layer to be called to execute the language processing task on the input data to obtain the task execution result.

[0140] It should be noted that the explanation of the task execution result in step S102 also applies to this embodiment, and will not be elaborated here.

[0141] As an example, the to-be-called encoding layer can be invoked to encode the input data to obtain the corresponding feature vector, and then a language processing task can be executed based on the feature vector to obtain the task execution result.

[0142] Step S304, display the task execution result.

[0143] It should be noted that the explanation of step S304 can be referred to the execution process of any embodiment of this application, and will not be elaborated here.

[0144] In the task execution method of the embodiment of this application, the to-be-called encoding layer is determined from the multi-modal model according to the language processing task; wherein, the to-be-called encoding layer includes at least one of a multi-language text encoding layer and a multi-domain audio encoding layer; the to-be-called encoding layer is invoked to execute the language processing task on the input data to obtain the task execution result. Thus, a task-driven multi-modal encoding layer invocation mechanism is adopted, and the required encoding layer can be dynamically loaded or activated for encoding during runtime, avoiding unnecessary computational overhead and significantly improving the flexibility and efficiency of the model in processing diverse language tasks.

[0145] The embodiment of this application provides another task execution method, Figure 4 which is a schematic flowchart of the fourth task execution method provided by the embodiment of this application. It should be noted that this task execution method can be executed alone, or can be executed in combination with any one embodiment or possible implementation manner in this application, or can also be executed in combination with any one technical solution in related technologies. The embodiment of this application does not limit this.

[0146] As Figure 4 shown, this task execution method may include the following steps S401 to S406:

[0147] Step S401, obtain the input data and the language processing task associated with the input data.

[0148] It should be noted that the explanation of step S401 can be referred to the execution process of any embodiment of this application, and will not be elaborated here.

[0149] In the embodiment of this application, the language processing task may include a retrieval task, and the input data may include input text.

[0150] Step S402, determine the to-be-called encoding layer from the multi-modal model according to the language processing task.

[0151] It should be noted that the explanation of step S402 can be referred to the execution process of any embodiment of this application, and will not be elaborated here.

[0152] In the embodiments of the present application, the encoding layer to be called may include a multi - language text encoding layer and a multi - domain audio encoding layer.

[0153] Step S403: Call the multi - language text encoding layer to encode the input text to obtain text features.

[0154] In the embodiments of the present application, the multi - language text encoding layer may be called to encode the input text, so as to obtain the features of the input text, which are denoted as text features in the present application, and the multi - modal model can learn the semantic representation information of the input text.

[0155] Step S404: Call the contrastive projection layer in the multi - modal model to project the text features and multiple first audio features into the same semantic space to obtain multiple audio projection features and text projection features.

[0156] Among them, the first audio features may be obtained by calling the multi - domain audio encoding layer to encode the candidate resources.

[0157] Among them, the candidate resources may be audio resources and / or video resources, and the present application does not limit this. It should be noted that the present application does not limit the number of candidate resources.

[0158] As a possible implementation, the candidate resources may be the resources indicated by the input text. For example, if the input text is "Please help me retrieve the video containing XX from the local album", then the candidate resources are "the videos in the local album".

[0159] As another possible implementation, the candidate resources may also be the resources indicated by the input object of the input data.

[0160] In the embodiments of the present application, the text projection feature refers to the feature representation obtained by projecting the text features of the text into the corresponding semantic space, while the audio projection feature is the feature representation obtained by projecting the audio features of the audio into the corresponding semantic space.

[0161] In the embodiments of the present application, the multi - modal model may also include a contrastive projection layer (Contrastive Projection Layers). Among them, the contrastive projection layer can project the text features and audio features into the same semantic space. For example, it can project the text features and audio features into the same semantic space through a two - way (or two - channel) multi - layer perceptron (Multi - Layer Perceptron, abbreviated as MLP), etc.

[0162] In the embodiments of the present application, the contrast projection layer in the multimodal model can be called to project the text features and multiple first audio features into the same semantic space, so as to obtain the audio projection features corresponding to any first audio feature and the text projection features corresponding to the text features. It should be noted that the present application does not limit the dimension of the semantic space into which the text features and audio features are projected. For example, it can be 1024, 2048, etc.

[0163] Step S405: Retrieve the target resource from the candidate resources according to the similarity between each audio projection feature and the text projection feature.

[0164] Among them, the similarity can be used to measure the semantic similarity degree between the corresponding audio and text.

[0165] In the embodiments of the present application, the target resource is semantically matched with the input text, and the target resource may include an audio resource and / or a video resource.

[0166] In the embodiments of the present application, the similarity between each audio projection feature and the text projection feature can be calculated, and based on the similarity between each audio projection feature and the text projection feature, the target resource that is speech-matched with the input text is retrieved from the candidate resources. Among them, it should be noted that the present application does not limit the calculation method of the similarity between the audio projection feature and the text projection feature. Exemplarily, similarity calculation methods such as cosine similarity and Euclidean distance can be used to determine the similarity between the audio projection feature and the text projection feature.

[0167] In the embodiments of the present application, the target resource that is speech-matched with the input text can be retrieved from the candidate resources based on the similarity between each audio projection feature and the text projection feature.

[0168] As an example, for any audio projection feature, when the similarity between the audio projection feature and the text projection feature is greater than the second set threshold, it can be determined that the candidate resource corresponding to the audio projection feature is the target resource. On the contrary, when the similarity between the audio projection feature and the text projection feature is not greater than the second set threshold, it can be determined that the candidate resource corresponding to the audio projection feature is not the target resource.

[0169] As another example, the candidate resource corresponding to the audio projection feature with the maximum similarity among the similarities between each audio projection feature and the text projection feature can be determined as the target resource.

[0170] Step S406: Display the task execution result.

[0171] In the embodiments of the present application, the task execution result may include the target resource that is semantically matched with the input text.

[0172] It should be noted that the explanation of step S406 can be referred to the execution process of any embodiment of this application, and will not be elaborated here.

[0173] As a possible implementation manner, when there are multiple target resources, the target resources can be sorted according to the corresponding similarity, and a sorting sequence is obtained, and the target resources are displayed according to the sorting sequence.

[0174] The task execution method of the embodiment of this application encodes the input text by calling the text encoding layer of multiple languages to obtain text features; calls the contrast projection layer in the multi-modal model to project the text features and multiple first audio features into the same semantic space to obtain multiple audio projection features and text projection features; wherein, the first audio features are obtained by encoding the candidate resources by calling the audio encoding layer of multiple domains; according to the similarity between each audio projection feature and the text projection feature, the target resources are retrieved from the candidate resources. Thus, in the application scenario of the retrieval task, the multi-modal model is used to align the text and audio in the same semantic space, and cross-modal semantic matching is realized based on the similarity, so as to improve the retrieval accuracy of the retrieval task.

[0175] The embodiment of this application provides another task execution method. Figure 5 It is a schematic flowchart of the fifth task execution method provided by the embodiment of this application. It should be noted that this task execution method can be executed alone, or can be executed together with any one embodiment or possible implementation manner in this application, or can also be executed together with any one technical solution in the related art. The embodiment of this application does not limit this.

[0176] Such as Figure 5 shown, this task execution method may include the following steps S501 to S507:

[0177] Step S501, obtain the input data and the language processing task associated with the input data.

[0178] It should be noted that the explanation of step S501 can be referred to the execution process of any embodiment of this application, and will not be elaborated here.

[0179] In the embodiment of this application, the language processing task is an instruction understanding task, and the input data may include the input voice instruction and multiple candidate texts associated with the voice instruction.

[0180] To effectively obtain multiple candidate texts associated with a voice command, as a possible implementation, keyword extraction can be performed on the voice command to obtain multiple target keywords; according to the target keywords, texts matching each target keyword are retrieved from a relevant database, and the retrieved texts matching each target keyword are determined as candidate texts. Exemplarily, the voice command is "find realistic works created by xx", keyword extraction is performed on this voice command, and multiple target keywords obtained include "xx", "created", and "realistic works"; according to the above target keywords, texts matching each target keyword are retrieved from a relevant database, and the retrieved texts matching each target keyword are determined as candidate texts. It should be noted that the above example of the voice command is only exemplary.

[0181] As another possible implementation, multiple candidate texts associated with the voice command can be obtained by means of manual input (such as voice input, gesture input, etc.).

[0182] It should be noted that this application places no restrictions on the method for obtaining multiple candidate texts associated with the voice command.

[0183] Step S502: Determine the encoding layer to be called from the multi-modal model according to the language processing task.

[0184] It should be noted that the explanation of step S502 can be referred to the execution process of any embodiment of this application and will not be elaborated here.

[0185] In the embodiments of this application, the encoding layer to be called may include a multi-language text encoding layer and a multi-domain audio encoding layer.

[0186] Step S503: Call the multi-language text encoding layer to encode the multiple candidate texts to obtain text features of the multiple candidate texts.

[0187] In the embodiments of this application, the multi-language text encoding layer can be called to encode any one of the multiple candidate features to obtain the corresponding text feature.

[0188] Step S504: Call the multi-domain audio encoding layer to encode the voice command to obtain a voice feature.

[0189] In the embodiments of this application, the multi-domain audio encoding layer can be called to encode the voice command to obtain the corresponding voice feature.

[0190] Step S505: Call the contrast projection layer in the multi-modal model to project the text features and voice features of the multiple candidate texts into the same semantic space to obtain text projection features and voice projection features of the multiple candidate texts.

[0191] Among them, the text projection feature refers to the feature representation obtained by projecting the text features of the text into the corresponding semantic space, while the speech projection feature refers to the feature representation obtained by projecting the speech features of the speech into the corresponding semantic space.

[0192] In the embodiment of the present application, the multi-modal model may further include contrastive projection layers. Among them, the contrastive projection layer can project the text features and speech features into the same semantic space.

[0193] In the embodiment of the present application, the contrastive projection layer in the multi-modal model can be called to project the text features and speech features of multiple candidate texts into the same semantic space, so as to obtain the text projection feature corresponding to any candidate text and the speech projection feature corresponding to the speech feature. It should be noted that the present application does not limit the dimension of the semantic space into which the text features and speech features are projected. For example, it can be 1024, 2048, etc.

[0194] Step S506, determine the target text from multiple candidate texts according to the similarity between the text projection features and speech projection features of the multiple candidate texts.

[0195] Among them, the similarity can be used to measure the similarity between the corresponding speech and text.

[0196] In the embodiment of the present application, the similarity between the text projection feature and the speech projection feature of each candidate text can be calculated, and based on the similarity between the text projection feature and the speech projection feature of each candidate text, the target text that semantically matches the speech instruction can be determined from multiple candidate texts. Among them, it should be noted that the present application does not limit the calculation method of the similarity between the speech projection feature and the text projection feature. Exemplarily, similarity calculation methods such as cosine similarity and Euclidean distance can be used to determine the similarity between the speech projection feature and the text projection feature.

[0197] In the embodiment of the present application, when determining the target text from multiple candidate texts based on the similarity between each text projection feature and the speech projection feature, in one example, for any text projection feature, when the similarity between the speech projection feature and the text projection feature is greater than the third set threshold, it can be determined that the candidate text corresponding to the speech projection feature is the target text that semantically matches the speech instruction. On the contrary, when the similarity between the speech projection feature and the text projection feature is not greater than the third set threshold, it can be determined that the candidate text corresponding to the text projection feature is not the target text that semantically matches the speech instruction.

[0198] As another example, among the similarities between the text projection features and the speech projection features, the candidate text corresponding to the text projection feature with the maximum similarity can be determined as the target text that semantically matches the speech instruction.

[0199] Step S507, display the task execution result.

[0200] In the embodiments of the present application, the task execution result may include the target text that semantically matches the speech instruction.

[0201] It should be noted that the execution process of step S507 can refer to the execution process of any embodiment of the present application and will not be elaborated here.

[0202] As a possible implementation manner, when there are multiple target texts, the target texts can be sorted according to the magnitudes of the corresponding similarities to obtain a sorting sequence, and the target texts are displayed according to the sorting sequence.

[0203] The task execution method in the embodiments of the present application calls the text encoding layer in multiple languages to encode multiple candidate texts to obtain the text features of the multiple candidate texts; calls the audio encoding layer in multiple domains to encode the speech instruction to obtain the speech features; calls the contrast projection layer in the multimodal model to project the text features and the speech features of the multiple candidate texts into the same semantic space to obtain the text projection features and the speech projection features of the multiple candidate texts; and determines the target text from the multiple candidate texts according to the similarities between the text projection features and the speech projection features of the multiple candidate texts. Thus, in the application scenario of the instruction understanding task, a multimodal model can be used to achieve the alignment of text and speech in the same semantic space, and accurately determine the target text that semantically matches the speech instruction based on the similarity, improving the accuracy of instruction understanding.

[0204] The embodiments of the present application provide another task execution method. Figure 6 It is a schematic flowchart of the sixth task execution method provided by the embodiments of the present application. It should be noted that this task execution method can be executed alone, or can be executed in combination with any one embodiment or possible implementation manner in the present application, or can also be executed in combination with any one technical solution in the related art. The embodiments of the present application do not limit this.

[0205] As Figure 6 shown, this task execution method may include the following steps S601 to S606:

[0206] Step S601, obtain the input data and the language processing task associated with the input data.

[0207] In the embodiments of the present application, the language processing task may include an audio classification task, and the input data may include the input audio to be classified. Among them, the audio to be classified refers to the audio whose category is to be determined. It should be noted that the present application does not limit the number of audios to be classified. For example, it can be one, or it can also be multiple.

[0208] It should be noted that the explanatory description of step S601 can be referred to the execution process of any embodiment of the present application, and will not be elaborated here.

[0209] Step S602, determine the encoding layer to be called from the multi-modal model according to the language processing task.

[0210] In the embodiments of the present application, the encoding layer to be called may include audio encoding layers in multiple domains.

[0211] It should be noted that the explanatory description of step S602 can be referred to the execution process of any embodiment of the present application, and will not be elaborated here.

[0212] Step S603, call the audio encoding layers in multiple domains to encode the audio to be classified, and obtain the second audio feature.

[0213] In the embodiments of the present application, the audio encoding layers in multiple domains can be called to encode the audio to be classified, and obtain the feature representation corresponding to the audio to be classified, which is denoted as the second audio feature in the present application.

[0214] Step S604, call the contrastive projection layer in the multi-modal model to project the second audio feature into the language space associated with the large language model, and obtain the first projection feature.

[0215] In the embodiments of the present application, the multi-modal model may also include contrastive projection layers (ContrastiveProjection Layers).

[0216] In the embodiments of the present application, the language space associated with the large language model may refer to the language space that the large language model can understand, learn, or process, etc. It should be noted that the present application does not limit the large language model, and the large language model can perform a classification function.

[0217] In the embodiments of the present application, the contrastive projection layer in the multi-modal model can be called to project the second audio feature into the language space associated with the large language model, and obtain the feature representation of the second audio feature in the language space, that is, which is denoted as the first projection feature in the present application. It should be noted that the present application does not limit the dimension of the language space.

[0218] Step S605, call the large language model to classify the first projection feature to obtain the target category.

[0219] In an embodiment of the present application, a large language model can be called to classify the first projection feature to obtain the target category corresponding to the audio to be classified. Exemplarily, the first projection feature can be input into the large language model, and the target category to which the audio to be classified belongs can be determined in response to the output of the large language model.

[0220] It should be noted that the classification of the audio to be classified in the present application can be performed according to content type, language features, emotion, intention, technical requirements, application scenarios, etc., and the present application does not limit this.

[0221] Step S606, display the task execution result.

[0222] In an embodiment of the present application, the task execution result may include the target category to which the audio to be classified belongs.

[0223] It should be noted that the explanation of step S606 can be referred to in the execution process of any embodiment of the present application, and will not be elaborated here.

[0224] It should be noted that the present application does not limit the display method of the target category to which the audio to be classified belongs. Exemplarily, it can be displayed in text form, or it can also be displayed in voice form, etc., and the present application does not limit this.

[0225] The task execution method in the embodiment of the present application encodes the audio to be classified by calling the audio coding layers in multiple domains to obtain the second audio feature; calls the contrast projection layer in the multimodal model to project the second audio feature into the language space associated with the large language model to obtain the first projection feature; calls the large language model to classify the first projection feature to obtain the target category. Thus, in the application scenario of the audio classification task, by calling the multimodal model that has undergone contrastive pre-training to process the audio classification task, the accuracy of the audio classification result can be provided.

[0226] The embodiment of the present application provides another task execution method, Figure 7 which is a schematic flowchart of the seventh task execution method provided by the embodiment of the present application. It should be noted that this task execution method can be executed alone, or it can also be executed in combination with any one embodiment in the present application or possible implementation manners in the embodiment, or it can also be executed in combination with any one technical solution in the related art. The embodiment of the present application does not limit this.

[0227] As Figure 7 shown, this task execution method may include the following steps S701 to S706:

[0228] Step S701, obtain the input data and the language processing task associated with the input data.

[0229] In an embodiment of the present application, the language processing task may include a question-and-answer task, and the input data may include an input voice question. Among them, the voice question is a question in voice form.

[0230] It should be noted that the explanation of step S701 can be referred to the execution process of any embodiment of the present application, and will not be elaborated here.

[0231] Step S702, determine the encoding layer to be called from the multimodal model according to the language processing task.

[0232] In an embodiment of the present application, the encoding layer to be called may include audio encoding layers in multiple domains.

[0233] It should be noted that the execution process of step S702 can be referred to the execution process of any embodiment of the present application, and will not be elaborated here.

[0234] Step S703, call the audio encoding layers in multiple domains to encode the voice question to obtain question features.

[0235] In an embodiment of the present application, the audio encoding layers in multiple domains can be called to encode the voice question to obtain a feature representation corresponding to the voice question, which is denoted as question features in the present application.

[0236] Step S704, call the contrast projection layer in the multimodal model to project the question features into the language space associated with the large language model to obtain question projection features.

[0237] In an embodiment of the present application, the multimodal model may further include a contrast projection layer.

[0238] In an embodiment of the present application, the language space associated with the large language model may refer to a language space that the large language model can understand, learn, or process, etc. It should be noted that the present application does not limit the large language model, and the large language model has a function of generating answers.

[0239] In an embodiment of the present application, the contrast projection layer in the multimodal model can be called to project the question features into the language space associated with the large language model to obtain the feature representation of the question-and-answer features in the language space, that is, which is denoted as question projection features in the present application. It should be noted that the present application does not limit the dimension of the language space.

[0240] Step S705, call the large language model to process the question projection features to obtain a text answer.

[0241] In an embodiment of the present application, the large language model can be called to process the question projection features to obtain a text answer that matches the input voice question. Exemplarily, the question-and-answer projection features can be input into the large language model, and the large language model outputs a text answer that matches the voice question.

[0242] Step S706: Display the task execution result.

[0243] In an embodiment of the present application, the task execution result may include a text answer that matches the voice question.

[0244] It should be noted that the execution process of step S706 can refer to the execution process of any embodiment of the present application and will not be repeated here.

[0245] It should be noted that this application does not limit the display method of the text answers matching the voice questions. For example, they can be displayed in text form, or in voice form, etc. This application does not limit this.

[0246] The task execution method in the embodiment of the present application encodes the spoken question by invoking a multi-domain audio encoding layer to obtain question features; then invoking a contrastive projection layer in a multimodal model to project the question features into a language space associated with a large language model to obtain question projection features; and finally, invoking the large language model to process the question projection features to obtain a textual answer. Thus, in the application scenario of question-answering tasks, invoking a pre-trained contrastive multimodal model to perform audio classification tasks can improve the accuracy of the corresponding textual answer output.

[0247] This embodiment of the application provides another task execution method. Figure 8 This is a flowchart of the eighth task execution method provided in the embodiments of the present application. It should be noted that this task execution method can be executed alone, or it can also be executed in combination with any embodiment of the present application or any possible implementation method in the embodiment, or it can also be executed in combination with any technical solution in the related art, and the embodiments of the present application do not limit this.

[0248] like Figure 8 As shown, the task execution method may include the following steps S801 to S806:

[0249] Step S801: Acquire input data and a language processing task associated with the input data.

[0250] In an embodiment of the present application, the language processing task may include an information extraction task.

[0251] It should be noted that the execution process of step S801 can refer to the execution process of any embodiment of the present application and will not be repeated here.

[0252] Step S802: Determine the encoding layer to be called from the multimodal model according to the language processing task.

[0253] In the embodiments of the present application, the encoding layer to be called may include audio encoding layers in multiple domains.

[0254] It should be noted that the execution process of step S802 can refer to the execution process of any embodiment of the present application and will not be elaborated here.

[0255] Step S803: In response to the input data including the audio to be extracted, call the audio encoding layers in multiple domains to encode the audio to be extracted, and obtain the second audio feature.

[0256] Among them, the audio to be extracted can be a resource in audio form, or it can also be an audio resource extracted from a resource in video form. The present application does not limit this. It should also be noted that the present application does not limit the number of audios to be extracted.

[0257] In the embodiments of the present application, when the input data includes the audio to be extracted, the audio encoding layers in multiple domains can be called to encode the audio to be extracted, and the feature representation corresponding to the audio to be extracted is obtained, which is denoted as the second audio feature in the present application.

[0258] Step S804: Call the contrast projection layer in the multimodal model to project the second audio feature into the language space associated with the large language model, and obtain the second projection feature.

[0259] In the embodiments of the present application, the multimodal model may also include a contrast projection layer.

[0260] In the embodiments of the present application, the language space associated with the large language model may refer to the language space that the large language model can understand, learn, process, etc. It should be noted that the present application does not limit the large language model, and the large language model has an information extraction function.

[0261] In the embodiments of the present application, the contrast projection layer in the multimodal model can be called to project the second audio feature into the language space associated with the large language model, so as to convert the second audio feature into a feature representation that the large language model can process, that is, it is denoted as the second projection feature in the present application. It should be noted that the present application does not limit the dimension of the language space.

[0262] Step S805: Call the large language model to perform information extraction on the second projection feature to obtain the target information.

[0263] Among them, the target information may include keywords and / or entity words.

[0264] In the embodiments of the present application, a large language model can be called to extract information from the second projection feature, so as to obtain the target information in the audio to be extracted. Exemplarily, the second projection feature can be input into the large language model, and the target information in the audio to be extracted is output through the large language model.

[0265] Step S806, display the task execution result.

[0266] In the embodiments of the present application, the task execution result may include the target information in the audio to be extracted.

[0267] It should be noted that the explanation of step S806 can refer to the execution process of any embodiment of the present application, and will not be elaborated here.

[0268] In the case that the encoding layer to be called further includes a multilingual text encoding layer, in some possible implementation manners, in response to the input data including the text to be extracted, the multilingual text encoding layer is called to encode the text to be extracted, so as to obtain an encoding feature; the contrast projection layer in the multimodal model is called to project the encoding feature into the language space associated with the large language model, so as to obtain a third projection feature; the large language model is called to extract information from the third projection feature, so as to obtain the target information.

[0269] As an example, in the case that the input data includes the text to be extracted, the multilingual text encoding layer can be called to encode the text to be extracted, so as to obtain the feature representation of the text to be extracted, that is, denoted as the encoding feature in the present application; furthermore, the contrast projection layer in the multimodal model is called to project the encoding feature into the language space associated with the large language model, and the encoding feature is converted into the feature representation in the language space that can be processed by the large language model, that is, denoted as the third projection feature in the present application; finally, the large language model is called to extract information from the third projection feature, so as to obtain the target information.

[0270] It should be noted that the present application does not limit the display manner of the target information. Exemplarily, it can be displayed in text form, or can also be displayed in voice form, etc., and the present application does not limit this.

[0271] The task execution method of the embodiments of the present application, by responding to the input data including the audio to be extracted, calls the audio encoding layer in multiple domains to encode the audio to be extracted, so as to obtain a second audio feature; calls the contrast projection layer in the multimodal model to project the second audio feature into the language space associated with the large language model, so as to obtain a second projection feature; calls the large language model to extract information from the second projection feature, so as to obtain the target information. Thus, in the application scenario of the information extraction task, by calling the multimodal model that has undergone contrastive pre-training for task processing, the accuracy and effectiveness of the extracted information are improved.

[0272] It should be noted that the above are the respective embodiments corresponding to the application method (i.e., the task execution method) of the multi-modal model. The present application also provides a training method for the multi-modal model. Figure 9 It is a schematic flowchart of the first training method for the multi-modal model provided by the embodiments of the present application.

[0273] It should be noted that the training method of this multi-modal model can be executed alone, or it can also be executed in combination with any one of the embodiments in the present application or the possible implementation manners in the embodiments, or it can also be executed in combination with any one of the technical solutions in the related art. The embodiments of the present application do not limit this.

[0274] As Figure 9 shown, the training method of this multi-modal model may include the following steps S901 to S905:

[0275] Step S901, obtain a training set.

[0276] Among them, the training set may include text-audio pairs in multiple fields and text-audio pairs in multiple languages. The text-audio pair may include a sample audio and a sample text that are semantically related. Exemplarily, the sample audio and the sample text that are semantically related may be the sample audio and the sample text whose semantic similarity is greater than a first set threshold, where the first set threshold may be preset, and the present application does not limit its value.

[0277] It should be noted that the explanations of multiple fields and multiple languages in step S102 also apply to this embodiment and will not be elaborated here.

[0278] It should also be noted that in order to construct a comprehensive and diverse training set, text-audio pairs in multiple fields and multiple languages can be collected. Optionally, the training set may include text-audio pairs in multiple application scenarios, where the application scenarios may be classified according to language processing tasks. The language processing tasks may include but are not limited to: retrieval tasks, instruction understanding tasks, audio classification tasks, question answering tasks, information extraction tasks, etc.

[0279] In order to obtain the training set, in some possible implementation manners, text-audio pairs in multiple fields can be obtained; the sample texts in the text-audio pairs are translated to obtain enhanced texts in multiple languages; according to the enhanced texts and the sample audios in the text-audio pairs, text-audio pairs in multiple languages are generated; according to the text-audio pairs in multiple fields and the text-audio pairs in multiple languages, a training set is generated.

[0280] As an example, after obtaining text-audio pairs in multiple domains, the sample texts in the text-audio pairs in multiple domains can be translated into enhanced texts in multiple languages. Furthermore, the enhanced texts are paired with the sample audios in the corresponding text-audio pairs to form multilingual text-audio pairs. Finally, a training set is constructed based on the text-audio pairs in multiple domains and the multilingual text-audio pairs.

[0281] Thus, based on the sample enhancement technology, the training set can be expanded to improve the training effect of the model.

[0282] Step S902: Use the audio encoding layer in the multimodal model to encode the sample audios in N text-audio pairs to obtain the sample audio features of the N text-audio pairs.

[0283] In the embodiments of the present application, the multimodal model may include an audio encoding layer, where the audio encoding layer may have the function of converting audios in multiple domains into corresponding feature representations.

[0284] Where N can be a preset positive integer, used to indicate the number of text-audio pairs that the multimodal model can process in one batch. The present application does not limit the value of N.

[0285] As an example, when performing contrastive training on the multimodal model using N text-audio pairs in any batch in the training set, the audio encoding layer in the multimodal model can be used to sequentially convert the sample audios in the N text-audio pairs in this batch into corresponding feature representations, which are denoted as sample audio features in the present application.

[0286] Step S903: Use the text encoding layer in the multimodal model to encode the sample texts in N text-audio pairs respectively to obtain the sample text features of the N text-audio pairs.

[0287] In the embodiments of the present application, the multimodal model may include a text encoding layer, where the text encoding layer may have the function of converting texts in multiple languages into corresponding feature representations.

[0288] As an example, the text encoding layer in the multimodal model can be used to sequentially convert the sample texts in the N text-audio pairs in the corresponding batch into feature vectors corresponding to the text-audio pairs, which are denoted as sample text features in the present application.

[0289] Step S904: Use the contrastive projection layer in the multimodal model to project the sample audio features and sample text features corresponding to N text-audio pairs into the same semantic space to obtain the audio projection features and text projection features corresponding to the N text-audio pairs.

[0290] In an embodiment of the present application, the multimodal model may further include a contrast projection layer. The contrast projection layer may be used to map text features and audio features to the same semantic space. For example, the text features and audio features may be mapped to the same semantic space through a two-way (or two-channel) MLP or the like.

[0291] In an embodiment of the present application, the contrast projection layer in the multimodal model may be used to project the sample audio features and sample text features corresponding to any text-audio pair in a corresponding batch to the same semantic space, obtaining the audio projection features corresponding to the sample audio features and the text projection features corresponding to the sample text features, that is, obtaining the audio projection features and text projection features corresponding to the corresponding text-audio pair.

[0292] Step S905: Perform contrastive pre-training on the multimodal model based on the text projection features and audio projection features corresponding to N text-audio pairs.

[0293] In an embodiment of the present application, the multimodal model may be contrastively pre-trained based on the text projection features and audio projection features corresponding to N text-audio pairs in a corresponding batch.

[0294] The training method of the multimodal model according to the embodiment of the present application includes obtaining a training set, where the training set includes text-audio pairs in multiple domains and text-audio pairs in multiple languages, and the text-audio pairs include sample audio and sample text that are semantically related; encoding the sample audio in N text-audio pairs by using the audio encoding layer in the multimodal model to obtain the sample audio features of the N text-audio pairs, where N is a set positive integer; encoding the sample text in N text-audio pairs respectively by using the text encoding layer in the multimodal model to obtain the sample text features of the N text-audio pairs; projecting the sample audio features and sample text features corresponding to N text-audio pairs to the same semantic space by using the contrast projection layer in the multimodal model to obtain the audio projection features and text projection features corresponding to the N text-audio pairs; and performing contrastive pre-training on the multimodal model based on the text projection features and audio projection features corresponding to N text-audio pairs. Thus, by using the training set to perform contrastive pre-training on the multimodal model, the prediction accuracy of the multimodal model can be improved, and further the task execution accuracy in different application scenarios can be improved.

[0295] The embodiment of the present application provides another training method for a multimodal model. Figure 10Schematic flowchart of the second method for training a multimodal model provided by an embodiment of this application. It should be noted that this method for training a multimodal model can be executed independently, or it can be executed in combination with any one of the embodiments in this application or possible implementation manners in the embodiments, or it can also be executed in combination with any one of the technical solutions in related technologies. Embodiments of this application do not impose any restrictions on this.

[0296] As Figure 10 shown, this method for training a multimodal model may include the following steps S1001 to S1008:

[0297] Step S1001, obtain a training set.

[0298] Step S1002, use the audio encoding layer in the multimodal model to encode the sample audio in N text-audio pairs to obtain the audio sample audio features of the N text-audio pairs.

[0299] Step S1003, use the text encoding layer in the multimodal model to encode the sample text in N text-audio pairs respectively to obtain the sample text features of the N text-audio pairs.

[0300] Step S1004, use the contrast projection layer in the multimodal model to project the sample audio features and sample text features corresponding to N text-audio pairs into the same semantic space to obtain the audio projection features and text projection features corresponding to the N text-audio pairs.

[0301] It should be noted that the explanations of steps S1001 to S1004 can be referred to the execution process of any embodiment of this application and will not be elaborated here.

[0302] Step S1005, obtain the target distance between the text projection feature of the i-th text-audio pair and the audio projection feature of the j-th text-audio pair.

[0303] Among them, i and j can be positive integers less than or equal to N.

[0304] Among them, the target distance between the text projection feature of the i-th text-audio pair and the audio projection feature of the j-th text-audio pair can be used to measure the semantic similarity degree between the sample text of the i-th text-audio pair and the sample audio of the j-th text-audio pair.

[0305] In some possible implementation manners, the initial distance between the text projection feature of the i-th text-audio pair and the audio projection feature of the j-th text-audio pair can be determined, and the initial distance can be corrected according to the learnable parameters in the multimodal model to obtain the target distance.

[0306] It should be noted that the present application does not limit the method for obtaining the initial distance between the text projection feature of the i-th text-audio pair and the audio projection feature of the j-th text-audio pair. Exemplarily, methods such as cosine similarity and Euclidean distance can be used to determine the initial distance between the text projection feature of the i-th text-audio pair and the audio projection feature of the j-th text-audio pair.

[0307] As an example, assuming that the initial distance between the text projection feature of the i-th text-audio pair and the audio projection feature of the j-th text-audio pair is S(i,j), the initial distance between the text projection feature of the i-th text-audio pair and the audio projection feature of the j-th text-audio pair can be corrected according to the following formula to obtain the target distance S'(i,j) between the text projection feature of the i-th text-audio pair and the audio projection feature of the j-th text-audio pair:

[0308]

[0309] where τ and β are learnable parameters in the multimodal model.

[0310] Thus, by correcting the distance between the text feature and the audio feature in the same semantic space, the prediction accuracy of the multimodal model can be improved.

[0311] Step S1006, obtain the value of the target flag bit.

[0312] The target flag bit can be used to indicate whether the sample text in the i-th text-audio pair and the sample audio in the j-th text-audio pair are a positive sample pair. Among them, a positive sample pair refers to a semantically similar sample pair, and conversely, a negative sample pair refers to a semantically dissimilar sample pair.

[0313] In one example, when the target flag bit takes the target value, it indicates that the sample text in the i-th text-audio pair and the sample audio in the j-th text-audio pair are a positive sample pair. Conversely, when the target flag bit does not take the target value, it indicates that the sample text in the i-th text-audio pair and the sample audio in the j-th text-audio pair are a negative sample pair. It should be noted that the present application does not limit the value of the target value, such as the target value being 1, 2, etc.

[0314] Exemplarily, the value of the target flag bit corresponding to the sample text in the i-th text-audio pair and the sample audio in the j-th text-audio pair can be determined according to the following formula as:

[0315]

[0316] That is to say, when i = j, the value of the target flag bit corresponding to the sample text in the i-th text-audio pair and the sample audio in the j-th text-audio pair is 1, indicating that the sample text in the i-th text-audio pair and the sample audio in the j-th text-audio pair are positive sample pairs; when i ≠ j, the value of the target flag bit corresponding to the sample text in the i-th text-audio pair and the sample audio in the j-th text-audio pair is -1, indicating that the sample text in the i-th text-audio pair and the sample audio in the j-th text-audio pair are negative sample pairs.

[0317] Step S1007: Determine the loss value according to the product between the target distance and the value of the target flag bit.

[0318] In the embodiments of the present application, the loss value can be determined according to the product between the target distance and the value of the target flag bit. In one example, assuming that among N text-audio pairs, the target distance between the text projection feature of the i-th text-audio pair and the audio projection feature of the j-th text-audio pair is S'(i, j), and the value of the target flag bit corresponding to the text projection feature of the i-th text-audio pair and the audio projection feature of the j-th text-audio pair is Then the loss value can be determined according to the following formula:

[0319]

[0320] where L represents the loss value, σ(·) is the Sigmoid function, and "·" represents element-wise multiplication.

[0321] Step S1008: Perform contrastive pre-training on the multi-modal model based on the loss value.

[0322] In the embodiments of the present application, the multi-modal model can be contrastively pre-trained based on the loss value. Exemplarily, the multi-modal model can be contrastively pre-trained according to the loss value to minimize the loss value.

[0323] It should be noted that the above only takes the termination condition of contrastive pre-training as minimizing the loss value for example. In actual applications, other termination conditions can also be set, such as the number of training rounds reaching the set number of rounds, the training duration reaching the set duration, etc. The present application does not limit this.

[0324] Thus, by performing contrastive pre-training on the multi-modal model through the loss value, the accuracy of the multi-modal model can be improved, that is, the model can more accurately understand the semantic relationship between text and audio.

[0325] The training method of the multimodal model according to the embodiments of the present application obtains the target distance between the text projection feature of the i-th text-audio pair and the audio projection feature of the j-th text-audio pair, where i is a positive integer less than or equal to N; obtains the value of the target flag bit, where the target flag bit is used to indicate whether the sample text in the i-th text-audio pair and the sample audio in the j-th text-audio pair are a positive sample pair; determines the loss value according to the product of the target distance and the value of the target flag bit; and performs contrastive pre-training on the multimodal model based on the loss value. Thus, by performing contrastive pre-training on the multimodal model through the loss value, the accuracy of the multimodal model can be improved, that is, the model can more accurately understand the semantic relationship between text and audio.

[0326] In any one of the embodiments of the present application, the training method of the multimodal model provided by the present application is exemplified by a General Language-Audio Pretraining (GLAP) framework. Among them, the General Language-Audio Pretraining framework is as Figure 11 shown, and can be an end-to-end contrastive learning framework, which may include the following modules:

[0327] General audio encoder (denoted as a multi-domain audio coding layer in the present application): supports variable-length audio input, and extracts cross-domain robust representations through multi-domain task pre-training (such as pre-training with sample audio in fields such as speech recognition, sound event detection, and music classification).

[0328] Multilingual text encoder (denoted as a multilingual text coding layer in the present application): supports semantic mapping of multiple languages, and can expand English text annotations to other languages through automatic translation enhancement technology.

[0329] Contrastive projection layer: can map the audio representation and the text representation to a unified dimensional space (such as a dimension of 1024) through a two-way MLP, as Figure 11 shown, map the text representation to this space through the MLP T and map the audio representation to this space through the MLP A and use an improved Sigmoid contrast loss function to replace the cross-entropy loss to improve the stability of large-scale training.

[0330] The training process may include the following steps:

[0331] 1. Obtain a training set. Among them, the training set can include text-audio pairs in multiple domains and text-audio pairs in multiple languages. The text-audio pairs can include sample audio and sample text that are semantically related.

[0332] 2. Use the audio encoding layer in the multimodal model to encode the sample audio in any batch of N text-audio pairs to obtain audio sample features of the N text-audio pairs; where N is a set positive integer.

[0333] 3. Use the text encoding layer in the multimodal model to encode the sample text in N text-audio pairs respectively to obtain sample text features of the N text-audio pairs.

[0334] 4. Use the contrast projection layer in the multimodal model to project the sample audio features and sample text features corresponding to N text-audio pairs into the same semantic space to obtain audio projection features and text projection features corresponding to the N text-audio pairs.

[0335] 5. Determine the initial distance between the text projection feature of the i-th text-audio pair and the audio projection feature of the j-th text-audio pair among the N text-audio pairs. Where i and j are both positive integers less than or equal to N.

[0336] Figure 11 Among them, e ti represents the text projection feature of the sample text in the i-th text-audio pair, e aj represents the audio projection feature of the sample audio in the j-th text-audio pair, and S(i, j) represents the initial distance between the text projection feature of the i-th text-audio pair and the audio projection feature of the j-th text-audio pair. Figure 11 Only take N = 4 as an example.

[0337] Optionally, methods such as cosine similarity and Euclidean distance can be used to determine the initial distance between the text projection feature of the i-th text-audio pair and the audio projection feature of the j-th text-audio pair.

[0338] 6. According to the learnable parameters in the multimodal model, correct the initial distance to obtain the target distance. Exemplarily, formula (1) can be used to correct the initial distance based on the learnable parameters in the multimodal model to obtain the target distance.

[0339] 7. Obtain the value of the target flag bit; where the target flag bit is used to indicate whether the sample text in the i-th text-audio pair and the sample audio in the j-th text-audio pair are positive sample pairs. Exemplarily, formula (2) can be used to determine the value of the target flag bit corresponding to the sample text in the i-th text-audio pair and the sample audio in the j-th text-audio pair.

[0340] 8. Determine the loss value according to the product of the target distance and the value of the target flag bit. Exemplarily, formula (3) can be used to determine the loss value according to the product of the target distance and the value of the target flag bit.

[0341] 9. Based on the loss value, perform contrastive pre-training on the multimodal model.

[0342] In summary, the solution provided by this application uses multimodal, multi-domain, and multilingual text-audio pairs (including texts and audios with highly relevant semantics) to perform contrastive pre-training (i.e., contrastive learning) on the multimodal model, enabling the model to learn general cross-modal representations. This not only improves the adaptability of the model in different domains and languages but also enhances its applicability in diverse application scenarios. The model can perform well on unseen data or new tasks, especially in zero-shot or few-shot scenarios, and this characteristic enables the model to better handle complex tasks. Moreover, through contrastive pre-training, on the one hand, the model can perform well on unseen data or new tasks, especially in zero-shot or few-shot scenarios, and this characteristic enables the model to better handle complex tasks. On the other hand, it can make full use of unlabeled multimodal data (text-audio pairs) to pre-train the model, significantly reducing the dependence on labeled data and reducing the cost of data collection and annotation.

[0343] To implement the above embodiments, an embodiment of this application also proposes a task execution device.

[0344] Figure 12 It is a schematic structural diagram of a task execution device provided by an embodiment of this application.

[0345] As Figure 12 shown, the task execution device 1200 may include: an acquisition module 1201, a processing module 1202, and a display module 1203.

[0346] Among them, the acquisition module 1201 is used to acquire input data and the language processing task associated with the input data; the execution module 1202 is used to call the multimodal model to perform the language processing task on the input data to obtain a task execution result; among them, the multimodal model is obtained by performing contrastive pre-training using multi-domain and multilingual text-audio pairs; the display module 1203 is used to display the task execution result.

[0347] Further, in an implementation manner of an embodiment of this application, the language processing task is indicated by the input data and determined according to the intention to which the input data belongs; or the language processing task is indicated by the input object of the input data.

[0348] In an implementation manner of the embodiment of the present application, the execution module 1202 is configured to: determine an encoding layer to be called from a multi-modal model according to a language processing task; wherein, the encoding layer to be called includes at least one of a multi-language text encoding layer and a multi-domain audio encoding layer; call the encoding layer to be called to execute the language processing task on the input data to obtain a task execution result.

[0349] In an implementation manner of the embodiment of the present application, in response to the language processing task including a retrieval task, the input data includes an input text, the encoding layer to be called includes a multi-language text encoding layer and a multi-domain audio encoding layer, and the task execution result includes a target resource semantically matching the input text, and the target resource includes an audio resource and / or a video resource; the execution module 1202 is configured to: call the multi-language text encoding layer to encode the input text to obtain a text feature; call a contrast projection layer in the multi-modal model to project the text feature and multiple first audio features into the same semantic space to obtain multiple audio projection features and text projection features; wherein, the first audio feature is obtained by calling the multi-domain audio encoding layer to encode a candidate resource; retrieve the target resource from the candidate resources according to the similarity between each audio projection feature and the text projection feature.

[0350] In an implementation manner of the embodiment of the present application, in response to the language processing task including an instruction understanding task, the input data includes an input voice instruction and multiple candidate texts associated with the voice instruction, the encoding layer to be called includes a multi-language text encoding layer and a multi-domain audio encoding layer, and the task execution result includes a target text semantically matching the voice instruction; the execution module 1202 is configured to: call the multi-language text encoding layer to encode the multiple candidate texts to obtain text features of the multiple candidate texts; call the multi-domain audio encoding layer to encode the voice instruction to obtain a voice feature; call a contrast projection layer in the multi-modal model to project the text features of the multiple candidate texts and the voice feature into the same semantic space to obtain text projection features and voice projection features of the multiple candidate texts; determine the target text from the multiple candidate texts according to the similarity between the text projection features and the voice projection features of the multiple candidate texts.

[0351] In an implementation manner of the embodiment of the present application, in response to the language processing task including an audio classification task, the input data includes an input audio to be classified, the encoding layer to be called includes a multi-domain audio encoding layer, and the task execution result includes a target category to which the audio to be classified belongs; the execution module 1202 is configured to: call the multi-domain audio encoding layer to encode the audio to be classified to obtain a second audio feature; call a contrast projection layer in the multi-modal model to project the second audio feature into a language space associated with a large language model to obtain a first projection feature; call the large language model to classify the first projection feature to obtain the target category.

[0352] In an implementation manner of the embodiment of the present application, in response to the language processing task including a question-and-answer task, the input data includes the input voice question, the encoding layer to be called includes an audio encoding layer in multiple domains, and the task execution result includes the text answer matching the voice question; the execution module 1202 is configured to: call the audio encoding layer in multiple domains to encode the voice question to obtain question features; call the contrast projection layer in the multimodal model to project the question features into the language space associated with the large language model to obtain question projection features; call the large language model to process the question projection features to obtain the text answer.

[0353] In an implementation manner of the embodiment of the present application, in response to the language processing task including an information extraction task, the encoding layer to be called includes an audio encoding layer in multiple domains, and the task execution result includes the target information in the input data; wherein, the target information includes keywords and / or entity words; the execution module 1202 is configured to: in response to the input data including the audio to be extracted, call the audio encoding layer in multiple domains to encode the audio to be extracted to obtain second audio features; call the contrast projection layer in the multimodal model to project the second audio features into the language space associated with the large language model to obtain second projection features; call the large language model to perform information extraction on the second projection features to obtain the target information.

[0354] In an implementation manner of the embodiment of the present application, the encoding layer to be called further includes a text encoding layer in multiple languages, and the execution module 1202 is further configured to: in response to the input data including the text to be extracted, call the text encoding layer in multiple languages to encode the text to be extracted to obtain encoding features; call the contrast projection layer in the multimodal model to project the encoding features into the language space associated with the large language model to obtain third projection features; call the large language model to perform information extraction on the third projection features to obtain the target information.

[0355] In an implementation manner of the embodiment of the present application, the task execution device 1200 may further include:

[0356] A training module, configured to: obtain a training set; wherein, the training set includes text-audio pairs in multiple domains and text-audio pairs in multiple languages, and each text-audio pair includes a sample audio and a sample text that are semantically related; use the audio encoding layer in the multimodal model to encode the sample audios in N text-audio pairs to obtain sample audio features of the N text-audio pairs; wherein, N is a set positive integer; use the text encoding layer in the multimodal model to encode the sample texts in the N text-audio pairs respectively to obtain sample text features of the N text-audio pairs; use the contrast projection layer in the multimodal model to project the sample audio features and sample text features corresponding to the N text-audio pairs into the same semantic space to obtain audio projection features and text projection features corresponding to the N text-audio pairs; based on the text projection features and audio projection features corresponding to the N text-audio pairs, perform contrastive pre-training on the multimodal model.

[0357] In an implementation manner of the embodiment of the present application, the training module is further configured to: obtain the target distance between the text projection feature of the i-th text-audio pair and the audio projection feature of the j-th text-audio pair; wherein, both i and j are positive integers less than or equal to N; obtain the value of the target flag bit; wherein, the target flag bit is used to indicate whether the sample text in the i-th text-audio pair and the sample audio in the j-th text-audio pair are a positive sample pair; determine the loss value according to the product of the target distance and the value of the target flag bit; based on the loss value, perform contrastive pre-training on the multimodal model.

[0358] In an implementation manner of the embodiment of the present application, the training module is further configured to: determine the initial distance between the text projection feature of the i-th text-audio pair and the audio projection feature of the j-th text-audio pair; and correct the initial distance according to the learnable parameters in the multimodal model to obtain the target distance.

[0359] In an implementation manner of the embodiment of the present application, the training module is further configured to: obtain text-audio pairs in multiple domains; translate the sample texts in the text-audio pairs to obtain enhanced texts in multiple languages; generate text-audio pairs in multiple languages according to the enhanced texts and the sample audios in the text-audio pairs; generate a training set according to the text-audio pairs in multiple domains and the text-audio pairs in multiple languages.

[0360] In the task execution device of the embodiment of the present application, by obtaining input data and a language processing task associated with the input data; calling a multimodal model to perform the language processing task on the input data to obtain a task execution result; wherein, the multimodal model is obtained by contrastive pre-training using text-audio pairs in multiple domains and multiple languages; presenting the task execution result. Thus, by calling the contrastively pre-trained multimodal model for task processing, the semantic relationship between text and audio can be understood more accurately, thereby providing a higher-quality task execution result. Among them, using multimodal, multi-domain, and multi-language text-audio pairs (including text and audio with highly related semantics) to perform contrastive pre-training (i.e., contrastive learning) on the multimodal model enables the model to learn general cross-modal representations, which not only improves the adaptability of the model in different domains and languages but also enhances its applicability in diverse application scenarios. Moreover, through contrastive pre-training, on the one hand, the model can perform well on unseen data or new tasks, especially in zero-shot or few-shot scenarios, and this characteristic enables the model to better handle complex tasks. On the other hand, unlabeled multimodal data (text-audio pairs) can be fully utilized to pre-train the model, significantly reducing the dependence on labeled data and reducing the cost of data collection and annotation.

[0361] To implement the above embodiment, the embodiment of the present application also proposes a training device for a multimodal model.

[0362] Figure 13 It is a schematic structural diagram of a training device for a multimodal model provided by the embodiment of the present application.

[0363] As Figure 13 shown, the training device 1300 for the multimodal model may include: an acquisition module 1301, a first encoding module 1302, a second encoding module 1303, a projection module 1304, and a pre-training module 1305.

[0364] Among them, the acquisition module 1301 is used to obtain a training set; wherein, the training set includes text-audio pairs in multiple domains and text-audio pairs in multiple languages, and the text-audio pairs include sample audio and sample text with related semantics.

[0365] The first encoding module 1302 is used to encode the sample audio in N text-audio pairs using the audio encoding layer in the multimodal model to obtain the sample audio features of N text-audio pairs; wherein, N is a set positive integer.

[0366] The second encoding module 1303 is used to encode the sample text in N text-audio pairs respectively using the text encoding layer in the multimodal model to obtain the sample text features of N text-audio pairs.

[0367] A projection module 1304 is configured to project the sample audio features and sample text features corresponding to N text-audio pairs into the same semantic space by using a contrast projection layer in a multimodal model, so as to obtain the audio projection features and text projection features corresponding to the N text-audio pairs.

[0368] A pre-training module 1305 is configured to perform contrastive pre-training on the multimodal model based on the text projection features and audio projection features corresponding to the N text-audio pairs.

[0369] It should be noted that the foregoing explanation of the embodiments of the training method for the multimodal model is also applicable to the training apparatus for the multimodal model in this embodiment, and will not be elaborated here.

[0370] In the training apparatus for the multimodal model according to the embodiments of the present application, a training set is obtained; wherein, the training set includes text-audio pairs in multiple domains and text-audio pairs in multiple languages, and the text-audio pairs include sample audio and sample text that are semantically related; the sample audio in the N text-audio pairs is encoded by using an audio encoding layer in the multimodal model to obtain the sample audio features of the N text-audio pairs; wherein, N is a set positive integer; the sample text in the N text-audio pairs is respectively encoded by using a text encoding layer in the multimodal model to obtain the sample text features of the N text-audio pairs; the sample audio features and sample text features corresponding to the N text-audio pairs are projected into the same semantic space by using a contrast projection layer in the multimodal model to obtain the audio projection features and text projection features corresponding to the N text-audio pairs; and the multimodal model is subjected to contrastive pre-training based on the text projection features and audio projection features corresponding to the N text-audio pairs. Thus, by using text-audio pairs (including text and audio with highly related semantics) in multiple modalities, multiple domains, and multiple languages to perform contrastive pre-training (i.e., contrastive learning) on the multimodal model, the model can learn general cross-modal representations, which not only improves the adaptability of the model in different domains and languages, but also enhances its applicability in diverse application scenarios. Moreover, through contrastive pre-training, on the one hand, the model can perform well on unseen data or new tasks, especially in zero-shot or few-shot scenarios, and this characteristic enables the model to better handle complex tasks. On the other hand, unlabeled multimodal data (text-audio pairs) can be fully utilized to pre-train the model, significantly reducing the dependence on labeled data and reducing the cost of data collection and annotation.

[0371] To implement the above embodiments, the present application further provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the task execution method or the training method for the multimodal model described in any of the foregoing embodiments is implemented.

[0372] Figure 14 This is a schematic structural diagram of an electronic device provided by an embodiment of the present application. For example, the electronic device 1400 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0373] Refer to Figure 14 , the electronic device 1400 may include one or more of the following components: a processing component 1402, a memory 1404, a power component 1406, a multimedia component 1408, an audio component 1410, an input / output (I / O) interface 1412, a sensor component 1414, and a communication component 1416.

[0374] The processing component 1402 generally controls the overall operation of the electronic device 1400, such as operations associated with display, telephone calls, data communication, camera operations, and recording operations. The processing component 1402 may include one or more processors 1420 to execute instructions to complete all or part of the steps of the above methods. In addition, the processing component 1402 may include one or more modules to facilitate the interaction between the processing component 1402 and other components. For example, the processing component 1402 may include a multimedia module to facilitate the interaction between the multimedia component 1408 and the processing component 1402.

[0375] The memory 1404 is configured to store various types of data to support the operation of the electronic device 1400. Examples of these data include instructions for any application or method operating on the electronic device 1400, contact data, phone book data, messages, pictures, videos, etc. The memory 1404 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, a magnetic disk, or an optical disc.

[0376] The power component 1406 provides power for various components of the electronic device 1400. The power component 1406 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power for the electronic device 1400.

[0377] The multimedia component 1408 includes a screen that provides an output interface between the electronic device 1400 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can not only sense the boundaries of the touch or swipe actions, but also detect the duration and pressure associated with the touch or swipe operations. In some embodiments, the multimedia component 1408 includes a front camera and / or a rear camera. When the electronic device 1400 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each of the front camera and the rear camera can be a fixed optical lens system or have a focal length and optical zoom capabilities.

[0378] The audio component 1410 is configured to output and / or input audio signals. For example, the audio component 1410 includes a microphone (MIC), which is configured to receive external audio signals when the electronic device 1400 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals can be further stored in the memory 1404 or transmitted via the communication component 1416. In some embodiments, the audio component 1410 further includes a speaker for outputting audio signals.

[0379] The I / O interface 1412 provides an interface between the processing component 1402 and a peripheral interface module, and the peripheral interface module can be a keyboard, a click wheel, buttons, etc. These buttons can include, but are not limited to: a home button, a volume button, a power button, and a lock button.

[0380] The sensor assembly 1414 includes one or more sensors for providing an assessment of various aspects of the state of the electronic device 1400. For example, the sensor assembly 1414 can detect the on / off state of the electronic device 1400, the relative positioning of components, such as the display and keypad of the electronic device 1400. The sensor assembly 1414 can also detect a change in the position of the electronic device 1400 or a component of the electronic device 1400, the presence or absence of user contact with the electronic device 1400, the orientation or acceleration / deceleration of the electronic device 1400, and the temperature change of the electronic device 1400. The sensor assembly 1414 can include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 1414 can also include a light sensor, such as a complementary metal-oxide-semiconductor (CMOS) or charge-coupled device (CCD) image sensor, for use in imaging applications. In some embodiments, the sensor assembly 1414 can also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0381] The communication assembly 1416 is configured to facilitate communication between the electronic device 1400 and other devices in a wired or wireless manner. The electronic device 1400 can access a wireless network based on communication standards, such as WiFi, 4G, or 5G, or a combination thereof. In an exemplary embodiment, the communication assembly 1416 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication assembly 1416 further includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0382] In an exemplary embodiment, the electronic device 1400 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components, and is used to execute the above method.

[0383] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is further provided, such as the memory 1404 including instructions. The above instructions can be executed by the processor 1420 of the terminal 1400 to complete the above method. For example, the non-transitory computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0384] To implement the above embodiments, the present application further proposes a non-transitory computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, it implements the task execution method or the training method of the multi-modal model as described in any of the foregoing method embodiments.

[0385] To implement the above embodiments, the present application further proposes a computer program product, on which a computer program is stored. When the computer program is executed by a processor, it implements the task execution method or the training method of the multi-modal model as described in any of the foregoing method embodiments.

[0386] In the description of this specification, the descriptions referring to terms such as "one embodiment", "some embodiments", "examples", "specific examples", or "some examples", etc. mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of this application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0387] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one of such features. In the description of this application, "a plurality of" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0388] Any process or method description shown in a flowchart or described in other ways herein may be understood to represent a module, segment, or portion of code including one or more executable instructions for implementing a customized logic function or process, and the scope of the preferred embodiments of this application includes additional implementations, where the functions may be executed in a manner not shown or discussed, including in a substantially simultaneous manner or in a reverse order according to the functions involved, which should be understood by those skilled in the art to which the embodiments of this application belong.

[0389] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a definite sequence list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device), or used in combination with these instruction execution systems, apparatus, or devices. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection part with one or more wirings (electronic device), a portable computer diskette case (magnetic device), a random access memory (Random Access Memory, abbreviated as RAM), a read-only memory (Read-Only Memory, abbreviated as ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (Compact Disc Read-Only Memory, abbreviated as CDROM). Additionally, the computer-readable medium can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpretation, or other appropriate processing when necessary, and then stored in a computer memory.

[0390] It should be understood that various parts of the present application can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits with logic gate circuits for implementing logical functions on data signals, application-specific integrated circuits with suitable combinational logic gate circuits, programmable gate arrays (Programmable Gate Array, abbreviated as PGA), field-programmable gate arrays (Field-Programmable Gate Array, abbreviated as FPGA), etc.

[0391] Those of ordinary skill in the art of this technology can understand that all or part of the steps carried by the method of implementing the above embodiments can be completed by a program instructing relevant hardware, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiments.

[0392] In addition, each functional unit in various embodiments of the present application may be integrated into a processing module, may exist separately as individual physical units, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. When the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0393] The above-mentioned storage medium may be a read-only memory, a magnetic disk, an optical disc, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present application. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application.

Claims

1. A task execution method, characterized in that, Including: Obtaining input data and a language processing task associated with the input data; Invoking a multimodal model to perform the language processing task on the input data to obtain a task execution result; wherein, the multimodal model is obtained by contrastive pre-training using text-audio pairs in multiple domains and multiple languages; Displaying the task execution result.

2. The method according to claim 1, characterized in that, The language processing task is indicated by the input data and determined according to the intention to which the input data belongs; or, the language processing task is indicated by the input object of the input data.

3. The method according to claim 1, characterized in that, The invoking the multimodal model to perform the language processing task on the input data to obtain a task execution result includes: Determining a to-be-invoked encoding layer from the multimodal model according to the language processing task; wherein, the to-be-invoked encoding layer includes at least one of a multilingual text encoding layer and a multi-domain audio encoding layer; Invoking the to-be-invoked encoding layer to perform the language processing task on the input data to obtain the task execution result.

4. The method according to claim 3, wherein In response to the language processing task including a retrieval task, the input data includes an input text, the to-be-invoked encoding layer includes the multilingual text encoding layer and the multi-domain audio encoding layer, and the task execution result includes a target resource semantically matching the input text, and the target resource includes an audio resource and / or a video resource; The invoking the to-be-invoked encoding layer to perform the language processing task on the input data to obtain the task execution result includes: Invoking the multilingual text encoding layer to encode the input text to obtain text features; Invoking a contrastive projection layer in the multimodal model to project the text features and multiple first audio features into the same semantic space to obtain multiple audio projection features and text projection features; wherein, the first audio features are obtained by invoking the multi-domain audio encoding layer to encode candidate resources; Retrieving the target resource from the candidate resources according to the similarity between each of the audio projection features and the text projection features.

5. The method according to claim 3, characterized in that, In response to the language processing task including an instruction understanding task, the input data includes an input voice instruction and multiple candidate texts associated with the voice instruction, the to-be-invoked encoding layer includes the multilingual text encoding layer and the multi-domain audio encoding layer, and the task execution result includes a target text semantically matching the voice instruction; The invoking the to-be-invoked encoding layer to perform the language processing task on the input data to obtain the task execution result includes: Invoking the multilingual text encoding layer to encode the multiple candidate texts to obtain text features of the multiple candidate texts; Invoking the multi-domain audio encoding layer to encode the voice instruction to obtain voice features; Invoking a contrastive projection layer in the multimodal model to project the text features of the multiple candidate texts and the voice features into the same semantic space to obtain text projection features and voice projection features of the multiple candidate texts; Determine the target text from the multiple candidate texts according to the similarity between the text projection features and the speech projection features of the multiple candidate texts.

6. The method according to claim 3, wherein In response to the language processing task including an audio classification task, the input data includes the input audio to be classified, the encoding layer to be called includes the audio encoding layer of multiple domains, and the task execution result includes the target category to which the audio to be classified belongs; The calling of the encoding layer to be called to execute the language processing task on the input data to obtain the task execution result includes: Calling the audio encoding layer of multiple domains to encode the audio to be classified to obtain second audio features; Calling the contrast projection layer in the multimodal model to project the second audio features into the language space associated with the large language model to obtain first projection features; Calling the large language model to classify the first projection features to obtain the target category.

7. The method according to claim 3, wherein In response to the language processing task including a question-and-answer task, the input data includes the input speech question, the encoding layer to be called includes the audio encoding layer of multiple domains, and the task execution result includes the text answer matching the speech question; The calling of the encoding layer to be called to execute the language processing task on the input data to obtain the task execution result includes: Calling the audio encoding layer of multiple domains to encode the speech question to obtain question features; Calling the contrast projection layer in the multimodal model to project the question features into the language space associated with the large language model to obtain the question projection features; Calling the large language model to process the question projection features to obtain the text answer.

8. The method according to claim 3, characterized in that In response to the language processing task including an information extraction task, the encoding layer to be called includes the audio encoding layer of multiple domains, and the task execution result includes the target information in the input data; wherein, the target information includes keywords and / or entity words; The calling of the encoding layer to be called to execute the language processing task on the input data to obtain the task execution result includes: In response to the input data including the audio to be extracted, calling the audio encoding layer of multiple domains to encode the audio to be extracted to obtain second audio features; Calling the contrast projection layer in the multimodal model to project the second audio features into the language space associated with the large language model to obtain second projection features; Calling the large language model to perform information extraction on the second projection features to obtain the target information.

9. The method according to claim 8, wherein The encoding layer to be called further includes the text encoding layer of multiple languages, and the calling of the encoding layer to be called to execute the language processing task on the input data to obtain the task execution result further includes: In response to the input data including the text to be extracted, calling the text encoding layer of multiple languages to encode the text to be extracted to obtain encoding features; Calling the contrast projection layer in the multimodal model to project the encoding features into the language space associated with the large language model to obtain third projection features; Call the large language model to extract information from the third projection feature to obtain the target information.

10. The method according to any one of claims 1-9, characterized in that The multi-modal model is pre-trained in the following manner: Obtain a training set; wherein, the training set includes text-audio pairs in multiple domains and text-audio pairs in multiple languages, and the text-audio pairs include sample audio and sample text that are semantically related; Use the audio encoding layer in the multi-modal model to encode the sample audio in N text-audio pairs to obtain the sample audio features of the N text-audio pairs; where N is a set positive integer; Use the text encoding layer in the multi-modal model to encode the sample text in the N text-audio pairs respectively to obtain the sample text features of the N text-audio pairs; Use the contrast projection layer in the multi-modal model to project the sample audio features and sample text features corresponding to the N text-audio pairs into the same semantic space to obtain the audio projection features and text projection features corresponding to the N text-audio pairs; Based on the text projection features and audio projection features corresponding to the N text-audio pairs, perform contrastive pre-training on the multi-modal model.

11. The method according to claim 10, characterized in that, The performing contrastive pre-training on the multi-modal model based on the text projection features and audio projection features corresponding to the N text-audio pairs includes: Obtain the target distance between the text projection feature of the i-th text-audio pair and the audio projection feature of the j-th text-audio pair; where both i and j are positive integers less than or equal to N; Obtain the value of the target flag bit; wherein, the target flag bit is used to indicate whether the sample text in the i-th text-audio pair and the sample audio in the j-th text-audio pair are a positive sample pair; Determine the loss value according to the product between the target distance and the value of the target flag bit; Based on the loss value, perform contrastive pre-training on the multi-modal model.

12. The method according to claim 11, wherein The obtaining the target distance between the text projection feature of the i-th text-audio pair and the audio projection feature of the j-th text-audio pair includes: Determine the initial distance between the text projection feature of the i-th text-audio pair and the audio projection feature of the j-th text-audio pair; According to the learnable parameters in the multi-modal model, correct the initial distance to obtain the target distance.

13. The method according to claim 10, wherein The obtaining the training set includes: Obtain text-audio pairs in multiple domains; Translate the sample text in the text-audio pairs to obtain enhanced text in multiple languages; Generate text-audio pairs in multiple languages according to the enhanced text and the sample audio in the text-audio pairs; Generate the training set according to the text-audio pairs in multiple domains and the text-audio pairs in multiple languages.

14. A training method for a multimodal model, characterized in that, Includes: Obtain a training set; wherein, the training set includes text-audio pairs in multiple domains and text-audio pairs in multiple languages, and the text-audio pairs include sample audio and sample text that are semantically related; Use the audio encoding layer in the multi-modal model to encode the sample audio in N text-audio pairs to obtain the sample audio features of the N text-audio pairs; where N is a set positive integer; Use the text encoding layer in the multi-modal model to encode the sample texts in the N text-audio pairs respectively, to obtain the sample text features of the N text-audio pairs; Use the contrast projection layer in the multi-modal model to project the sample audio features and sample text features corresponding to the N text-audio pairs into the same semantic space, to obtain the audio projection features and text projection features corresponding to the N text-audio pairs; Based on the text projection features and audio projection features corresponding to the N text-audio pairs, perform contrast pre-training on the multi-modal model.

15. A task execution device, characterized in that, For implementing the task execution method according to any one of claims 1-13, the device includes: An acquisition module, configured to acquire input data and a language processing task associated with the input data; An execution module, configured to call a multi-modal model to execute the language processing task on the input data, to obtain a task execution result; wherein, the multi-modal model is obtained by performing contrast pre-training using text-audio pairs in multiple domains and multiple languages; A display module, configured to display the task execution result.

16. A training device for a multimodal model, characterized in that, Comprising: An acquisition module, configured to acquire a training set; wherein, the training set includes text-audio pairs in multiple domains and text-audio pairs in multiple languages, and the text-audio pairs include sample audio and sample text that are semantically related; A first encoding module, configured to use the audio encoding layer in the multi-modal model to encode the sample audio in N text-audio pairs, to obtain the sample audio features of the N text-audio pairs; wherein, N is a set positive integer; A second encoding module, configured to use the text encoding layer in the multi-modal model to encode the sample texts in the N text-audio pairs respectively, to obtain the sample text features of the N text-audio pairs; A projection module, configured to use the contrast projection layer in the multi-modal model to project the sample audio features and sample text features corresponding to the N text-audio pairs into the same semantic space, to obtain the audio projection features and text projection features corresponding to the N text-audio pairs; A pre-training module, configured to perform contrast pre-training on the multi-modal model based on the text projection features and audio projection features corresponding to the N text-audio pairs.

17. An electronic device, characterized in that, Comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, when the processor executes the program, implementing the steps of the method according to any one of claims 1 to 13, and / or, implementing the steps of the method according to claim 14.

18. A non - transitory computer - readable storage medium having computer program instructions stored thereon, characterized in that, When the program instructions are executed by the processor, implementing the steps of the method according to any one of claims 1 to 13, and / or, implementing the steps of the method according to claim 14.

19. A computer program product, characterized in that, Comprising a computer program, when the computer program is executed by the processor, implementing the steps of the method according to any one of claims 1 to 13, and / or, implementing the steps of the method according to claim 14.

Images