Data processing method and cloud server

Abstract

The present application provides a data processing method and a cloud server. The method of the present application comprises: adding a natural language processing module on a pre-trained model to form a pre-trained natural language processing model; on the basis of the pre-trained natural language processing model, constructing a fine-tuning model architecture of a multi-branch model; and on the basis of the fine-tuning model architecture of the multi-branch model, performing fine-tuning training on the pre-trained natural language processing model to obtain a trained natural language processing model. By constructing and using a fine-tuning model architecture of a multi-branch model, the multi-branch model can be used to simultaneously process multiple pieces of data, such that the training data processing efficiency can be improved, thereby improving the efficiency of obtaining the natural language processing model.

Description

Data processing method and cloud server

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on August 22, 2023, with application number 202311069985.9 and application name “Data Processing Method and Cloud Server”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to computer technology, and in particular to a data processing method and a cloud server. Background Art

[0003] As a fundamental capability, natural language processing is widely used in various fields, including information extraction, text summarization, question answering, and speech recognition. High-quality natural language processing is crucial to the effectiveness of downstream tasks.

[0004] Currently, large pre-trained models, as a general-purpose model, have significant differences between their training objectives during the pre-training phase and those of natural language processing models. When applied to downstream natural language processing tasks, the pre-trained model must be fine-tuned using the downstream task's dataset to obtain the natural language processing model. Traditional fine-tuning methods sequentially feed the pre-trained model with training data from the dataset. This results in inefficient data processing, resulting in a long and inefficient timeframe for obtaining a natural language processing model.

[0005] Summary of the Invention

[0006] The present application provides a data processing method and a cloud server to solve the problem of long cycle and low efficiency in obtaining a natural language processing model.

[0007] In a first aspect, the present application provides a data processing method, comprising:

[0008] Add a natural language processing module to the pre-trained model to form a pre-trained natural language processing model;

[0009] Building a fine-tuning model architecture of a multi-branch model based on the pre-trained natural language processing model;

[0010] Based on the fine-tuning model architecture of the multi-branch model, fine-tuning training of the pre-trained natural language processing model is performed to obtain a trained natural language processing model.

[0011] In a second aspect, the present application provides a cloud server comprising: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method described in the first aspect above.

[0012] The data processing method and cloud server provided in the present application form a pre-trained natural language processing model by adding a natural language processing module to the pre-trained model; based on the pre-trained natural language processing model, a fine-tuning model architecture of a multi-branch model is constructed; based on the fine-tuning model architecture of the multi-branch model, fine-tuning training of the pre-trained natural language processing model is performed to obtain a trained natural language processing model. By constructing and using the fine-tuning model architecture of the multi-branch model, the multi-branch model can be used to perform multiple data processing at the same time, which can improve the efficiency of training data processing, thereby improving the efficiency of obtaining the natural language processing model. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0014] FIG1 is a schematic diagram of an exemplary system architecture applicable to the present application;

[0015] FIG2 is a flow chart of a data processing method provided by an exemplary embodiment of the present application;

[0016] FIG3 is a flow chart of a method for natural language processing provided by an exemplary embodiment of the present application;

[0017] FIG4 is a schematic diagram of a system architecture of a text representation model provided by an exemplary embodiment of the present application;

[0018] FIG5 is a flow chart of a multilingual text processing method provided by an exemplary embodiment of the present application;

[0019] FIG6 is a schematic diagram of a fine-tuning model architecture of a dual-branch model provided by an exemplary embodiment of the present application;

[0020] FIG7 is a flowchart of a method for representing multilingual text according to an exemplary embodiment of the present application;

[0021] FIG8 is a flow chart of a multilingual text processing method provided by another exemplary embodiment of the present application;

[0022] FIG9 is a flow chart of a cross-language text retrieval method provided by an exemplary embodiment of the present application;

[0023] FIG10 is a schematic structural diagram of a cloud server provided by an exemplary embodiment of the present application.

[0024] The above drawings illustrate specific embodiments of the present application, which will be described in more detail below. These drawings and the textual description are not intended to limit the scope of the present application in any way, but rather to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0025] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0026] First, let’s explain the terms involved in this application:

[0027] Multilingualism: refers to supporting multiple languages ​​in a product, service, application or website to adapt to the needs of users in different languages.

[0028] Cross Language Information Retrieval (CLIR) refers to retrieving relevant information described in another language or multiple languages ​​based on query information written in one language. For example, if a Chinese search term is entered, the cross-language retrieval system will return information described in languages ​​such as English and Japanese. When applied to text information retrieval, it is also called cross-language text retrieval or cross-language text information retrieval. In cross-language information retrieval, the language used by the user when asking the question is usually called the source language, which is generally the user's native language; the language used in the retrieved information is called the target language, which can be a language that the user is unfamiliar with or even completely unfamiliar with.

[0029] Text embedding, also known as text vector representation, is a method for converting text into a vector form. This method mathematically encodes the text, converting it into a set of continuous values ​​to form a vector representation.

[0030] The Transformer is an attention-based neural network model for natural language processing tasks. It consists of a multi-layer encoder and decoder. The encoder transforms the input sequence into a series of hidden states and uses an attention mechanism to focus on different parts of the input sequence. The decoder uses a similar mechanism, but also employs a self-attention mechanism to ensure the smoothness and consistency of the output sequence.

[0031] Contrastive Learning: A machine learning method for learning similarities and differences. This method learns similarity representations by aggregating samples of the same type and distinguishing samples of different types.

[0032] BERT: The full name is Bidirectional Encoder Representations from Transformer. It is a bidirectional encoder representation model based on Transformer and a pre-trained language representation model.

[0033] Large Language Model (LLM): Also known as a large-scale language model or LLM, this refers to a language model with a large number of model parameters, typically containing hundreds of millions, tens of billions, or even hundreds of billions. These models typically consist of multi-layer neural networks, whose input is a text sequence and whose output is a linguistic representation of that text sequence.

[0034] Large-scale pre-trained language models, also known as pre-trained large language models, are pre-trained on large corpora and are used for natural language processing tasks. Pre-training means that the model has been trained and learned to process large amounts of language data before a specific task. This pre-training allows the model to capture more complex linguistic and semantic rules, resulting in excellent performance in various natural language processing tasks and reducing the large-scale data requirements for specific tasks.

[0035] Visual question answering task: Given an input image and a question, determine the answer to the question from the visual information of the input image.

[0036] Image description task: Generate description text for the input image.

[0037] Visual entailment task: predict the semantic relevance of an input image and text, i.e., entailment, neutrality, or contradiction.

[0038] Referential expression and comprehension task: locate the image area corresponding to the input text in the input image based on the input text.

[0039] Image generation task: Generate an image based on the input description text.

[0040] Text-based sentiment classification task: predict the sentiment classification information of the input text.

[0041] Text summarization task: Generate summary information of the input text.

[0042] Multimodal tasks: refers to downstream tasks whose input and output data involve multiple modal data such as images and text, such as visual question answering tasks, image description tasks, visual implication tasks, referential expression and understanding tasks, image generation tasks, etc.

[0043] Multimodal pre-trained model: refers to a pre-trained model whose input and output data involve multiple modal data such as images and text. After fine-tuning and training, it can be applied to multimodal task processing.

[0044] Large models refer to deep learning models with large-scale model parameters, typically containing hundreds of millions, tens of billions, or even hundreds of billions of model parameters. Large models, also known as foundation models, are pre-trained on large-scale unlabeled corpora, producing pre-trained models with parameters exceeding 100 million. Such models are adaptable to a wide range of downstream tasks and exhibit good generalization capabilities. Examples include large-scale language models (LLMs), large-scale pre-trained language models, and multi-modal pre-training models.

[0045] When large models are used in practice, only a small number of samples are needed to fine-tune the pre-trained model and it can be applied to different tasks. Large models can be widely used in natural language processing (NLP), computer vision and other fields. Specifically, they can be applied to computer vision tasks such as visual question answering (VQA), image caption (IC), and image generation, as well as natural language processing tasks such as text-based sentiment classification, text summary generation, and machine translation. The main application scenarios of large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.

[0046] Natural language processing, as a basic capability, is widely used in various fields such as information extraction, text summarization, question answering, and speech recognition. High-quality natural language processing is crucial to the effectiveness of downstream tasks. At present, as a general-purpose model, the pre-trained large model has a great difference between the training objectives in the pre-training stage and the objectives of the natural language processing model. When applied to downstream natural language processing tasks, the pre-trained model needs to be fine-tuned using the data set of the downstream task to obtain the natural language processing model. In the traditional fine-tuning method, the training data in the data set is input into the pre-training model for processing in sequence. The efficiency of data processing is low, resulting in a long cycle and low efficiency for obtaining the natural language processing model. The present application provides a data processing method, which forms a pre-trained natural language processing model by adding a natural language processing module to the pre-trained model; constructs a fine-tuning model architecture of a multi-branch model based on the pre-trained natural language processing model; and performs fine-tuning training on the pre-trained natural language processing model based on the fine-tuning model architecture of the multi-branch model to obtain a trained natural language processing model. By building and using a fine-tuning model architecture of a multi-branch model, the multi-branch model can be used to perform multiple data processing at the same time, which can improve the efficiency of training data processing and thus improve the efficiency of obtaining a natural language processing model.

[0047] Among them, the pre-trained model used to train the natural language processing model can be a pre-trained language model, or a large-scale pre-trained language model, or a multimodal pre-trained model, or a model with natural language processing capabilities built based on Transformer, which is not specifically limited here.

[0048] The natural language processing model obtained in this solution can be applied to any of the following natural language processing tasks: text representation, information extraction, text summarization, question answering, and speech recognition. In downstream natural language processing tasks, the natural language processing model is used to perform natural language processing on the input information and obtain natural language processing results.

[0049] In addition, in some usage scenarios, the natural language processing model obtained can also be used as a base model for downstream task models. Based on the natural language processing capabilities of the base model, a target task model for performing specific downstream tasks is constructed. For example, the natural language processing model obtained can be a multilingual text representation model. By using the text representation model as the base model and superimposing a classifier after the last layer of the base model, a multilingual text classification model can be obtained. The text representation model serves as a text representation module in the multilingual text classification model. In the multilingual text classification task, the text representation of the input text in various languages ​​is obtained through the text representation module (i.e., the text representation model) in the multilingual text classification model, and the classifier is used to perform classification prediction based on the text representation of the text in various languages ​​to obtain the classification results of the text in various languages. Based on the base model of the multilingual text representation model, models for other downstream tasks such as cross-language text retrieval (CLIR), cross-language question answering (CLQA), multilingual text clustering, and multimodal tasks can also be constructed.

[0050] Figure 1 is a schematic diagram of an example system architecture applicable to this application. As shown in Figure 1, the system architecture includes a server and a terminal device. A communication link is provided between the server and the terminal device, enabling communication between the server and the terminal device.

[0051] The server is a computing device deployed in the cloud or locally, such as a cloud cluster. It stores pre-trained models and training data for fine-tuning. The server is responsible for adding a natural language processing module to the pre-trained model to form a pre-trained natural language processing model. Based on the pre-trained natural language processing model, the server constructs a fine-tuning model architecture for the multi-branch model. Based on the fine-tuning model architecture for the multi-branch model, the server fine-tunes the pre-trained natural language processing model to obtain a trained natural language processing model.

[0052] End-side devices can be electronic devices that run downstream task systems. Specifically, they can be hardware devices with network communication, computing, and information display capabilities. These include, but are not limited to, smartphones, tablets, desktop computers, local servers, and cloud servers. When running downstream task systems, end-side devices require the natural language processing capabilities of natural language processing models.

[0053] Based on the system architecture shown in Figure 1, a user sends a call request to the server through a client device, requesting a natural language processing model. This call request contains the information to be processed. The server receives the call request, obtains the information to be processed, and inputs it into the natural language processing model for natural language processing. The model then generates a natural language processing result and returns it to the client device. The client device receives the natural language processing result from the server and, based on the result, executes the processing logic of the downstream task, completing the downstream processing task.

[0054] In addition, the server can also provide the natural language processing model (or the model parameters of the natural language processing model) to the end-side device. The end-side device can use the natural language processing model locally and implement specific downstream tasks based on the natural language processing capabilities of the natural language processing model, such as information extraction, text summarization, question answering, speech recognition, etc. Of course, the natural language processing model can also be applied to other tasks in the field of natural language processing and artificial intelligence, which are not specifically limited here.

[0055] The following specific embodiments describe in detail the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0056] FIG2 is a flow chart of a data processing method provided by an exemplary embodiment of the present application. The execution subject of this embodiment is the server in the aforementioned system architecture. As shown in FIG2, the specific steps of the method are as follows:

[0057] Step S201: Add a natural language processing module to the pre-trained model to form a pre-trained natural language processing model.

[0058] In this embodiment, to obtain a natural language processing model for downstream tasks, a pre-trained natural language processing model is first constructed by adding a natural language processing module to the pre-trained model. The added natural language processing model is used to perform natural language processing based on the features output by the pre-trained model and output the natural language processing results.

[0059] The pre-trained model can be a pre-trained language model, a large-scale pre-trained language model, a multimodal pre-trained model, or a Transformer-based model with natural language processing capabilities, without specific limitations here. The pre-trained model used to implement different natural language processing tasks can vary, and can be selected and configured based on the natural language processing requirements of the downstream tasks in the actual application, without specific limitations here.

[0060] Exemplarily, the natural language processing model to be obtained can be a text representation model. The pre-trained model used can be a pre-trained language model or a large-scale pre-trained language model, and a pre-trained text representation model is constructed by adding a text representation module to the pre-trained language model or the large-scale pre-trained language model. For example, an average pooling layer can be superimposed after the output layer of the last layer of the transformer structure of the pre-trained language model to obtain a pre-trained text representation model. When performing text representation, the text is input into the pre-trained language model, and after the average pooling operation is performed on the representation of the output layer of the last layer of the transformer structure of the pre-trained language model, a text representation of the input text is obtained. The added average pooling layer is used to output the text representation as a text representation module.

[0061] Exemplarily, the natural language processing model to be obtained can be an information extraction model (such as a named entity recognition model, a relationship extraction model). The pre-trained model used can be a pre-trained language model or a large-scale pre-trained language model, and a pre-trained information extraction model is constructed by adding an information extraction module to the pre-trained language model or the large-scale pre-trained language model. For example, a conditional random field (CRF) can be superimposed after the output layer of the last transformer structure of the pre-trained language model to obtain a pre-trained named entity recognition model. When performing named entity recognition, the information to be processed is input into the pre-trained language model, and named entity recognition is performed on the representation of the output layer of the last transformer structure of the pre-trained language model through the conditional random field CRF to obtain a named entity recognition result. For another example, a classifier can be superimposed after the output layer of the last transformer structure of the pre-trained language model to obtain a pre-trained relationship extraction model. When performing relationship extraction, the position encoding of the input character sequence and the two text fragments whose relationship categories are to be determined are input into the pre-trained language model, and relationship classification and recognition are performed on the representation of the output layer of the last layer of the transformer structure of the pre-trained language model to obtain the relationship extraction result.

[0062] Step S202: Based on the pre-trained natural language processing model, a fine-tuning model architecture of the multi-branch model is constructed.

[0063] After obtaining the pre-trained natural language processing model, a fine-tuning model architecture for the multi-branch model is constructed. The fine-tuning model architecture for the multi-branch model includes multiple parallel branch models, at least one of which uses the constructed pre-trained natural language processing model. The fine-tuning model architecture constructed in this embodiment is used to achieve rapid fine-tuning of the pre-trained natural language processing model.

[0064] Step S203: Based on the fine-tuning model architecture of the multi-branch model, fine-tune the pre-trained natural language processing model to obtain a trained natural language processing model.

[0065] In one example scenario, in step S202, a pre-trained natural language processing model is used as a branch model to construct a fine-tuning model architecture for a multi-branch model. The fine-tuning model architecture for the multi-branch model includes multiple parallel branch models. Each of the multiple parallel branch models uses the pre-trained natural language processing model and shares parameters during training. After training, the parameters of each branch model are consistent.

[0066] For example, the training set contains multiple training data, each of which includes multiple information samples. When fine-tuning the pre-trained natural language processing model, it is necessary to obtain the processing results of multiple information samples in a piece of training data. By comparing the processing results of multiple information samples in a piece of training data, the parameters of each branch model are updated. After training is completed, the branch model of the pre-trained natural language processing model will be used as the trained natural language processing model. In this case, a fine-tuning model architecture with a corresponding number of branch models can be constructed based on the number of information samples contained in a piece of training data.

[0067] During the fine-tuning training process of step S203, multiple information samples in a piece of training data are input into each branch model respectively. Natural language processing is performed on the input information samples simultaneously by each branch model to obtain natural language processing results for the multiple information samples. Furthermore, the parameters of each branch model are updated based on the natural language processing results of the multiple information samples. After the training is completed, the branch model of the pre-trained natural language processing model is used as the trained natural language processing model. By using multiple branch models of the fine-tuning model architecture of the multi-branch model to simultaneously process multiple information samples, the data processing efficiency during fine-tuning training can be improved, thereby improving the efficiency of obtaining a trained natural language processing model.

[0068] In another example scenario, in step S202, the constructed pre-trained natural language processing model is used as a branch model, and another (or multiple) processing modules are added to the pre-trained model to form another (or multiple) processing model as other branch models, and a fine-tuning model architecture of the multi-branch model is constructed.

[0069] Exemplarily, the training data in the training set includes samples of various types, including but not limited to language, text, and images. The pre-trained natural language processing model constructed is used to perform natural language processing on one type of sample. The constructed pre-trained natural language processing model is used as a branch model. For other types of samples, auxiliary branch models for processing other different types of samples are constructed based on the same pre-trained model, and a fine-tuning model architecture of the multi-branch model is constructed. Some shared parameters of the pre-trained model in each branch model in the fine-tuning model architecture are used.

[0070] During fine-tuning training in step S203, different types of samples in the training data are input into corresponding branch models. Each type of sample is processed by the multi-branch model to obtain corresponding processing results. Based on the processing results of the training data including multiple different types of samples, the parameters of each branch model are optimized. After training is completed, the branch model corresponding to the pre-trained natural language processing model is used as the trained pre-trained natural language processing model.

[0071] For example, the training data in the training set includes speech and the corresponding annotated text. To obtain a speech representation model, a speech representation module can be added to the pre-trained model to form a pre-trained speech representation model. To quickly fine-tune the pre-trained speech representation model, a text representation module can be added to the pre-trained model to form a text representation model for fine-tuning training. The pre-trained speech representation model is used as one branch model, and the text representation model as another branch model to construct a two-branch fine-tuning model architecture. These two branches share some parameters of the pre-trained model.

[0072] During fine-tuning training in step S203, the speech in the training data is input into a branch model corresponding to the pre-trained speech representation model, and the annotated text corresponding to the speech is input into another branch model. The two branch models are used to obtain feature representations of the speech and the annotated text, respectively. The parameters of the two branch models are optimized based on the similarity between the feature representations of the speech and the annotated text. After training is complete, the branch model corresponding to the pre-trained speech representation model is used as the trained speech representation model. This also yields a high-performance text representation model.

[0073] By using multiple branch models of the multi-branch model fine-tuning model architecture to simultaneously process multiple different types of samples in the training data, the data processing efficiency during fine-tuning training can be improved, thereby improving the efficiency of obtaining a trained natural language processing model.

[0074] The method of this embodiment forms a pre-trained natural language processing model by adding a natural language processing module to the pre-trained model; constructs a fine-tuning model architecture of a multi-branch model based on the pre-trained natural language processing model; and performs fine-tuning training on the pre-trained natural language processing model based on the fine-tuning model architecture of the multi-branch model to obtain a trained natural language processing model. By constructing and using the fine-tuning model architecture of the multi-branch model, the multi-branch model can be used to perform multiple data processing at the same time, which can improve the efficiency of training data processing, thereby improving the efficiency of obtaining the natural language processing model.

[0075] In an optional embodiment, after obtaining the natural language processing model in the aforementioned embodiment, the cloud server can also provide a calling interface of the natural language processing model to the external end-side device based on the natural language processing model, and implement the natural language processing function based on the natural language processing capability of the natural language processing model. As shown in Figure 3, the process of implementing natural language processing based on the natural language processing model is as follows:

[0076] Step S301: A call request from a receiving device to a natural language processing model, where the call request includes information to be processed.

[0077] Based on the natural language processing model call interface provided by the cloud server, the client device sends a call request to the cloud server when natural language processing is needed. The call request also carries the information to be processed and sends it to the cloud server.

[0078] Step S302: Input the information to be processed into a natural language processing model for natural language processing to obtain a natural language processing result.

[0079] The cloud server receives a request from the client device for the natural language processing model and obtains the information to be processed contained in the request. The cloud server inputs the information to be processed into the natural language processing model, which then performs natural language processing on the information to obtain the natural language processing results.

[0080] Step S303: Send the natural language processing result to the end-side device.

[0081] The cloud server returns the natural language processing results to the end-device. The end-device receives the natural language processing results returned by the cloud server and executes the subsequent processing logic of the downstream tasks based on the natural language processing results.

[0082] Through the solution of this embodiment, various natural language processing tasks such as text representation, speech recognition, text summarization, and question answering can be implemented based on the natural language processing capabilities of the natural language processing model. Of course, the natural language processing model can also be applied to other tasks in the natural language processing and artificial intelligence fields, which are not specifically limited here.

[0083] The method of this embodiment applies the natural language processing model to various downstream processing tasks. Based on the powerful representation ability of the natural language processing model, it can improve the quality of natural language processing, thereby improving the accuracy and effectiveness of downstream text processing tasks.

[0084] In practical applications, text representation, as a fundamental capability in natural language processing, is widely used in downstream tasks such as text classification, clustering, retrieval, question answering, and recommendation. High-quality text representation is crucial to the effectiveness of downstream tasks.

[0085] In an optional embodiment, the method of the aforementioned embodiment of the present application can be applied to obtain a text representation model. Specifically, in step S201, a text representation module is added to the pre-trained model to form a pre-trained text representation model. In step S202, the constructed pre-trained text representation model is used as a branch model to construct a fine-tuning model architecture of a dual-branch model. The fine-tuning model architecture of the dual-branch model includes two parallel branch models. The two branch models have the same model structure and share parameters.

[0086] Furthermore, in step S203, text data for fine-tuning training is obtained, and the text data includes multiple related text pairs, and any related text pair includes two content-related texts; based on the fine-tuning model architecture of the dual-branch model, the two texts in each related text pair are respectively input into the two branch models of the fine-tuning model architecture of the dual-branch model for text representation, and the text representation of the two texts in each related text pair is obtained; according to the text representation of the two texts in each related text pair, the branch model is subjected to comparative learning training, and the parameters of the two branch models are updated. After the training is completed, any branch model is used as the trained text representation model.

[0087] In practical downstream text processing tasks, processing text information in multiple languages ​​is often required. Examples include multilingual text classification and clustering, cross-language text retrieval, cross-language question-answering, and recommender systems. This requires text representation for text information in multiple languages. The traditional solution is to customize a text representation model for each language based on the domain of the downstream task and the required language type, which is time-consuming and resource-intensive. Recently, multilingual pre-trained large models have emerged that can process text information in multiple languages. However, as a general-purpose model, the training objectives of the pre-training phase differ significantly from those of the text representation model, resulting in poor text representation capabilities. Using multilingual pre-trained large models as text representation models for downstream tasks results in low text representation quality, which in turn leads to low accuracy and poor performance in downstream text processing tasks.

[0088] In an optional embodiment, the multilingual pre-trained large model is fine-tuned based on the method of the aforementioned embodiment to obtain a multilingual text representation model with powerful multilingual text representation capabilities.

[0089] Figure 4 is a schematic diagram of an example system architecture of the multilingual text representation model provided by this application. As shown in Figure 4, the system architecture includes a cloud server and one or more application systems.

[0090] Among them, the cloud server is a device with computing capabilities deployed in the cloud, such as a cloud cluster. The cloud server is used to obtain and store text data in multiple different language types. The cloud server can use text data in multiple different language types to perform comparative learning and training on the multilingual pre-trained large model to obtain a text representation model suitable for multiple languages. Furthermore, the cloud server can provide multilingual text representation services to the outside world based on the obtained text representation model; or, the cloud server can also provide the text representation model or the model parameters of the text representation model to the end-side device, and the end-side device can obtain the text representation model, or initialize the parameters of the multilingual pre-trained large model based on the model parameters of the obtained text representation model to obtain a text representation model suitable for multiple languages.

[0091] An application system refers to an application system that is implemented based on the multilingual text representation capability applicable to a text representation model, including but not limited to a cross-language text retrieval system, a cross-language question-answering system, a machine translation system, a multilingual text classification system, and a multilingual text clustering system. An application system may include one or more computing devices, which execute the application system processing logic to implement multilingual text processing tasks. Relative to the cloud server, the computing device of the application system is an end-side device that uses the multilingual text representation capability provided by the cloud server. There is a communicative communication link between the cloud server and the end-side device where the application system is located, which can realize the communication connection between the cloud server and the application system (end-side device). The end-side device can be an electronic device that runs a downstream task system, specifically a hardware device with network communication function, computing function and information display function, including but not limited to smartphones, tablet computers, desktop computers, local cloud servers, cloud servers, etc.

[0092] When the end-side device needs to obtain the text representation of text information in various languages ​​while executing the processing logic of the application system and implementing multilingual text processing tasks, it sends a call request for the text representation model to the cloud server. The call request contains text information of each language type. The cloud server receives the call request, obtains the text information of each language type, inputs the text information of each language type into a unified text representation model for encoding, obtains the text representation of the text information of each language type, and returns the text representation of the text information of each language type to the end-side device. The end-side device receives the text representation of the text information of each language type returned by the cloud server, and continues to execute the processing logic of the downstream task based on the text representation of the text information of each language type to implement the downstream multilingual text processing task.

[0093] Furthermore, the cloud server can also provide the text representation model (or its parameters) to the client-side device. The client-side device can then use the text representation model locally and implement specific downstream tasks based on the multilingual text representation capabilities of the text representation model, such as clustering and classification based on multilingual text, cross-language text retrieval, and cross-language question-answering. Of course, the text representation model can also be applied to other tasks in the fields of natural language processing and artificial intelligence, which are not specifically limited here.

[0094] FIG5 is a flowchart of a multilingual text processing method provided by an exemplary embodiment of the present application. The execution subject of this embodiment is the cloud server in the system architecture shown in FIG4 . As shown in FIG5 , the specific steps of the method are as follows:

[0095] Step S501: Add a text representation module to the multilingual pre-trained large model to form a pre-trained text representation model.

[0096] The multilingual pre-trained large model is obtained by pre-training an initial large model using pre-training data from multiple languages. The initial large model used to train the text representation model can be any existing open-source LLM, such as a large-scale pre-trained language model, or a large model built on the Transformer, without specific limitations.

[0097] In this step, a pre-trained text representation model is obtained by adding an average pooling layer after the output layer of the last transformer structure of the multilingual pre-trained large model. When performing text representation, the text is input into the multilingual pre-trained large model, and after performing an average pooling operation on the representation of the output layer of the last transformer structure of the multilingual pre-trained large model, a text representation of the input text is obtained. In this embodiment, a large model with stronger versatility, stronger text representation capability, and support for ultra-long text input is selected as the initial model, so that the trained multilingual text representation model also has strong versatility, multilingual text expression capability, and support for ultra-long text input.

[0098] The pre-training data used in the pre-training process of the multi-language pre-training large model covers data of multiple different language types, which can be obtained by collecting pre-training data of multiple different language types used in the pre-training stage of the large model. The multiple different language types include, but are not limited to: different languages ​​of human natural language and various programming languages. For the natural language text in the pre-training data, its language type is the language used in the text, such as Chinese, English, etc. In addition, the pre-training data can include code data, and the language type of the code is the programming language used, such as C language, Python, Java, etc.

[0099] In this embodiment, text data of multiple different language types are used to perform comparative learning training on the pre-trained text representation model to obtain a trained text representation model. The trained text representation model is used to perform text representation on text information of multiple different language types and can be used as a text representation model that is common to multiple languages.

[0100] Step S502: Use the pre-trained text representation model as a branch model to construct a fine-tuning model architecture of a dual-branch model. The fine-tuning model architecture of the dual-branch model includes two parallel branch models.

[0101] In this embodiment, considering that fine-tuning training uses a contrastive learning method based on positive and negative sample pairs, a sample pair (a sample and its positive sample, or a sample and its negative sample) includes two texts. Therefore, a fine-tuning model architecture of a dual-branch model is constructed, in which two parallel branch models both use the constructed pre-trained text representation model. The two parallel branch models have the same model structure and share parameters.

[0102] For example, the fine-tuning model architecture of the dual-branch model is shown in Figure 6, which includes two branch models. Both branch models are pre-trained text representation models constructed by adding a text representation module to a multilingual pre-trained large model. Based on the fine-tuning model architecture of the dual-branch model, the input of the two branch models is the two texts in the sample pair (represented as text 1 and text 2), and the output is the text representation of the input text. Based on the correlation between the text representations of the two texts in the sample pair, the parameters of the pre-trained text representation model are optimized to implement fine-tuning training and obtain a trained text representation model.

[0103] Step S503: Acquire text data in multiple different language types.

[0104] Among them, the text data of multiple different language types include multiple related text pairs, and any related text pair includes two texts with related contents, namely a first text and a second text related to the content of the first text. Any related text pair can be represented as s i Indicates the first text, Indicates that i A content-related second text.

[0105] In an optional embodiment, the plurality of text data of different language types may include single language related text pairs and cross-language related text pairs, wherein the single language related text pairs include two texts of the same language type, and the cross-language related text pairs include two texts of different language types.

[0106] The cloud server can construct text data in multiple different language types. Specifically, the cloud server can construct related texts in multiple different language types by obtaining text corpora in multiple different language types, extracting text pairs with relevant content from the text corpora in each language type, and obtaining related text pairs in each language type.

[0107] One text corpus covers one language type, and different text corpora can cover different language types. The cloud server can collect text corpora from as many different language types as possible, collecting as much text corpora as possible for each language type. Furthermore, text corpora from a single language type can cover as many technical fields as possible. For example, text corpora of articles in different languages, question-and-answer data in different languages, and code and comments in different programming languages. Based on text corpora of any language type, content-related text pairs are extracted from the text corpus according to the data type of the text corpus to obtain related text pairs in a single language.

[0108] In addition, the cloud server can also collect related text pairs that have been constructed in the prior art for different language types as single language related text pairs. By collecting single language related text pairs of multiple different language types, single language related text pairs of multiple language types can be obtained.

[0109] For example, relevant text pairs in Chinese used in the prior art for comparative learning and training of the text representation capabilities of Chinese text representation models, relevant text pairs in English used in the prior art for comparative learning and training of the text representation capabilities of English text representation models, and relevant text pairs of other language types can be obtained to obtain single-language relevant text pairs of multiple language types.

[0110] Optionally, the cloud server can construct cross-language related texts in the following manner: obtain a cross-language text corpus, the cross-language text corpus includes text segments of at least two different language types; extract two text segments of different language types and related content from the cross-language text corpus to obtain a cross-language related text pair.

[0111] A cross-language text corpus covers more than two language types, and different cross-language text corpora may not cover exactly the same language types. Based on any cross-language text corpus, two text pairs of different language types and related content are extracted from the cross-language text corpus to obtain cross-language related text pairs.

[0112] For example, a cross-language text corpus may include cross-language search information generated by a cross-language text search system, including query information and search results, where the query information and the corresponding search results are texts in different languages. The query information and the corresponding search results are extracted from the cross-language search information to obtain two texts in different languages ​​with related content, forming a cross-language related text pair.

[0113] For example, a cross-language text corpus can include question-and-answer data generated by a cross-language question-and-answer system, where the questions and answers are in different languages. Questions and answers are extracted from the cross-language question-and-answer data to obtain two texts in different languages ​​with related content, forming a cross-language related text pair.

[0114] For example, a cross-language text corpus might include code and its annotations. The code is written in a programming language, while the annotations are written in a different language. By extracting the annotations and corresponding code snippets from the code, we can obtain two texts of different languages ​​with related content, forming a cross-language related text pair.

[0115] Optionally, after obtaining the single-language related text pairs, the cloud server can construct cross-language related texts in the following manner: for the two texts contained in any single-language related text pair of any language type, one of the texts is combined with the translation text of at least one language type of the other text to form a cross-language related text pair.

[0116] For example, for any single-language related text pair, including a first text T1 and a second text T2 related to the content of the first text T1, the first text T1 and the second text T2 are both in the first language. The second text is translated into multiple other different language types to obtain translated texts T2', T2", T2"'..., wherein the language types of T2', T2", T2"'... are different and none of them is the first language. The first text T1 and the translated texts T2', T2", T2"'... are respectively formed into cross-language related text pairs to obtain (T1, T2'), (T1, T2"), (T1, T2"')... Of course, the first text T1 can also be translated into multiple different language types to obtain translated texts T1', T1", T1"'..., wherein the language types of T1', T1", T1"'... are different and none of them is the first language. The translated texts T1', T1", T1'"... of the first text are respectively combined with the second text T2 to form cross-language related text pairs, and (T1', T2), (T1", T2), (T1'", T2)... are obtained.

[0117] Optionally, the cloud server may further form a cross-language related text pair for any text included in any single-language related text pair of any language type by combining the text with a translated text of at least one language type of the text.

[0118] For example, the aforementioned second text T2 and the translated texts T2', T2", T2'... are respectively formed into cross-language related text pairs, and the results are (T2, T2'), (T2, T2"), (T2, T2')... In addition, the first text T1 and the translated texts T1', T1", T1'... can also be respectively formed into cross-language related text pairs, and the results are (T1, T1'), (T1, T1"), (T1, T1')...

[0119] In other embodiments, the text data of multiple different language types may include only single-language related text pairs of multiple different language types, or only cross-language related text pairs, which is not specifically limited in this embodiment.

[0120] In an optional embodiment, in order to improve the comparative learning training effect of the text representation model, after constructing text data of multiple different language types, the constructed related text pairs (including single-language related text pairs and cross-language related text pairs) can be annotated based on the correlation between the two texts in the related text pairs, so that the annotated related text pairs have a stronger correlation, which can improve the training effect of the model's text representation ability.

[0121] Specifically, the cloud server can display related text pairs contained in text data of multiple different language types through the annotation interface. Through the annotation interface, the annotator can view the related text pairs and mark whether the related text pairs have a strong correlation. Through the annotation interface, the annotator can edit any text in the related text pair, and modify one or two texts of the related text pair by editing so that the edited related text pair has a strong correlation, that is, the two texts contained in the related text pair have a strong correlation. In addition, through the annotation interface, the annotator can also modify any displayed related text pair that has no correlation or a weak correlation.

[0122] In response to an edit operation on any related text pair displayed on the annotation interface, the cloud server updates the related text pair. The edit operation on any related text pair may be an edit operation on any one or both texts in the related text pair, or a delete operation on any related text pair.

[0123] This embodiment can improve the quality of text data in multiple different language types by annotating the text data obtained in multiple different language types, and can improve the training effect of the model's multi-language text representation capability, thereby improving the multi-language text representation capability of the obtained text representation model and improving the multi-language text representation quality.

[0124] In this embodiment, after obtaining text data of multiple language types, through subsequent steps S504-S506, the constructed pre-trained text representation model is subjected to comparative learning and training using the text data of multiple language types to obtain a final text representation model.

[0125] Step S504: construct sample pairs based on the text data used for fine-tuning training, where the sample pairs include positive sample pairs and negative sample pairs.

[0126] Specifically, when using text data of multiple different language types to conduct comparative learning training on the pre-trained text representation model, multiple iterative trainings are performed based on the relevant text pairs contained in the text data of multiple different language types. During each iteration, batches of relevant text pairs are sampled, and the model parameters are optimized based on the sampled batches of relevant text pairs. When the iteration stopping condition is met, the training is completed and a trained text representation model is obtained.

[0127] The batch size of the relevant text pairs sampled in each iteration can be represented as K. The value of K can be set based on actual application scenarios and empirical values. For example, K can be set to 128, 300, 512, etc., and this embodiment does not impose a specific limitation on this. In addition, the iteration stopping condition can be when the number of iterations reaches a preset threshold or the model effect is relatively stable. The specific setting can be based on actual application scenarios and experience, and this embodiment does not impose a specific limitation on this.

[0128] In an optional embodiment, during an iterative training process, for any relevant text pair The first text in s i , will be combined with the first text s i The second text belonging to the same related text pair As with the first text i Related positive samples The relevant text pairs are taken as positive sample pairs. i The second text belonging to a different related text pair is used as the first text s i Irrelevant negative samples (denoted as The first text s i A negative sample pair is formed with any negative sample. Based on the constructed positive sample pair and negative sample pair, the pre-trained text representation model is trained by comparative learning to obtain a trained text representation model.

[0129] For example, taking the sampling batch size as 128, the 128 related text pairs obtained by sampling are used as 128 positive sample pairs. For the first text s i , with the first text s i There are 127 second texts belonging to different related text pairs, and the first text s i There are 127 negative samples, including: First Texts i Together with its 127 negative samples, they constitute negative sample pairs.

[0130] In another optional embodiment, during one iterative training process, for any relevant text pair The first text in si , you can also add the first text s i The first and second texts belonging to different related text pairs are both taken as the first text s i Irrelevant negative samples (denoted as ), to construct negative sample pairs.

[0131] For example, taking the sampling batch size as 128, the 128 related text pairs obtained by sampling are used as 128 positive sample pairs. For the first text s i , with the first text s i There are 254 (127×2) first texts and second texts belonging to different related text pairs, so the first text s i There are 254 negative samples, including: First Texts i Together with its 254 negative samples, they form negative sample pairs.

[0132] In another optional embodiment, during one iterative training process, for any relevant text pair The first text in s i , first text s i The negative samples can be obtained from other texts that do not contain the first text s i The negative samples are obtained by sampling a specified number of texts from the first text and / or the second text of the relevant text pair. The specified number of negative samples of the first text can be set according to the actual application scenario and is not specifically limited here.

[0133] Step S505: Based on the fine-tuning model architecture of the dual-branch model, the two texts in each sample pair are respectively input into the two branch models of the fine-tuning model architecture of the dual-branch model for text representation, thereby obtaining text representations of the two texts in each sample pair.

[0134] In this step, during iterative training, the two texts contained in each sample pair (including positive sample pairs and negative sample pairs) are respectively input into two branch models, and text representations of the two texts in the sample pair are generated by the two branch models.

[0135] Step S506: Based on the text representations of the two texts in each sample pair, comparative learning training is performed on the branch models, and the parameters of the two branch models are updated. After the training is completed, any branch model is used as the trained text representation model.

[0136] In this step, the correlation between the two text representations in the sample pairs output by the two branch models is calculated as the sample pair correlation. Based on the correlation of each sample pair, a contrastive learning loss function is calculated. With the contrastive learning loss function as the optimization target, the parameters of the two branch models (shared parameters) are optimized based on the contrastive learning loss function to increase the correlation of positive sample pairs and decrease the correlation of negative sample pairs. This implements contrastive learning training of the pre-trained text representation model, improving the text representation capabilities and characteristics of the text representation model.

[0137] For example, the contrastive learning loss function can be expressed as:

[0138] Where n represents the first text s i The number of negative samples, e is a natural constant. sim(,) represents the correlation between the sample pair (between the two texts involved), specifically the cosine similarity of the text representations of the two texts. Indicates s i and The correlation between the positive sample pairs, Indicates s i and The correlation of the negative samples.

[0139] Exemplarily, the contrastive learning loss function may also be:

[0140] or, Where log is the logarithmic function.

[0141] In this embodiment, the contrastive learning loss function can also be based on The specific settings can be made according to actual needs and experience, and are not limited here.

[0142] After multiple iterations of training, the training is completed when the iteration stopping condition is met, and any branch model is used as the trained text representation model.

[0143] After obtaining the trained text representation model, the text representation model is provided to the end-side device, so that the end-side device can implement downstream text processing tasks based on the multilingual text representation capability of the text representation model.

[0144] Optionally, the cloud server provides a download address of the text representation model to the terminal device. The terminal device downloads the text representation model according to the download address and runs the text representation model locally.

[0145] Optionally, the cloud server sends model parameters of the text representation model to the end-side device, and the end-side device uses the received model parameters to initialize a local text representation model to obtain a trained text representation model.

[0146] Optionally, the cloud server provides an application programming interface (API) of the text representation model to the end-side device. When the end-side device needs to use the text representation model, it calls the API of the text representation model to encode the incoming text using the text representation model to obtain a text representation of the text.

[0147] In this embodiment, the text data used for training covers a variety of different language types. Using text data from a variety of different language types to conduct comparative learning and training on the pre-trained text representation model can specifically enhance the multilingual text representation capabilities of the text representation model. The trained text representation model with strong multilingual text representation capabilities can be used as a universal / unified text representation model for text representation in various languages. Based on this text representation model, text representation of text in any language type can be achieved.

[0148] The text representation model obtained in this embodiment is highly applicable to the various language types covered by the training data and can also be generalized well to other language types not covered by the training data. In other words, the text representation model is applicable to both language types covered by the training data and language types not covered by the training data. Therefore, this text representation model is a universal / unified text representation model applicable to various language types, with strong multilingual text representation capabilities. It can generate high-quality text representations for text information in various languages, thereby improving the accuracy of downstream multilingual text processing tasks.

[0149] In an optional embodiment, after obtaining a text representation model suitable for multiple languages ​​in the aforementioned embodiment, the cloud server can also provide a calling interface of the multiple text representation model to the external terminal device based on the text representation model, and realize the multilingual text representation function based on the multilingual text representation capability of the text representation model.

[0150] As shown in Figure 7, the process of implementing multilingual text representation based on the text representation model is as follows:

[0151] Step S701: The cloud server outputs the language type applicable to the text representation model through the front-end interface.

[0152] In this embodiment, the cloud server displays the language type applicable to the text representation model through the front-end interface. The user can decide whether to use the text representation model based on the applicable language type displayed on the front-end interface.

[0153] Among them, the text representation model can be very well applied to various language types covered by the training data, and can also be well generalized to other language types not covered by the training data. That is, the language types applicable to the text representation model include the language types covered by the training data and the language types not covered by the training data.

[0154] In practical applications, before a text representation model is launched, its text representation capabilities for each of several pre-set mainstream language types can be evaluated. Based on the evaluation results, the pre-set language types in which the text representation model performs well can be identified. The cloud server displays the pre-set language types for which the text representation model performs well through the front-end interface. Furthermore, as a universal / unified text representation model applicable to various language types, the text representation model has strong multilingual text representation capabilities and can also be well applied to text representation in other language types.

[0155] Step S702: The client device sends a model service request to the cloud server.

[0156] Step S703: The cloud server sends a calling interface of the text representation model to the terminal device.

[0157] When the user decides to use the text representation model, the user sends a model service request to the cloud server through the end-side device. In response to the model service request sent by the end-side device, the cloud server sends a call interface of the text representation model to the end-side device.

[0158] Step S704: The client device sends a call request for the text representation model to the cloud server, where the call request includes text information of any language type.

[0159] When text representation is needed, the client device sends a request to the cloud server using the text representation model's call interface provided by the cloud server. The request carries the text information to be represented. The client device can use text information in any language as the text to be represented, depending on the application's needs.

[0160] Step S705: The cloud server inputs the text information into the text representation model for text representation to obtain a text representation of the text information.

[0161] The cloud server receives a request from the client device for a text representation model and retrieves the text information in any language included in the request. The cloud server inputs the text information into the text representation model, generates a text representation of the text information using the text representation model, and returns the text representation of the text information to the client device.

[0162] Step S706: The cloud server sends a text representation of the text information to the end-side device.

[0163] Step S707: The terminal device implements the text processing task based on the text representation of the text information.

[0164] The end-side device receives the text representation of the text information returned by the cloud server, and executes the subsequent processing logic of the text processing task based on the text representation of the text information.

[0165] Through the solution of this embodiment, the multilingual text representation capabilities of the text representation model can be used to implement various multilingual or monolingual text processing tasks, such as clustering and classification based on multilingual text, cross-language text retrieval, and cross-language question answering. Of course, the text representation model can also be applied to other tasks in the fields of natural language processing and artificial intelligence, which are not specifically limited here.

[0166] The method of this embodiment applies the text representation model to various downstream text processing tasks. Based on the powerful multilingual text representation capabilities of the text representation model, the text representation quality can be improved, thereby improving the accuracy and effectiveness of downstream text processing tasks, especially multilingual text processing tasks.

[0167] Figure 8 is a flowchart of a multilingual text processing method provided by another exemplary embodiment of the present application. In this embodiment, the cloud server provides a multilingual text representation service based on the multilingual text representation model obtained through the aforementioned method embodiment. As shown in Figure 8, the specific steps for implementing multilingual text representation based on this text representation model are as follows:

[0168] Step S801: Acquire text information to be processed, where the text information to be processed covers a variety of different language types.

[0169] In this embodiment, the text information to be processed obtained by the cloud server can cover multiple different language types. The text information in multiple different language types can come from different application systems.

[0170] For example, the cloud server can receive calls to the text representation model from application systems of different language types, where the text representation requests contain text information of the corresponding language type. For example, application systems of different languages ​​(such as a Chinese text retrieval system and an English text retrieval system) developed for users with different native languages ​​may generate text information to be processed in the corresponding language.

[0171] For example, a cloud server may receive a call request from a multilingual application system for a text representation model, where the call request includes text information in at least one language type. Different call requests may include text information of different semantic types. For example, an application system that implements a multilingual text processing task (such as a cross-language text retrieval system) may generate text to be processed in multiple different language types.

[0172] Step S802: Input text information of various language types into a text representation model for text representation to obtain text representation of text information of various language types. The text representation model is obtained by comparative learning and training a multilingual pre-trained large model using text data of multiple language types, and is suitable for text representation of text information of multiple language types.

[0173] For the text information obtained covering multiple different language types, the cloud server inputs the text information of each language type into the text representation model for text representation, and uses the text representation model as a multi-language universal / unified text representation model, without the need to train specific text representation models for different language types. This can effectively reduce time and resource costs and improve the accuracy and efficiency of multi-language text representation.

[0174] Step S803: Output the text representation of the text information.

[0175] In this step, the cloud server outputs a text representation of the text information in a preset manner. The manner in which the cloud server outputs the text representation can be configured according to application requirements. For example, the cloud server may send the text representation of the text information to the end-side device that provides the text information, output the text representation of the text information through a front-end interface, or return the text representation of the text information through an interface callback, etc., and is not specifically limited here.

[0176] In the method of this embodiment, the cloud server can provide multilingual text representation services based on the text representation model, thereby realizing the multilingual text representation function. The powerful multilingual text representation capability of the text representation model can improve the text representation quality.

[0177] FIG9 is a flowchart of a cross-language text retrieval method provided by an exemplary embodiment of the present application. In this embodiment, based on the multilingual text representation model obtained through the aforementioned method embodiment, a server running the text representation model can implement cross-language text retrieval. As shown in FIG9 , the specific steps for implementing cross-language text retrieval based on the text representation model are as follows:

[0178] Step S901: In response to a query request, obtain a query text in a first language type.

[0179] The user can input a query text through the client and send a query request to the server, which carries the query text. The language type used in the query text is recorded as the first language type, which is usually the user's native language or a language type that the user is familiar with.

[0180] In response to the received query request, the server obtains the query text in the first language type input by the user.

[0181] Step S902: Input the query text of the first language type into the text representation model for encoding to generate a text representation of the query text.

[0182] The server uses a unified text representation model to encode the input query text in the first language type to obtain a text representation of the query text.

[0183] Step S903: Based on the text representation of the query text and the text representation of the candidate texts in the retrieval library, search the retrieval library for candidate texts that semantically match the query text, wherein there is at least one candidate text in the retrieval library that has a different language type from the query text, and the text representation of the candidate text in the retrieval library is generated by encoding the text representation model.

[0184] In this embodiment, a search library can be pre-built using a text representation model. The search library includes candidate texts and text representations of the candidate texts. The candidate texts in the search library can cover one or more different language types. When building the search library, the candidate texts in the search library in one or more different language types are uniformly encoded using the text representation model to obtain text representations of the candidate texts.

[0185] For example, in an exemplary cross-language text retrieval system, there is at least one candidate text in the retrieval library that has a different language type from the query text input by the user. The candidate texts in the retrieval library can all be texts of a second language type, which is different from the first language type. The candidate texts of the second language type in the retrieval library and the query text of the first language type input by the user are encoded using a unified text representation model to obtain corresponding text representations. Based on the method of this embodiment, it is possible to retrieve and return candidate texts of a second language type that the user may be less familiar with based on texts of the first language type that the user is familiar with.

[0186] For example, in an exemplary cross-language text retrieval system, the retrieval library may contain candidate texts of multiple language types, so that the retrieval library contains richer and more comprehensive knowledge. The candidate texts of multiple language types in the retrieval library are all encoded through a unified text representation model to obtain corresponding text representations. The query information input by the user can be of a specific language type or of any language type. Based on the method of this embodiment, it is possible to retrieve the knowledge that the user wants from knowledge texts of multiple language types, realize cross-language text retrieval, and improve the recall rate of the retrieval system.

[0187] When performing text retrieval, the similarity between the text representation of the query text and the text representation of each candidate text in the retrieval library is calculated based on the text representation of the query text and the text representation of each candidate text in the retrieval library, which is used as the semantic matching degree between the query text and the candidate text. At least one candidate text with a high semantic matching degree with the query text is retrieved from the retrieval library as the candidate text that semantically matches the query text.

[0188] Among them, based on the text representation retrieval of candidate texts that semantically match the query text, any existing retrieval matching strategy can be used to determine at least one candidate text with a high degree of semantic matching with the query text. The specific retrieval matching strategy can be configured according to the actual needs of the retrieval system and is not specifically limited here.

[0189] Step S904: Output candidate texts that semantically match the query text.

[0190] After the retrieval is completed, the server outputs candidate texts that semantically match the query text to display the retrieval results to the user.

[0191] The method of this embodiment applies the text representation model obtained in the previous embodiment to the cross-language text retrieval task. Based on the powerful multilingual text representation capability of the text representation model, the candidate texts of various language types in the retrieval library and the query text of any language type input by the user are uniformly encoded to obtain the corresponding text representation, which can improve the quality of multilingual text representation and thus improve the precision and recall rate of cross-language text retrieval.

[0192] FIG10 is a schematic diagram of the structure of a cloud server provided in an embodiment of the present application. As shown in FIG10 , the cloud server includes: a memory 1001 and a processor 1002. The memory 1001 is used to store computer-executable instructions and can be configured to store various other data to support operations on the cloud server. The processor 1002 is communicatively connected to the memory 1001 and is used to execute the computer-executable instructions stored in the memory 1001 to implement the technical solutions provided by any of the above-mentioned method embodiments. The specific functions and technical effects that can be achieved are similar and will not be repeated here.

[0193] Optionally, as shown in Figure 10, the cloud server also includes other components such as a firewall 1003, a load balancer 1004, a communication component 1005, and a power supply component 1006. Figure 10 only schematically illustrates some components, which does not mean that the cloud server only includes the components shown in Figure 10.

[0194] An embodiment of the present application also provides a computer-readable storage medium, which stores computer execution instructions. When the computer execution instructions are executed by a processor, they are used to implement the technical solutions provided by any of the above method embodiments. The specific functions and technical effects that can be achieved are not repeated here.

[0195] An embodiment of the present application also provides a computer program product, which includes: a computer program, the computer program is stored in a readable storage medium, at least one processor of the cloud server can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the cloud server executes the technical solution provided by any of the above method embodiments. The specific functions and technical effects that can be achieved are not repeated here.

[0196] An embodiment of the present application provides a chip comprising: a processing module and a communication interface. The processing module is capable of executing the technical solution of the cloud server in the aforementioned method embodiments. Optionally, the chip also includes a storage module (e.g., a memory) configured to store instructions, and the processing module configured to execute the instructions stored in the storage module. Execution of the instructions stored in the storage module causes the processing module to execute the technical solution provided by any of the aforementioned method embodiments.

[0197] The above-mentioned storage may be an object storage service (OSS).

[0198] The above-mentioned memory can be implemented by any type of volatile or non-volatile memory device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0199] The above-mentioned communication component is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as a mobile hotspot (WiFi), a second-generation mobile communication system (2G), a third-generation mobile communication system (3G), a fourth-generation mobile communication system (4G) / long-term evolution (LTE), a fifth-generation mobile communication system (5G) and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0200] The power supply assembly provides power to various components of the device in which the power supply assembly is located. The power supply assembly may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which the power supply assembly is located.

[0201] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0202] The present invention is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0203] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0204] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0205] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0206] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0207] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0208] It should be noted that the user information (including but not limited to user device information, user attribute information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0209] In addition, some of the processes described in the above embodiments and the accompanying drawings include multiple operations that appear in a specific order, but it should be clearly understood that these operations may not be executed in the order in which they appear in this document or in parallel. They are only used to distinguish between different operations, and the sequence number itself does not represent any order of execution. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequence, nor do they limit "first" and "second" to different types. The meaning of "multiple" is more than two, unless otherwise clearly and specifically defined.

[0210] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of the present application and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0211] It should be understood that the present application is not limited to the exact structure described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A data processing method, characterized in that: include: Add a natural language processing module to the pre-trained model to form a pre-trained natural language processing model; Based on the pre-trained natural language processing model, construct a fine-tuning model architecture of a multi-branch model; Based on the fine-tuning model architecture of the multi-branch model, fine-tuning training of the pre-trained natural language processing model is performed to obtain a trained natural language processing model.

2. The method according to claim 1, characterized in that The fine-tuning model architecture of the multi-branch model includes multiple parallel branch models, at least one of which uses the pre-trained natural language processing model.

3. The method according to claim 2, characterized in that The fine-tuning model architecture based on the multi-branch model is used to perform fine-tuning training on the pre-trained natural language processing model to obtain a trained natural language processing model, including: Obtaining a training set, wherein the training set includes multiple pieces of training data, and one piece of training data includes multiple information samples; Inputting multiple information samples of a training data into each branch model of the fine-tuning model architecture of the multi-branch model respectively to obtain processing results corresponding to the multiple information samples; According to the processing results corresponding to the multiple information samples, the parameters of each branch model are updated, and after the training is completed, the branch model of the pre-trained natural language processing model will be used as the trained natural language processing model.

4. The method according to any one of claims 1 to 3, characterized in that The adding of a natural language processing module to the pre-trained model to form a pre-trained natural language processing model includes: Add a text representation module to the pre-trained model to form a pre-trained text representation model; The fine-tuning model architecture of the multi-branch model is constructed based on the pre-trained natural language processing model, including: The pre-trained text representation model is used as a branch model to construct a fine-tuning model architecture of a dual-branch model, wherein the fine-tuning model architecture of the dual-branch model includes two parallel branch models.

5. The method according to claim 4, characterized in that The fine-tuning model architecture based on the multi-branch model is used to perform fine-tuning training on the pre-trained natural language processing model to obtain a trained natural language processing model, including: Constructing sample pairs according to text data used for fine-tuning training, wherein the sample pairs include positive sample pairs and negative sample pairs; Based on the fine-tuning model architecture of the dual-branch model, the two texts in each of the sample pairs are respectively input into the two branch models of the fine-tuning model architecture of the dual-branch model for text representation, so as to obtain text representations of the two texts in each of the sample pairs; According to the text representation of the two texts in each of the sample pairs, the branch models are subjected to comparative learning training, the parameters of the two branch models are updated, and after the training is completed, any branch model is used as a trained text representation model.

6. The method according to claim 5, characterized in that Also includes: Constructing text data of multiple different language types, wherein the text data of multiple different language types include: single language related text pairs and cross-language related text pairs, The monolingual related text pair includes two texts of the same language type; The cross-language related text pair includes two texts of different language types.

7. The method according to claim 6, characterized in that The text data of various languages ​​are constructed, including: Obtain text corpora in multiple different languages; Content-related text pairs are extracted from text corpora of various language types to obtain monolingual related text pairs corresponding to various language types.

8. The method according to claim 6, characterized in that The text data of various languages ​​are constructed, including: Acquire a cross-language text corpus, wherein the cross-language text corpus includes text segments in at least two different language types; From the cross-language text corpus, two text segments of different language types and related content are extracted to obtain a cross-language related text pair.

9. The method according to claim 7, characterized in that: Also includes: For two texts included in any single-language related text pair of any language type, one of the texts and a translation text of at least one language type of the other text are used to form a cross-language related text pair.

10. The method according to claim 6, characterized in that Also includes: Displaying the related text pairs contained in the text data of the plurality of different language types through a marking interface; In response to an editing operation on any related text pair displayed on the annotation interface, the related text pair is updated.

11. The method according to claim 6, characterized in that Also includes: A call request from a receiving end-side device for the text representation model, wherein the call request includes text information of any language type; Inputting the text information into the text representation model for text representation to obtain a text representation of the text information; Sending a text representation of the text information to the end-side device.

12. The method according to claim 6, characterized in that Also includes: Outputting the language type applicable to the text representation model through a front-end interface; In response to receiving a model service request sent by an end-side device, a calling interface of the text representation model is sent to the end-side device.

13. The method according to any one of claims 1 to 3, characterized in that: Also includes: A call request from a receiving end-side device to the natural language processing model, wherein the call request includes information to be processed; Inputting the information to be processed into the natural language processing model for natural language processing to obtain a natural language processing result; The natural language processing result is sent to the end-side device.

14. A cloud server, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 13.

Images