Text processing method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a text processing method and device, electronic equipment and a storage medium, and belongs to the technical field of artificial intelligence. The method comprises the steps that obtained to-be-processed initial text information is input into preset large language models, initial features correspondingly output by all the large language models are obtained, and the large language models comprise target large language models of different model structures and a plurality of heterogeneous large language models; based on each vector conversion matrix corresponding to the target large language model and the model structure of each heterogeneous large language model, performing mapping processing on any initial feature to obtain a mapping feature corresponding to each large language model in the same vector space; and determining a target feature based on the plurality of different mapping features, and performing inverse mapping processing on the target feature based on the target large language model to obtain target text information corresponding to the initial text information. The accuracy of the output target text information can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a text processing method, device, electronic device and storage medium. Background Art

[0002] Large Language Model (LLM) refers to a deep learning model trained with a large amount of text data, which can understand the meaning of language text or generate natural language text. Since different LLM models differ in storage capacity and training data, the model performance of each LLM model is usually different.

[0003] In order to fully utilize the advantages of different LLM models, the related technology first allows each independently trained large language model to perform natural language processing on the input data and output corresponding candidate results respectively, so as to derive the final common output result from multiple candidate results. However, this processing method ignores the rich and valuable processing information contained in each LLM model, and the LLM models do not interact and collaborate well, which leads to poor accuracy of the final common output result. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a text processing method, device, electronic device and storage medium, aiming to improve the accuracy of the common output results obtained by multiple LLM models based on the same input data.

[0005] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a text processing method, the method comprising:

[0006] Get the initial text information to be processed;

[0007] Inputting the initial text information into the preset large language models respectively to obtain the initial features of the corresponding outputs of the large language models, wherein the large language models include a target large language model and multiple heterogeneous large language models, and each heterogeneous large language model has a different model structure from the target large language model;

[0008] Based on the vector conversion matrices corresponding to the target large language model and the model structure of each heterogeneous large language model, any initial feature is mapped to obtain the mapping features corresponding to each large language model in the same vector space;

[0009] A target feature is determined based on a plurality of different mapping features, and an inverse mapping process is performed on the target feature based on a target large language model to obtain target text information corresponding to the initial text information.

[0010] In some embodiments, the mapping process includes a linear mapping process and a non-linear mapping process.

[0011] In some embodiments, the target feature is determined based on a plurality of different mapping features, including:

[0012] Perform average fusion processing on multiple different mapping features to obtain the target feature;

[0013] Alternatively, based on the model weights pre-set for each language model, a plurality of different mapping features are weightedly fused to obtain the target feature.

[0014] In some embodiments, the vector transformation matrix is ​​constructed by the following steps, comprising:

[0015] Obtaining the corresponding vocabulary of each language model, and determining the same words between different vocabulary as anchor words, wherein the vocabulary includes multiple words;

[0016] Based on the association between each word in each vocabulary and the anchor word, a corresponding vector conversion matrix is ​​constructed.

[0017] In some embodiments, based on the association relationship between each word in each vocabulary and the anchor word, a corresponding vector conversion matrix is ​​constructed, including:

[0018] For each vocabulary, the word embedding similarity information between each word and the anchor word is calculated, and the initial vector conversion matrix corresponding to the vocabulary is constructed based on the word embedding similarity information;

[0019] Based on the preset normalized exponential function, all initial vector conversion matrices are normalized to obtain the corresponding vector conversion matrices of each major language model.

[0020] In some embodiments, the target text information is obtained by outputting an integrated model, and the integrated model is trained by the following steps, including:

[0021] Obtaining sample initial text information to be processed and sample verification text information corresponding to the sample initial text information, inputting the sample initial text information into a pre-built integrated model, and obtaining sample initial features corresponding to the output of each large language model, wherein the integrated model is composed of multiple large language models, the large language model includes a target large language model and multiple heterogeneous large language models, and each heterogeneous large language model has a different model structure from the target large language model;

[0022] Based on the vector conversion matrices corresponding to the target large language model and the model structure of each heterogeneous large language model, the initial features of any sample are mapped to obtain the sample mapping features corresponding to each large language model in the same vector space;

[0023] Determine a sample target feature based on a plurality of different sample mapping features, and perform inverse mapping processing on the sample target feature based on a target large language model to obtain sample target text information corresponding to the sample initial text information;

[0024] A model loss value is calculated according to sample target information and sample verification text information, and parameters of a target large language model are adjusted based on the model loss value to obtain a target large language model after parameter adjustment;

[0025] After all the large language models have completed parameter adjustment, a trained integrated model is obtained.

[0026] In some embodiments, adjusting the parameters of the target large language model based on the model loss value to obtain the target large language model after parameter adjustment includes:

[0027] Taking a preset first training condition as a target, adjusting parameters of a target large language model according to a model loss value, and obtaining updated sample target text information according to the target large language model after parameter adjustment, wherein the first training condition is used to indicate that a model loss value between the sample target text information and the sample verification text information is minimum;

[0028] The model loss value is updated according to the updated sample target text information and sample verification text information, and the parameters of the target large language model are readjusted according to the updated model loss value;

[0029] When the preset second training condition is reached, the target large language model with adjusted parameters is obtained.

[0030] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a text processing device, the device comprising:

[0031] An acquisition module, used to acquire initial text information to be processed;

[0032] A natural language processing module is used to input the initial text information into the preset large language models respectively to obtain the initial features of the corresponding outputs of each large language model, wherein the large language model includes a target large language model and multiple heterogeneous large language models, and each heterogeneous large language model has a different model structure from the target large language model;

[0033] A mapping module, used to map any initial feature based on the vector conversion matrices corresponding to the target large language model and the model structure of each heterogeneous large language model, to obtain mapping features corresponding to each large language model in the same vector space;

[0034] The target module is used to determine the target feature based on a plurality of different mapping features, and to perform inverse mapping processing on the target feature based on the target large language model to obtain the target text information corresponding to the initial text information.

[0035] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the text processing method of the above-mentioned first aspect when executing the computer program.

[0036] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the text processing method of the above-mentioned first aspect is implemented.

[0037] The text processing method, device, electronic device and storage medium proposed in the present application obtain initial text information to be processed; input the initial text information into preset large language models respectively, and obtain initial features corresponding to the output of each large language model, wherein the large language model includes a target large language model and multiple heterogeneous large language models, and the model structure of each heterogeneous large language model is different from that of the target large language model. In this way, the heterogeneous large language model can extract valuable information from the initial text information based on its own model processing capability, so as to coordinate the initial text information with the target large language model, thereby enhancing the integrity of information extraction and processing; then, based on each vector conversion matrix corresponding to the model structure of the target large language model and each heterogeneous large language model, any initial feature is mapped to obtain mapping features corresponding to each large language model in the same vector space, so that the processing advantages of each large language model are fully utilized, the target feature is determined based on multiple different mapping features, and the target feature is inversely mapped based on the target large language model, thereby reducing the dependence of the initial text information on a single model, and obtaining high-accuracy target text information corresponding to the initial text information. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a schematic diagram of an optional application scenario of the text processing device provided in an embodiment of the present application;

[0039] Figure 2 is an optional flowchart of the text processing method provided in the embodiment of the present application;

[0040] Figure 3 is another optional flowchart of the text processing method provided in the embodiment of the present application;

[0041] Figure 4 yes Figure 3 A flowchart of an implementation of step 202 in FIG.

[0042] Figure 5 yes Figure 2 A flowchart of an implementation of step 104 in FIG.

[0043] Figure 6 It is another optional flowchart of the text processing method provided in the embodiment of the present application;

[0044] Figure 7 yes Figure 6 A flowchart of an implementation of step 504 in FIG.

[0045] Figure 8 is an optional flowchart of the text processing device provided in an embodiment of the present application;

[0046] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0048] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0049] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0050] First, some nouns involved in this application are analyzed:

[0051] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0052] Natural language processing (NLP): NLP uses computers to process, understand and apply human languages ​​(such as Chinese, English, etc.). NLP is a branch of artificial intelligence and an interdisciplinary subject between computer science and linguistics. It is often referred to as computational linguistics. Natural language processing includes grammatical analysis, semantic analysis, and text understanding. Natural language processing is often used in technical fields such as machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information intent recognition, information extraction and filtering, text classification and clustering, public opinion analysis and opinion mining. It involves data mining related to language processing, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computing.

[0053] Large language model refers to a deep learning model trained with a large amount of text data, which can understand the meaning of language text or generate natural language text. Since different LLM models differ in storage capacity and training data, the model performance of each LLM model is usually different.

[0054] In order to fully utilize the advantages of different LLM models, the related technology first allows each independently trained large language model to perform natural language processing on the input data and output corresponding candidate results respectively, so as to derive the final common output result from multiple candidate results. However, this processing method ignores the rich and valuable processing information contained in each LLM model, and the LLM models do not interact and collaborate well, which leads to poor accuracy of the final common output result.

[0055] Based on this, the embodiments of the present application provide a text processing method, device, electronic device and storage medium, which aim to improve the accuracy of the common output results obtained by multiple LLM models based on the same input data.

[0056] It should be noted that in the embodiments of the present application, when it comes to information related to user characteristics such as user basic information or user identity, the user's permission or consent will be obtained first, and the collection, use and processing of these data will comply with relevant laws, regulations and standards. In addition, when the embodiments of the present application need to obtain sensitive personal information of the user, the user's separate permission or consent will be obtained first. After the user's separate permission or consent is clearly obtained, the necessary data for the normal operation of the embodiments of the present application will be obtained. For example, the text processing device proposed in the embodiments of the present application will first obtain the user's authorization or consent when obtaining the initial text information related to the user, otherwise it cannot be used for the initial text information of the embodiments of the present application. In addition, the other relevant data obtained by the text processing device of the present application are all legal data after obtaining authorization or consent, which will not be repeated here.

[0057] The text processing method, device, electronic device, and storage medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the application scenario of the text processing device in the embodiments of the present application is described. Figure 1 As shown, Figure 1 The present invention is a schematic diagram of an optional application scenario of a text processing device provided in an embodiment of the present application, wherein n large language models are provided in the text processing device, and the multiple large language models include a target large language model and multiple heterogeneous large language models; first, the text processing device obtains initial text information to be processed; then, the text processing device inputs the initial text information into a preset large language model to obtain initial feature 1, initial feature 2...initial feature n corresponding to the output of each large language model, wherein the large language model includes a target large language model and multiple heterogeneous large language models, and each heterogeneous large language model has a different model structure from the target large language model; then, based on each vector conversion matrix corresponding to the target large language model and the model structure of each heterogeneous large language model, any initial feature is mapped to obtain mapping feature 1, mapping feature 2...mapping feature n corresponding to each large language model in the same vector space; then, the target feature is determined based on the n mapping features, and the target feature is inversely mapped based on the target large language model to obtain the target text information corresponding to the initial text information. It can be understood that the text processing device proposed in the present application can map the initial features obtained by processing the initial text information by each large language model into the same vector space, so as to make full use of the rich information contained in the major language models to jointly process the initial text information, thereby improving the information accuracy of the target text information finally obtained.

[0058] After understanding the example application scenarios of the text processing device proposed in the present application, the text processing method proposed in the embodiment of the present application is described next.

[0059] In the embodiment of the present application, the text processing device will be described from the perspective of the text processing device, which can be integrated into a computer device, such as a server. Figure 2 As shown, Figure 2 is an optional flowchart of the text processing method provided in the embodiment of the present application. Figure 2 The method in the embodiment may include but is not limited to the following steps 101 to 104. When the text processing device executes the prediction method, the specific process is as follows. It should be noted that the present embodiment Figure 2 The order of step 101 to step 104 is not specifically limited, and the order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0060] Step 101: Obtain initial text information to be processed.

[0061] The following is a detailed description of step 101.

[0062] Among them, the initial text information refers to the original text data that has not undergone any specific processing or analysis. The preset large language model can extract features, convert and process the initial text information to achieve the purpose of analyzing, understanding or generating new text (ie, "target text information").

[0063] Furthermore, for the initial text information obtained in different application scenarios, the preset large language model can process it under the corresponding task to generate output results consistent with the content of the initial text information. The following are some examples of initial text information obtained in specific application scenarios and the corresponding expected output:

[0064] (1) Natural language retrieval task:

[0065] Application scenario: intelligent question and answer;

[0066] Initial text information entered: Query the latest flight information;

[0067] Expected output: the latest flight information starting from the time of the query input.

[0068] (2) Machine translation tasks:

[0069] Application scenario: intelligent translation;

[0070] Initial text input: an English news article;

[0071] Expected output: The Chinese translation result corresponding to the English news article.

[0072] (3) Natural language understanding tasks

[0073] Application scenario: text sentiment analysis;

[0074] Initial text input: a comment from a user on social media, such as "This movie is great!";

[0075] Desired output: The sentiment of the review, such as positive, negative, or neutral.

[0076] It should be noted that the corresponding initial text information can also be obtained from multiple task scenarios such as code understanding and intelligent assisted writing. The input and expected output of the above task scenarios are only examples, and can be adaptively adjusted according to actual conditions. The embodiments of the present application do not limit this.

[0077] Furthermore, the initial text information can be obtained by real-time input by the user, or by reading the initial text information from a file, or by crawling text content from web pages, social media, news websites, etc. based on a web crawler, or by calling a third-party application programming interface (Application Programming Interface, API); the source of the initial text information and the corresponding information format obtained can be set according to actual conditions, and the embodiments of the present application do not limit this.

[0078] Step 102, input the initial text information into the preset large language models respectively to obtain the initial features of the corresponding outputs of each large language model, wherein the large language model includes a target large language model and multiple heterogeneous large language models, and each heterogeneous large language model has a different model structure from the target large language model.

[0079] The following is a detailed description of step 102.

[0080] Among them, the large language model refers to a deep learning model trained with a large amount of text data, which can generate natural language text or understand the meaning of language text; the large language model can handle a variety of natural language tasks, such as text classification, question-answering, dialogue, etc., and is an important path to artificial intelligence. Furthermore, the target large language model is used to finally output the target text information corresponding to the initial text information; the heterogeneous large language model can extract valuable information from the initial text information based on its own model processing capabilities, so as to coordinate the initial text information with the target large language model, enhance the integrity of information extraction and processing, and then help improve the accuracy of the final target text information.

[0081] Furthermore, the large language model can be one of a generative pre-trained model (Generative Pre-trained Transformer, GPT), a pre-trained language model (Pretrained Language Model) and a neural network language model (Neural Network Language Model, NLM); or, the large language model can also be specifically selected according to actual conditions, and the embodiments of the present application do not limit this.

[0082] Furthermore, each language model will perform natural language processing (NLP) on the initial text information, and use the results of natural language processing as the initial features of the corresponding output of each language model. Depending on the task scenario (such as machine translation, sentiment analysis, etc.), each language model uses different methods to perform natural language processing on the initial text information.

[0083] Furthermore, when the task scenarios of multiple large language models are consistent, the model performances of the large language models are different, and the large language model with the best performance in the specified task scenario is usually selected as the target large language model. That is, according to the specific requirements of the task scenario, the target large language model can be selected from multiple preset large language models. In addition, the number of heterogeneous large language models can be adaptively adjusted according to the actual situation, and the embodiments of the present application do not limit this.

[0084] The model structure refers to the architecture or framework of the big language model, which defines how the model processes input data and generates output. The model structures of the target big language model and multiple heterogeneous big language models are different, but they usually include the following common model structure elements:

[0085] (1) Transformer architecture:

[0086] Among them, the Transformer architecture is the core of the large language model, which mainly consists of two parts: the encoder and the decoder. The encoder is responsible for processing the input text information and converting it into an internal representation; the decoder is responsible for converting the internal representation into output text information.

[0087] Encoder: It consists of multiple layers (or "layers"), each of which contains a self-attention mechanism and a feedforward neural network. The self-attention mechanism enables the model to focus on different parts of the input sequence and understand the dependencies between words. The feedforward neural network is used to further process and transform the encoded representation.

[0088] Decoder: Also composed of multiple layers, the structure is similar to the encoder, but with an added attention mechanism on the encoder output so that the entire input sequence can be taken into account when generating the output text.

[0089] (2) Key components:

[0090] Word embeddings: Convert words into continuous vectors so that neural networks can process them. These vectors represent words that contain semantic information, making similar words closer in the vector space. Typical methods include Global Vectors for Word Representation (GloVe), Bidirectional Encoder Representations from Transformers (BERT), etc.

[0091] Self-attention mechanism: When processing an input sequence, the model can focus on different parts of the sequence and understand the dependencies between words. It can process all words in the sequence in parallel, improving computational efficiency.

[0092] Feedforward Neural Network: In each layer of Transformer, a feedforward neural network is used to further process and transform the encoded representation. It is usually a fully connected layer with an activation function (such as ReLU).

[0093] Positional encoding: Since the Transformer architecture has no sequential information, positional encoding needs to be added to provide the position information of each word in the sequence. This can be achieved through fixed positional encoding generated by sine and cosine functions or trainable positional encoding.

[0094] Furthermore, for each major language model that contains the basic model structural elements, the connection order of each key component can be adaptively adjusted, or the layers of the architecture can be adjusted and other key components can be added based on its own needs; and the model can be trained based on different training data sets and training conditions to ultimately obtain the trained major language models.

[0095] Step 103 , based on the vector conversion matrices corresponding to the target large language model and the model structure of each heterogeneous large language model, any initial feature is mapped to obtain mapping features corresponding to each large language model in the same vector space.

[0096] The following is a detailed description of step 103.

[0097] Among them, the vector conversion matrix is ​​usually determined according to the model structure of the target large language model and each heterogeneous large language model, which can realize the mapping processing of any initial feature, so as to obtain the mapping features corresponding to each large language model in the same vector space. In this way, the fusion processing of the initial text information based on multiple large language models helps to make full use of the processing advantages of each large language model, reduce the dependence on a single model, and improve the generalization ability of multiple large language models while improving the accuracy of the final output results.

[0098] In some embodiments, the mapping process includes a linear mapping process and a non-linear mapping process.

[0099] Among them, a linear mapping is a continuous mapping from one vector space to another vector space, which satisfies the linear property, that is, it satisfies the closure of addition and multiplication. Specifically, if there are two vector spaces V and W, and a mapping f:V→W, then for any two vectors α and β in V and any real number k, if f(α+β)=f(α)+f(β) and f(kα)=kf(α), then f is called a linear mapping.

[0100] Among them, nonlinear mapping is also called nonlinear operator, which is an operator that does not satisfy linear conditions. That is, for any two vectors α and β in V and any real number k, the nonlinear mapping f does not satisfy f(α+β)=f(α)+f(β) and f(kα)=kf(α).

[0101] In some embodiments, Figure 3 As shown, Figure 3 is another optional flow chart of the text processing method provided in the embodiment of the present application, wherein the vector conversion matrix is ​​constructed by following steps 201 to 202:

[0102] Step 201, obtain the vocabulary corresponding to each major language model, and determine the same words between different vocabulary as anchor words, wherein the vocabulary includes multiple words.

[0103] Step 202: construct a corresponding vector conversion matrix based on the association between each word in each vocabulary and the anchor word.

[0104] Steps 201 to 202 are described in detail below.

[0105] Among them, the vocabulary refers to the vocabulary set used by the large language model when processing input text information. The vocabulary set includes all words, phrases or other language units that the large language model may use when generating text (that is, "words" in the embodiment of the present application). The large language model can perform natural language processing on the input initial text information through this vocabulary set.

[0106] Further, all the same words between different word lists are determined as anchor words, or some of the same words are selected as anchor words. That is, the selection range of anchor words can be specifically set according to actual conditions, and the embodiment of the present application does not limit this.

[0107] In some embodiments, Figure 4 As shown, Figure 4 yes Figure 3 A flowchart for implementing step 202 in FIG. 1 is provided, based on the association relationship between each word in each vocabulary and the anchor word, a corresponding vector conversion matrix is ​​constructed, including the following steps 301 to 302:

[0108] Step 301, for each vocabulary, calculate the word embedding similarity information between each word and the anchor word, and construct an initial vector conversion matrix corresponding to the vocabulary according to the word embedding similarity information.

[0109] Step 302: normalize all initial vector conversion matrices based on a preset normalized exponential function to obtain vector conversion matrices corresponding to each language model.

[0110] Steps 301 to 302 are described in detail below.

[0111] In some embodiments, the association between each word in each vocabulary and the anchor word is the similarity between the two words (word embedding similarity information). That is, for each large language model, firstly, the word embedding similarity between each word in the vocabulary V of the large language model and each anchor word is calculated as its relative representation, and then, the initial vector conversion matrix of the large language model is constructed based on the relative representation of all words. The specific construction process of the initial vector conversion matrix R[i] is expressed as follows <1> As shown:

[0112]

[0113] in, represents the word embedding vector of the i-th word in the vocabulary, represents the word embedding vector of the j-th anchor word, |A| represents the total number of anchor words, cos(·) is the cosine similarity calculation formula, and R[i] represents the initial vector conversion matrix corresponding to the i-th large language model.

[0114] Next, in order to overcome the relative representation degradation problem of outlier word vectors, multiple initial vector transformation matrices R[i] are normalized to obtain the vector transformation matrices corresponding to each language model. The specific construction process is expressed as follows <2> As shown:

[0115]

[0116] Among them, softmax is a standard normalization function. Softmax can ensure that the relative representation value of each word embedding vector is in the interval [0,1] and the sum is 1, thus avoiding the situation where outlier vectors are difficult to distinguish in relative space; It is used to transform the probability distribution of each model output into a relative representation space composed of all anchor words.

[0117] It should be noted that the function used for normalization processing can be adaptively adjusted according to actual conditions, and the embodiments of the present application do not limit this.

[0118] Next, in the decoding process after obtaining the vector conversion matrix corresponding to each large language model, each large language model uses its corresponding vector conversion matrix For the initial feature p output in step 102 i Perform mapping processing to obtain the mapping features r corresponding to each large language model in the same vector space i The specific processing process is as follows <3> As shown:

[0119]

[0120] Among them, r i is the mapping feature corresponding to the output of the i-th large language model.

[0121] Step 104 , determining a target feature based on a plurality of different mapping features, and performing inverse mapping processing on the target feature based on a target large language model to obtain target text information corresponding to the initial text information.

[0122] Step 104 is described in detail below.

[0123] Compared with the related art in which the initial text information is mapped by various language models and the multiple target text information obtained after reverse mapping and then compared and analyzed again, the embodiment of the present application can fully utilize the complementarity between different features by fusing multiple mapping features, so as to perform reverse mapping processing on the target features based on the target large language model, and obtain target text information with higher accuracy corresponding to the initial text information. At the same time, the computing overhead used is less and the inference speed is faster, which has extremely high practical application value.

[0124] Among them, the inverse mapping process is to convert the target feature back to a representation that is closer to or directly related to the original text information, that is, the target text information. This process may involve operations such as decoding, denoising, and reconstruction of the target feature to restore or extract useful information corresponding to the original text information.

[0125] In some embodiments, Figure 5 As shown, Figure 5 yes Figure 2 A flowchart of an implementation of step 104 in the embodiment of the present invention, determining a target feature based on a plurality of different mapping features, comprises the following steps 401 to 402:

[0126] Step 401, average fusion processing is performed on multiple different mapping features to obtain target features.

[0127] Step 402 , alternatively, based on the model weights pre-set for each language model, a weighted fusion process is performed on a plurality of different mapping features to obtain a target feature.

[0128] Steps 401 to 402 are described in detail below.

[0129] Determining the target feature based on multiple different mapping features may be to average and fuse the multiple different mapping features to obtain the target feature. i After that, a simple average aggregation strategy is used to obtain the target feature The specific processing process is as follows <4> As shown:

[0130]

[0131] Where N represents the total number of large language models.

[0132] Alternatively, each language model is pre-set with a model weight, which is used to characterize the relative importance or contribution of each corresponding mapping feature in the fusion process; then, based on different model weights, multiple mapping features are weighted and fused to obtain the target feature. The model weight can be set according to actual conditions, and the embodiment of the present application does not limit this.

[0133] In some embodiments, Figure 6 As shown, Figure 6 It is another optional flow chart of the text processing method provided in the embodiment of the present application. The target text information is obtained by outputting the integrated model, and the integrated model is trained by the following steps 501 to 505:

[0134] Step 501, obtain the sample initial text information to be processed and the sample verification text information corresponding to the sample initial text information, input the sample initial text information into a pre-built integrated model, and obtain the sample initial features corresponding to the output of each large language model, wherein the integrated model is composed of multiple large language models, the large language model includes a target large language model and multiple heterogeneous large language models, and each heterogeneous large language model has a different model structure from the target large language model.

[0135] Step 502 , based on the vector conversion matrices corresponding to the target large language model and the model structure of each heterogeneous large language model, any sample initial feature is mapped to obtain sample mapping features corresponding to each large language model in the same vector space.

[0136] Step 503 : determining a sample target feature based on a plurality of different sample mapping features, and performing inverse mapping processing on the sample target feature based on a target large language model to obtain sample target text information corresponding to the sample initial text information.

[0137] Step 504: A model loss value is calculated according to the sample target information and the sample verification text information, and the parameters of the target large language model are adjusted based on the model loss value to obtain a target large language model with adjusted parameters.

[0138] Step 505: After all large language models have completed parameter adjustment, a trained integrated model is obtained.

[0139] Steps 501 to 505 are described in detail below.

[0140] In some embodiments, multiple large language models can be constructed to obtain a comprehensive integrated model, and a trained integrated model is obtained by training the integrated model; in this way, the acquired initial text information can be input into the trained integrated model, so that the trained integrated model can perform feature processing on the initial text information and finally output the corresponding target features.

[0141] Furthermore, the training steps 501 to 503 of the integrated model are similar to steps 101 to 104 and will not be repeated here; in particular, sample verification text information corresponding to the sample initial text information will be additionally obtained, wherein the sample verification text information is used to characterize the expected output result of the initial text information, so as to adaptively adjust the parameters of the integrated model according to the actual output result of the integrated model.

[0142] Furthermore, the method for obtaining the sample verification text information may be the same as the method for obtaining the initial text information, that is, it may be obtained through manual input by the user, or it may be obtained through other open data sources, and the embodiments of the present application do not impose any limitation on this.

[0143] Furthermore, after obtaining the sample target text information output by the integrated model, the performance of the target large language model on the current task is accurately evaluated by calculating the model loss value to reflect the difference between the predicted result and the actual expected result, and then the trained integrated model is obtained by adjusting the parameters of the target large language model.

[0144] Alternatively, the parameters of each large language model in the integrated model are adjusted until all large language models have completed parameter adjustment, thereby obtaining a trained integrated model. The adjusted parameters are, but not limited to, learning rate, batch size, number of training iterations, number of network layers, number of hidden units, etc.; the specific adjusted parameters can be set according to actual conditions, and the embodiments of the present application do not limit this.

[0145] In another embodiment, the model loss value may not need to be calculated based on sample verification text information. In the absence of real verification data, a subset of the sample data set can be used in turn as a verification set, and the remaining subsets can be used as training sets to calculate the model loss value.

[0146] In some embodiments, Figure 7 As shown, Figure 7 yes Figure 6 A flowchart for implementing step 504 in the embodiment of the present invention is shown in Figure 6, wherein the parameters of the target large language model are adjusted based on the model loss value to obtain the target large language model after the parameters are adjusted, including the following steps 601 to 603:

[0147] Step 601, taking a preset first training condition as a target, adjusting the parameters of a target large language model according to a model loss value, and obtaining updated sample target text information according to the target large language model after parameter adjustment, wherein the first training condition is used to indicate that the model loss value between the sample target text information and the sample verification text information is minimum;

[0148] Step 602, updating the model loss value according to the updated sample target text information and the sample verification text information, and re-adjusting the parameters of the target large language model according to the updated model loss value;

[0149] Step 603, until the preset second training condition is met, a target large language model with adjusted parameters is obtained.

[0150] Steps 601 to 603 are described in detail below.

[0151] The first training condition indicates that in the process of training the parameters of the target large language model, the difference between the sample target text information and the sample verification text information output by the target large language model is expected to be the smallest. The specific training process can be expressed as follows: <5> express:

[0152]

[0153] in, Represents the integrated model θ i The absolute probability distribution space of , l(·) represents the loss function of the distance between two relative representations measured; Map features for samples; is the sample target feature; argmin is used to characterize the first training condition, is the model loss value; To obtain updated sample target text information based on the target large language model after parameter adjustment.

[0154] Furthermore, the loss function may be relative entropy (KL divergence), where KL divergence is a measure of the asymmetry of the difference between two probability distributions P and Q. In information theory, it represents the information loss generated when the probability distribution Q is used to fit the true distribution P. Here, P usually represents the true distribution, and Q represents the fitted distribution or model distribution of P; of course, this is only an example, and the specific loss function can be selected according to the actual situation, and the embodiments of the present application do not limit this.

[0155] Next, after the sample target text information is regained based on the target large language model after parameter update, the model loss value is updated according to the updated sample target text information and the sample verification text information, and the parameters of the target large language model are adjusted again according to the updated model loss value, so as to obtain the final trained target large language model according to multiple iterative trainings.

[0156] For example, from the initial feature p′ (i) Starting from the target large language model with adjusted parameters, the sample initial text information is re-processed with natural language to obtain the updated initial feature p′ (i) , as follows <6> As shown:

[0157]

[0158] Where η is an adjustable preset hyperparameter, also known as the relative ensemble learning rate; Used to characterize gradient optimization algorithms.

[0159] Next, based on the updated initial feature p′ (i)The mapping process obtains the corresponding updated sample target text information. The specific mapping process is similar to step 102 to step 104 and will not be repeated here.

[0160] Among them, the second training condition can be reaching a preset training time, or reaching a preset number of training times, which can be set specifically according to actual conditions, and the embodiments of the present application do not limit this.

[0161] In order to better understand the text processing device proposed in this application, the same initial text information is input into any model in the related art and the text processing device proposed in this application, and the output results of each are observed:

[0162] Initial text message: John buys cans of soup for buy 1get one free.He gets30cans with a normal price of$0.60.How much does he pay?

[0163] Any model in the related art: John buys 30cans of soup, and he gets 30cans for free. The normal price of one can of soup is $0.60, so the total cost of the 30cans he bought is 30*0.60=$18\nSince he gets 30cans for free, he only needs to pay for30-30=0cans.The total cost of the 0cans he bought is 0*0.60=$0.Therefore,John pays$18-$0=$18\nThe answer is 18.

[0164] The text processing device proposed in this application: John gets 30 cans of soup with a normal price of $0.60 each. Since it's buy 1 get 1 free, he only needs to pay for half of the cans. So he pays for 30 / 2 = 15 cans. The total cost of the cans is 15*0.60 = $9. \nThe answer is $9.

[0165] Reference answer: He pays for 30 / 2=15cans.So he spends 15*.6=$9.Therefore, the answer is 9.

[0166] It can be found that the output result obtained by the text processing device proposed in the present application after processing the initial text information is closer to the reference answer, that is, the output result is more accurate.

[0167] like Figure 8 As shown, Figure 8 701 is an optional flowchart of a text processing device provided in an embodiment of the present application. The text processing device includes the following modules 701 to 704:

[0168] The acquisition module 701 is used to acquire the initial text information to be processed.

[0169] The natural language processing module 702 is used to input the initial text information into the preset large language models respectively to obtain the initial features of the corresponding outputs of each large language model, wherein the large language model includes a target large language model and multiple heterogeneous large language models, and each heterogeneous large language model has a different model structure from the target large language model.

[0170] The mapping module 703 is used to map any initial feature based on the vector conversion matrices corresponding to the target large language model and the model structure of each heterogeneous large language model to obtain the mapping features corresponding to each large language model in the same vector space.

[0171] The target module 704 is used to determine a target feature based on a plurality of different mapping features, and perform inverse mapping processing on the target feature based on a target large language model to obtain target text information corresponding to the initial text information.

[0172] The text processing method, device, electronic device and storage medium proposed in the present application obtain initial text information to be processed; input the initial text information into preset large language models respectively, and obtain initial features corresponding to the output of each large language model, wherein the large language model includes a target large language model and multiple heterogeneous large language models, and the model structure of each heterogeneous large language model is different from that of the target large language model. In this way, the heterogeneous large language model can extract valuable information from the initial text information based on its own model processing capability, so as to coordinate the initial text information with the target large language model, thereby enhancing the integrity of information extraction and processing; then, based on each vector conversion matrix corresponding to the model structure of the target large language model and each heterogeneous large language model, any initial feature is mapped to obtain mapping features corresponding to each large language model in the same vector space, so that the processing advantages of each large language model are fully utilized, the target feature is determined based on multiple different mapping features, and the target feature is inversely mapped based on the target large language model, thereby reducing the dependence of the initial text information on a single model, and obtaining high-accuracy target text information corresponding to the initial text information.

[0173] The specific implementation of the text processing device is basically the same as the specific implementation of the above-mentioned text processing method, and will not be repeated here.

[0174] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above text processing method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0175] like Fig. 9 As shown, Fig. 9 : is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application, the electronic device includes:

[0176] The processor 801 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0177] The memory 802 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 802 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 802, and the processor 801 calls and executes the text processing method of the embodiment of this application;

[0178] Input / output interface 803, used to implement information input and output;

[0179] The communication interface 804 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0180] A bus 805 that transmits information between the various components of the device (e.g., the processor 801, the memory 802, the input / output interface 803, and the communication interface 804);

[0181] The processor 801 , the memory 802 , the input / output interface 803 and the communication interface 804 are connected to each other in communication within the device via a bus 805 .

[0182] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned text processing method is implemented.

[0183] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0184] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0185] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0186] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0187] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0188] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0189] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0190] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0191] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0192] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0193] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0194] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A text processing method, characterized in that: The method comprises: Get the initial text information to be processed; Inputting the initial text information into preset large language models respectively to obtain initial features outputted by the large language models, wherein the large language models include a target large language model and a plurality of heterogeneous large language models, and each of the heterogeneous large language models has a different model structure from the target large language model; Based on the vector conversion matrices corresponding to the model structures of the target large language model and each of the heterogeneous large language models, mapping processing is performed on any of the initial features to obtain mapping features corresponding to each of the large language models in the same vector space; A target feature is determined based on a plurality of different mapping features, and an inverse mapping process is performed on the target feature based on the target large language model to obtain target text information corresponding to the initial text information.

2. The method according to claim 1, characterized in that The mapping process includes a linear mapping process and a nonlinear mapping process.

3. The method according to claim 2, characterized in that The determining of the target feature based on a plurality of different mapping features comprises: Performing an average fusion process on a plurality of different mapping features to obtain the target feature; Alternatively, based on the model weights preset in each of the large language models, a weighted fusion process is performed on a plurality of different mapping features to obtain the target feature.

4. The method according to claim 1, characterized in that The vector transformation matrix is ​​constructed by the following steps, which include: Obtaining a vocabulary corresponding to each of the large language models, and determining the same words between different vocabulary as anchor words, wherein the vocabulary includes a plurality of the words; Based on the association between each of the words in each of the vocabularies and the anchor word, the corresponding vector conversion matrix is ​​constructed.

5. The method according to claim 4, characterized in that The constructing of the corresponding vector conversion matrix based on the association relationship between each of the words in each of the vocabulary and the anchor word comprises: For each of the vocabulary, word embedding similarity information between each of the words and the anchor word is calculated, and an initial vector conversion matrix corresponding to the vocabulary is constructed according to the word embedding similarity information; Based on a preset normalized exponential function, all the initial vector conversion matrices are normalized to obtain the vector conversion matrices corresponding to each major language model.

6. The method according to claim 1, characterized in that The target text information is obtained by outputting an integrated model, and the integrated model is trained by the following steps, which include: Acquire sample initial text information to be processed and sample verification text information corresponding to the sample initial text information, input the sample initial text information into the pre-built integrated model, and obtain sample initial features corresponding to the output of each large language model, wherein the integrated model is composed of a plurality of the large language models, the large language models include a target large language model and a plurality of heterogeneous large language models, and each of the heterogeneous large language models has a different model structure from the target large language model; Based on the vector conversion matrices corresponding to the target large language model and the model structure of each of the heterogeneous large language models, mapping processing is performed on any one of the sample initial features to obtain sample mapping features corresponding to each of the large language models in the same vector space; Determining a sample target feature based on a plurality of different sample mapping features, and performing inverse mapping processing on the sample target feature based on the target large language model to obtain sample target text information corresponding to the sample initial text information; Calculating a model loss value according to the sample target information and the sample verification text information, and adjusting parameters of the target large language model based on the model loss value to obtain the target large language model after parameter adjustment; After all the large language models have completed parameter adjustment, the trained integrated model is obtained.

7. The method according to claim 6, characterized in that The adjusting the parameters of the target large language model based on the model loss value to obtain the target large language model after parameter adjustment includes: Taking a preset first training condition as a target, adjusting the parameters of the target large language model according to the model loss value, and obtaining the updated sample target text information according to the target large language model after parameter adjustment, wherein the first training condition is used to indicate that the model loss value between the sample target text information and the sample verification text information is the smallest; The model loss value is updated according to the updated sample target text information and the sample verification text information, and the parameters of the target large language model are readjusted according to the updated model loss value; When the preset second training condition is reached, the target large language model after parameter adjustment is obtained.

8. A text processing device, characterized in that: The device comprises: An acquisition module, used to acquire initial text information to be processed; A natural language processing module, used for inputting the initial text information into preset large language models respectively, and obtaining initial features corresponding to the outputs of the large language models, wherein the large language models include a target large language model and a plurality of heterogeneous large language models, and each of the heterogeneous large language models has a different model structure from the target large language model; A mapping module, configured to perform mapping processing on any one of the initial features based on the vector conversion matrices corresponding to the target large language model and the model structure of each of the heterogeneous large language models, to obtain mapping features corresponding to each of the large language models in the same vector space; The target module is used to determine a target feature based on a plurality of different mapping features, and to perform inverse mapping processing on the target feature based on the target large language model to obtain target text information corresponding to the initial text information.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the text processing method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the text processing method according to any one of claims 1 to 7 is implemented.

Images