Text processing method and device, electronic equipment and storage medium

Abstract

The invention discloses a text processing method and device, electronic equipment and a storage medium. The method comprises the steps of obtaining a first to-be-processed text; dividing the first to-be-processed text into a first text and a second text based on a preset condition; encoding the first text to obtain target encoded data of the first text; inputting the second text and the target coded data into a target model to obtain a second to-be-processed text; wherein the semantics of the second to-be-processed text is the same as that of the first to-be-processed text, and the character quantity of the second to-be-processed text is smaller than that of the first to-be-processed text; inputting the second to-be-processed text into a large language model to obtain a target result of natural language processing of the second to-be-processed text by the large language model; the target result is used for replying the first to-be-processed text.

Description

Technical Field

[0001] The present application relates to the technical field of text processing, and in particular to a text processing method, device, electronic device and storage medium. Background Art

[0002] Considering the accuracy of large language models in natural language processing tasks, the length of text input to large language models is usually limited to a certain range. However, in practical applications, texts that require large language models for natural language processing may be too long. It can be seen that how to solve the natural language processing of overly long texts has become a technical problem that needs to be solved urgently. Summary of the invention

[0003] The present application provides a text processing method, device, electronic device and storage medium to at least solve the above technical problems existing in the prior art.

[0004] According to a first aspect of the present application, a text processing method is provided, comprising:

[0005] Get the first text to be processed;

[0006] dividing the first to-be-processed text into a first text and a second text based on a preset condition;

[0007] Encoding the first text to obtain target encoded data of the first text;

[0008] Inputting the second text and the target coded data into the target model to obtain a second text to be processed; wherein the semantics of the second text to be processed are partially the same as those of the first text to be processed, and the amount of characters in the second text to be processed is less than that in the first text to be processed;

[0009] The second text to be processed is input into a large language model to obtain a target result of the large language model performing natural language processing on the second text to be processed; the target result is used to reply to the first text to be processed.

[0010] In one possible implementation manner, dividing the first to-be-processed text into the first text and the second text based on a preset condition includes:

[0011] Detecting whether the first to-be-processed text contains text that needs to be kept confidential and / or needs to have a reduced amount of text;

[0012] In response to detecting that the first to-be-processed text contains text that needs to be kept confidential and / or needs to have a reduced amount of text, taking the text that needs to be kept confidential and / or needs to have a reduced amount of text as the first text;

[0013] The text in the first to-be-processed text other than the text that needs to be kept confidential and / or needs to have a reduced amount of words is used as the second text.

[0014] In one embodiment, the method further comprises:

[0015] Inputting the first text into a coding model to obtain output data of the coding model;

[0016] Using the output data of the encoding model as target encoding data of the first text;

[0017] The second text and the target coded data are input into the target model to obtain a second text to be processed; wherein the confidentiality of the second text to be processed to the text that needs to be kept confidential is stronger than the confidentiality of the first text to be processed to the text that needs to be kept confidential.

[0018] In one embodiment, the method further comprises:

[0019] Obtain training data, wherein the training data includes N training texts and actual labels of each training text, wherein N is a positive integer greater than or equal to 2;

[0020] Input N training texts into the encoding model to be trained to obtain encoding data of each training text;

[0021] Inputting the encoded data of each training text into the decoding model to be trained to obtain N third texts;

[0022] Inputting the encoded data of each training text and the N third texts into the target model to be trained to obtain N fourth texts;

[0023] Inputting the N fourth texts into the large language model to obtain N prediction results of natural language prediction performed by the large language model on the N fourth texts;

[0024] Based on the actual labels of the training texts and the N prediction results, the encoding model to be trained, the decoding model to be trained and the target model to be trained are trained.

[0025] In one possible implementation, the step of inputting the encoded data of each training text and the N third texts into a target model to be trained to obtain N fourth texts comprises:

[0026] Determine, among the coded data of the N training texts and the N third texts, the coded data and the third texts belonging to the same training text;

[0027] Concatenate the encoded data belonging to the same training text and the third text to obtain N concatenated data;

[0028] The N concatenated data are input into the target model to be trained to obtain N fourth texts.

[0029] In one possible implementation, the step of inputting N concatenated data into a target model to be trained to obtain N fourth texts includes:

[0030] Performing a mask operation on each spliced ​​data to obtain unmasked data and masked data in each spliced ​​data;

[0031] The target model to be trained obtains N fourth texts by predicting the masked data using the unmasked data.

[0032] In one possible implementation, the training of the encoding model to be trained, the decoding model to be trained, and the target model to be trained based on the actual labels of each training text and the N prediction results includes:

[0033] Based on the actual labels of each training document and N prediction results, the loss value of the loss function is obtained;

[0034] According to the loss value of the loss function, the encoding model to be trained, the decoding model to be trained, and the target model to be trained are trained.

[0035] According to a second aspect of the present application, there is provided a text processing device, comprising:

[0036] A first obtaining unit, used for obtaining a first text to be processed;

[0037] A division unit, configured to divide the first to-be-processed text into a first text and a second text based on a preset condition;

[0038] A second obtaining unit, used for encoding the first text to obtain target encoding data of the first text;

[0039] a third obtaining unit, configured to input the second text and the target coding data into a target model to obtain a second text to be processed; wherein the semantics of the second text to be processed are partially the same as those of the first text to be processed, and the amount of characters in the second text to be processed is less than that in the first text to be processed;

[0040] The fourth obtaining unit is used to input the second text to be processed into the large language model to obtain a target result of the large language model performing natural language processing on the second text to be processed; the target result is used to reply the first text to be processed.

[0041] According to a third aspect of the present application, an electronic device is provided, including:

[0042] at least one processor; and

[0043] a memory communicatively connected to the at least one processor; wherein,

[0044] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the present application.

[0045] According to a fourth aspect of the present application, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method described in the present application.

[0046] According to a fifth aspect of the present application, a computer program product is provided, comprising a computer program or instructions, which implement the method described in the present application when executed by a processor.

[0047] The technical solution of the present application can effectively obtain a short text equivalent to the original text, and input the short text equivalent to the original text into a large language model, so as to realize the accurate processing of natural language tasks by the large language model, and also meet the requirement of limiting the length of the text input into the large language model, thereby avoiding problems caused by the text being too long being input into the large language model.

[0048] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] By reading the detailed description below with reference to the accompanying drawings, the above and other purposes, features and advantages of the exemplary embodiments of the present application will become readily understood. In the accompanying drawings, several embodiments of the present application are shown in an exemplary and non-limiting manner, wherein:

[0050] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts.

[0051] Figure 1 A schematic diagram of the implementation flow of the text processing method in the embodiment of the present application is shown;

[0052] Figure 2 A block diagram showing an implementation of a text processing method in an embodiment of the present application is shown;

[0053] Figure 3 A schematic diagram of the implementation process of the training method in the embodiment of the present application is shown;

[0054] Figure 4 A block diagram showing an implementation of the training method in an embodiment of the present application is shown;

[0055] Figure 5 A schematic diagram of the structure of a text processing device in an embodiment of the present application is shown;

[0056] Figure 6 A schematic diagram of the structure of an electronic device in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0057] In order to make the purpose, features, and advantages of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0058] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0059] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0060] In the following description, the terms "first\second" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0061] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0062] It should be understood that in the various embodiments of the present application, the size of the serial number of each implementation process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0063] The text processing method provided in the present application is for (original) texts that require a large language model (LLM) for natural language processing. Before the text is input into the large language model, it can be processed into a text that is semantically partially the same as the original text but has less text than the original text. If the processed text is called a new text, which is a short text, then the new text in the present application can express partially the same or similar semantics using a smaller amount of text compared to the original text. Inputting such a new text into the large language model can realize the accurate processing of natural language tasks by the large language model, and can also meet the requirement of limiting the length of the text input to the large language model, thereby realizing the successful processing of natural language tasks by the large language model. Please combine the following description to further understand the text processing method of the present application.

[0064] Figure 1 The following is a schematic diagram showing the implementation process of the text processing method in the embodiment of the present application. Figure 1 The method is applied to a text processing device. Figure 1 As shown, the method includes:

[0065] S (step, Step) 101: obtain the first text to be processed.

[0066] In the present application, the first text to be processed may be the original text that requires a large language model for natural language task processing. The first text to be processed may be obtained by receiving or reading a text that requires natural language task processing. The first text to be processed may be a text with a long text length or a text with a moderate text length. If it is a text with a long text length, it may be processed into a short text with the same semantics as the long text portion after subsequent related processing of the present application. If it is a text with a moderate text length, it may be processed into a text with the same semantics as the text with a moderate length and less text after subsequent related processing of the present application.

[0067] S102: dividing the first to-be-processed text into a first text and a second text based on a preset condition.

[0068] In this step, the preset condition may be that the first text to be processed contains text that needs to be reduced in amount, may be that the first text to be processed contains text that needs to be kept confidential, or may be that the first text to be processed contains text that needs to be reduced in amount and needs to be kept confidential.

[0069] The text that needs to reduce the amount of text can be text content in the text that is irrelevant to the natural language processing of the large language model, or content that has little to do with the natural language processing of the large language model. For example, for the movie review text of the moviegoer: A Cinema is easy to find, the seats are not very good, it is uncomfortable to sit on, and the service attitude is average. But the plot of the movie is compact, the picture texture is good, and the money is worth it. When the large language model is required to give the natural language processing task of whether the moviegoer's evaluation of the movie is positive or negative, the text content that has little to do with the movie evaluation itself, such as "A Cinema is easy to find, the seats are not very good, it is uncomfortable to sit on, and the service attitude is average", can be used as the text that needs to reduce the amount of text. When the first text to be processed contains text that needs to reduce the amount of text, the text that needs to reduce the amount of text is used as the first text. The text in the first text to be processed other than the text that needs to reduce the amount of text is used as the second text.

[0070] The text that needs to be kept confidential can be the text content that contains private information. For example, taking personal privacy as an example, the text that needs to be kept confidential can be the text such as the user's name, gender, age, preferences and habits that appear in the text. When the first text to be processed contains text that needs to be kept confidential, the text that needs to be kept confidential is used as the first text. The text other than the text that needs to be kept confidential in the first text to be processed is used as the second text. Exemplarily, taking the scenario where a natural language task that requires a large language model to process is to recommend a product to the user, the first text to be processed is: Name: Luo Moumou; Age: 27; Previously purchased company products: Product A, Product B, Product C. The user browsed the P web page, what products can be recommended for the user. The text that needs to be kept confidential can be a text with the following content: Name: Luo Moumou; Age: 27; Previously purchased company products: Product A, Product B, Product C.

[0071] If the first text to be processed contains both text that needs to be reduced in amount and text that needs to be kept confidential, the text that needs to be reduced in amount and the text that needs to be kept confidential are taken as the first text, and the remaining text in the first text to be processed except the text that needs to be reduced in amount and the text that needs to be kept confidential are taken as the second text. Taking the aforementioned example of requiring a large language model to recommend products to users as an example, the first text to be processed may involve not only privacy information such as user name, age, and historical product purchases, but also job title, working age, home address, contact information, etc. There is too much privacy information, and there may be information that is irrelevant or not very relevant to the scenario of recommending products. In this case, such privacy information can be regarded as text that needs to be reduced in amount.

[0072] During implementation, it can be detected whether the first text to be processed contains text that needs to be kept confidential and / or needs to reduce the amount of text. If it is detected that the first text to be processed contains text that needs to be kept confidential and / or needs to reduce the amount of text, the first text to be processed is divided into two parts (first text and second text) based on preset conditions.

[0073] Among them, a sensitive word or privacy data detection algorithm can be used to detect whether the first text to be processed contains text that needs to be kept confidential. Whether the first text to be processed contains text that needs to be reduced in amount can be detected by judging whether the amount of text in the first text to be processed exceeds the preset amount of text. If the amount of text in the first text to be processed exceeds the preset amount of text, it is detected that the first text to be processed contains text that needs to be reduced in amount. It is possible to detect whether there is data in the first text to be processed that is irrelevant or not very relevant to the natural language processing task of the large language model. If it is detected that it exists, it is detected that the first text to be processed contains text that needs to be reduced in amount.

[0074] S103: Encode the first text to obtain target encoded data of the first text.

[0075] In this step, any reasonable encoding method may be used to encode the first text. For example, a neuron method may be used to encode the first text. The present application provides a trained deep learning model - an encoding model, which can realize the encoding of the first text. During implementation, the first text may be input into the encoding model, and the encoding model may be used to realize neuron encoding of the first text, thereby obtaining the output data of the encoding model, and the data output by the encoding model may be used as the encoding data of the first text. In practical applications, the encoding data of the first text may be vector data.

[0076] It can be understood that encoding a text achieves compression of the text. For the original text (the first text to be processed) in this application, when it is divided into two parts, the text can be compressed by encoding the text of the first part (the first text), which facilitates the acquisition of short texts.

[0077] S104: Input the second text and the target coding data into the target model to obtain a second text to be processed; wherein the semantics of the second text to be processed is partially the same as that of the first text to be processed, and the amount of text in the second text to be processed is less than that in the first text to be processed.

[0078] In the present application, the target model is a trained neural network model (NN network). During implementation, the vector data of the text that needs to be kept confidential in the original text and the rest of the text in the original text can be input into the trained NN network. Alternatively, the vector data of the text that needs to reduce the text in the original text and the rest of the text in the original text are input into the trained NN network. Alternatively, the vector data of the text that needs to reduce the text and needs to be kept confidential in the original text, as well as the rest of the text in the original text are input into the trained NN network. The trained NN network processes the input data and has the ability to output new text that has some of the same semantics as the original text and has less text. The text output by the trained NN network is used as the second text to be processed or the new text.

[0079] It should be noted that if the first text is the text that needs to be kept confidential in the first text to be processed, and the second text is the text other than the text that needs to be kept confidential in the first text to be processed, then the encoding in S103 is equivalent to encrypting the text that needs to be kept confidential in the original text, thereby avoiding the problem of privacy leakage caused by the private information being directly input into the target model in plain text. Inputting the vector data of the text that needs to be kept confidential in the original text and the remaining text into the NN network can be regarded as inputting the encrypted private information and the remaining plain text into the NN network, thereby avoiding privacy leakage. For the second text to be processed output by the NN network, because it is a text that is semantically close or similar but has less text, the confidentiality of the second text to be processed to the text that needs to be kept confidential is stronger than the confidentiality of the first text to be processed to the text that needs to be kept confidential.

[0080] S105: Input the second text to be processed into the large language model to obtain a target result of the large language model performing natural language processing on the second text to be processed; the target result is used to reply to the first text to be processed.

[0081] Large language models can handle a variety of natural language tasks, such as text classification, question-answering, and conversation. When processing natural language tasks, LLM can combine the context of the text to output the natural language processing results. For example, by splicing the previous text content to the current text content and inputting it into the LLM, the LLM outputs the question-answering content or the conversation content.

[0082] It can be understood that in this application, because the new text output by the NN network is partially identical to the original text in terms of semantics, and the new text has less text than the original text, this text that uses fewer words to express similar or similar semantics can be regarded as a short text equivalent to the original text. Inputting the new text (the second text to be processed) equivalent to the original text into the large language model can realize the accurate processing of natural language tasks by the large language model, and can also meet the requirement of limiting the length of the text input to the large language model, avoiding the problem that the large language model cannot successfully perform natural language processing due to the excessive length of the text input to the large language model.

[0083] Figure 2 FIG. 1 shows a block diagram of the text processing method in the embodiment of the present application. Figure 2 In the article, the whole system including the encoding model Encode, the splicing module, the NN network and the large language model, as well as the application scenario of the large language model recommending products to users are taken as examples to illustrate.

[0084] In this application scenario, the first text to be processed, i.e. the original text, is: Name: Luo Moumou; Age: 27; Occupation: Advertising planner; Working experience: 3 years; Home address: No. 5 Dachang Road, Qianshan District; Products purchased from the company: Product A, Product B, Product C. The user browsed web page P. What products can be recommended for the user?

[0085] After testing the original text, it was found that the original text contained text that needs to be kept confidential. The text that needs to be kept confidential is the data in the original text that involves user privacy: Name: Luo Moumou; Age: 27; Occupation: Advertising Planner; Years of Work: 3 years; Home Address: No. 5 Dachang Road, Qianshan District; Products purchased from the company: Product A, Product B, Product C. This part of the data in the original text that involves user privacy is taken as the first text (represented by Text 1). The second text in the original text (represented by Text 2) is the remaining text in the original text: The user browsed the P webpage, and what products can be recommended to the user.

[0086] Text 1 is input into the encoding model to obtain the vector data Embedding of Text 1 output by the encoding model. The encoding process of Text 1 by the encoding model is equivalent to the compression of Text 1. The compressed text makes it easier to obtain a short text equivalent to the original text. In the present application, the encoding model is a trained deep learning model with strong robustness and stability. The vector data of Text 1 obtained by encoding Text 1 with a deep learning model with such characteristics can make the acquisition of vector data more accurate.

[0087] Input Embedding and Text 2 to the splicing module to realize the splicing of vector data and text data. When splicing, the encoded data can be placed first and the text can be placed second, or the text can be placed second and the encoded data can be placed first.

[0088] Assuming that the data obtained by the splicing module is Combo, then Combo = Embedding + Text2, or Combo = Text 2 + Embedding. Input Combo into the NN network to obtain a new text output by the NN network (represented by Text 3). Compared with the plain text of the private information, Embedding is obtained by encoding the private information through the encoding model, which is equivalent to encrypting the private information in the original text, avoiding the problem of privacy leakage caused by directly inputting the private information into the NN network in plain text. The NN network uses the encrypted data and the plain text of Text 2 to obtain a new text Text 3 with fewer words but close or similar semantics to the original text.

[0089] In this application scenario, the aforementioned Combo is input into the NN network, and the resulting Text 3 is: The user is a young female white-collar worker who has just started working and has just browsed the P webpage. What products can be recommended to the user? It can be understood that in this scenario, the NN network uses the embedding of private information to process or integrate the user portrait, a private information, into a text with strong privacy confidentiality, such as "The user is a young female white-collar worker who has just started working." This processing method, on the one hand, avoids the leakage of user privacy information, and on the other hand, uses fewer words to describe the user portrait, which enables the normal use of the user portrait data by the subsequent large language model.

[0090] Text 3 is input into the large language model to obtain the output data Text_out of the large language model, which is used to reply to the original text. For example, Text_out = "Product D" is used to reply to the question "What products can be recommended for the user" for the original text.

[0091] In this application scenario, Text 3 output by the NN network is semantically similar to the original text, and describes the user portrait with fewer words without leaking the user's privacy information. It can be regarded as a short text equivalent to the original text. Inputting the short text equivalent to the original text - Text 3 into the large language model can enable the large language model to accurately process the natural language task of recommending products to users, and can also meet the requirement of limiting the length of the text input to the large language model, avoiding the problem of too long text input to the large language model.

[0092] The above scheme is an explanation of the text processing method from the perspective of model application or model application stage. In addition, this application also includes a scheme for training the model, and further, a scheme for training the target model and the encoding model, see Figure 3 shown. Figure 3 FIG. 2 shows a schematic diagram of the implementation process of the training method in the embodiment of the present application. Figure 3 As shown, the training method includes:

[0093] S301: Obtain training data, where the training data includes N training texts and actual labels of the training texts, where N is a positive integer greater than or equal to 2.

[0094] The training text can be a text that requires a large language model to perform natural language processing tasks. For example, the aforementioned movie review text is the original text in the scenario of recommending products to users. The N training texts can be different texts that require a large language model to perform the same natural language processing task, such as different movie review texts for the natural language processing task of movie review emotion recognition. The N training texts can also be texts for different natural language processing tasks, such as movie review texts for the natural language processing task of movie review emotion recognition, and texts including user portrait information for the natural language processing task of product recommendation.

[0095] Usually, the training text appears in documents, files or networks. The training data can be obtained by collecting relevant text content in the documents, files or web pages. Alternatively, part or all of the text can be read from the collected data and used as the training text. In practical applications, the training text can be text that includes private information or text that does not include private information. The number of training texts depends on the actual situation.

[0096] The actual label Text_r of each training text can be a pre-labeled label. For example, for the training text in the scenario of movie review sentiment recognition, its actual label can be "positive evaluation", "negative evaluation", "general evaluation", etc. For the training text in the scenario of recommending products to users, its actual label can be "product D", "product F", "product E", etc.

[0097] The actual label Text_r of each training text can also be obtained by the following method: input each training text into the large language model, obtain the output data of the large language model, and use the output data of the large language model as the actual label Text_r of each training text. For example, input a movie review text into the large language model, and the large language model outputs the data "positive evaluation", and "positive evaluation" can be used as the actual label Text_r of the movie review text.

[0098] S302: Input N training texts into the encoding model to be trained to obtain encoding data of each training text.

[0099] The encoded model is the deep learning model to be trained. Figure 4 A block diagram of an implementation of the training method in an embodiment of the present application is shown. Figure 4 The Encode in can be used as the encoding model to be trained in this step, and the encoding data of the training text can be recorded as Embedding 1. During implementation, each training text is input into the encoding model to be trained to obtain the encoding data of each training text. In practical applications, the encoding data of each training text is vector data.

[0100] By inputting each training text into the encoding model to be trained, it is possible to compress each training text. The compressed text and the text obtained by decoding the compressed text (the third text) are used to train the target model to be trained and the decoding model to be trained. This makes it easier to train the target model and the decoding model in the desired direction (the direction in which the loss function becomes smaller and smaller).

[0101] S303: Input the encoded data of each training text into the decoding model to be trained to obtain N third texts.

[0102] The decoding model is the deep learning model to be trained. Figure 4 Decode in can be used as a decoding model to be trained. The decoding model to be trained in this application is intended to restore each training text by decoding the encoded data of each training text. In this way, the data input to the target model to be trained can include both the training text or the text with semantics similar to the training text, and the encoded data of the training text, so as to train the target model in the direction of being able to output text that is semantically similar to the training text but with less text.

[0103] It can be understood that for the encoded data of a training text input to the decoding model, the text obtained by decoding the decoding model may be the training text itself, or may be a text that is semantically similar to the training text but different in word expression. Figure 4 As shown, if the training text is recorded as Text 1, the third text decoded by the decoding model to be trained can be recorded as Text 1'.

[0104] S304: Input the encoded data of each training text and the N third texts into the target model to be trained to obtain N fourth texts.

[0105] In the present application, among the encoded data of N training texts and N third texts, the encoded data and the third text belonging to the same training text are determined; the encoded data and the third text belonging to the same training text are spliced ​​to obtain N spliced ​​data; the N spliced ​​data are input into the target model to be trained to obtain N fourth texts.

[0106] Among the coded data of multiple training texts and multiple third texts, if a third text is obtained by inputting the coded data of a training text into a decoding model to be trained, it can be considered that the third text and the coded data of the training text are the third text and coded data belonging to the same training text. For example, coded data 10 and text 10 are the coded data and third text belonging to the same training text - training text 10. Figure 4 The splicing module shown in the figure performs character splicing. When splicing, the coded data can be placed first and the text can be placed second, or the text can be placed second and the coded data can be placed first. Figure 4 As shown, the concatenated data Combo' can be Combo'=Embedding 1+Text 1', or Combo'=Text 1'+Embedding 1.

[0107] Among them, the implementation method of inputting N spliced ​​data into the target model to be trained to obtain N fourth texts can be: performing a mask operation on each spliced ​​data to obtain unmasked data and masked data in each spliced ​​data; the target model to be trained obtains N fourth texts by using the unmasked data to predict the masked data.

[0108] In the present application, the spliced ​​data input to the target model to be trained is composed of two parts: vector and text. For any spliced ​​data, one or more characters in the text part of the spliced ​​data can be masked through a masking operation, and the target model to be trained predicts the masked characters in the text part based on the unmasked content in the vector part and the text part. Alternatively, one or more data in the vector part of the spliced ​​data are masked, and the target model to be trained predicts the masked data in the vector part based on the unmasked content in the text part and the vector part.

[0109] Because the text part is input to the target model to be trained in plain text, and the vector part is input to the target model to be trained in the form of compressed data of the text part, these two parts are essentially different forms of the same training text and can be used as prediction references for each other. Whether the text part in the spliced ​​data is masked or the vector part is masked, the other unmasked part can be used as a prediction reference for the content of the masked data. Assume that the "expensive" in the text part "This movie ticket is a bit expensive" is masked, and the target model to be trained predicts that the probability of the masked text being "expensive" is 0.9 and the probability of the masked text being "cheap" is 0.3 based on the vector data of "This movie ticket is a bit expensive" and the unmasked data context in the text part. Among them, if the probability of predicting that the masked text is "expensive" is greater than the preset probability threshold such as 0.85, the target model to be trained believes that the masked text in the text part can be the word "expensive". The target model can be regarded as an inference model, which can infer what the masked data may be based on the unmasked data in each concatenated data. The output text of the target model to be trained is Figure 4 It can be written as Text 4.

[0110] For each concatenated data consisting of two parts, vector and text, the target model to be trained gives a corresponding predicted text. The purpose of training the target model to be trained is to give a text that is semantically similar to the training text and has fewer words than the training text through the above processing of the concatenated data. That is, the purpose of training the target model to be trained is to enable the training target model to give a short text equivalent to the training text.

[0111] For example, under ideal training conditions, for the training text: Name: Luo Moumou; Age: 27; Occupation: Advertising Planner; Years of Work: 3 years; Home Address: No. 5 Dachang Road, Qianshan District; Products Purchased: Product A, Product B, Product C. The user browsed the P webpage, what products can be recommended for the user. It is expected that by training the target model, the target model can give a short text equivalent to the training text, such as: The user is a young female white-collar worker who has just started working and has just browsed the P webpage, what products can be recommended for the user.

[0112] During the training process, because the target model to be trained has not been trained yet, the (fourth) text output by the target model may be a short text that is equivalent or nearly equivalent to the training text, or it may be a text that is not equivalent to the training text, such as a text that is different from the training text. The purpose of training is to continuously adjust the model parameters of the target model and make the text output by the target model a short text equivalent to the training text through continuous iterative calculation.

[0113] S305: Inputting the N fourth texts into the large language model to obtain N prediction results of the large language model performing natural language prediction on the N fourth texts.

[0114] In this step, each (fourth) text output by the target model to be trained is input into the large language model, and the large language model processes each text for a natural language task and outputs a prediction result Text_out' for the natural language task. For example, for the natural language task of film review, the Text_out' given by the large language model is "positive evaluation", "negative evaluation", "general evaluation", etc. For the natural language task of product recommendation, the Text_out' given by the large language model can be "product D", "product F", "product E", etc.

[0115] In the training scheme of the present application, the encoding model, decoding model and target model to be trained are trained, and the large language model is not trained. The large language model can be used as an evaluation model in the training scheme of the present application, and the loss between the prediction result output by it and the actual label of the training text can be used to evaluate whether the model to be trained has been trained or trained well.

[0116] S306: Based on the actual labels of the training texts and the N prediction results, the encoding model to be trained, the decoding model to be trained, and the target model to be trained are trained.

[0117] During implementation, the loss value of the loss function can be obtained based on the actual labels of each training document and N prediction results; according to the loss value of the loss function, the encoding model to be trained, the decoding model to be trained and the target model to be trained are trained.

[0118] In the training scheme of the present application, a loss function is pre-constructed. The loss function can be any reasonable function. For example, a square loss function, a cross entropy loss function, an exponential loss function, etc. It can be understood that in the training scheme of the present application, the training of the model is achieved by multiple iterative calculations. S301 to S306 can be regarded as the scheme under each iterative calculation. It is also possible to divide the N training texts into multiple groups, each group of training texts is used as a single iteration text, and each group of training texts is processed similarly to the scheme shown in S302 to S306 to perform multiple iterations on the training model.

[0119] Considering that in each iteration, the data output by the target model to be trained may be text equivalent or nearly equivalent to the training text, or may be text not equivalent to the training text. After such text is processed by the natural language of the large language model, the prediction result obtained may be an ideal result that is the same as the actual label of the training text, or may be an undesirable result that is different from the actual label of the training text. Based on this, the Text_out' output by the model to be trained and the actual label Text_r of the training text in each iteration can be input into the pre-built loss function to calculate the loss value of the loss function at this iteration number. It is judged whether the loss value is lower than or equal to the preset threshold. If it is judged to be yes or the loss value of the loss function converges, it is considered that the target model, encoding model and decoding model are trained or completed, and the training or iteration is stopped. If it is judged to be no or the loss value of the loss function does not converge, it is considered that the target model, encoding model and decoding model are not trained, and the iteration is continued until the loss value of the loss function is lower than or equal to the preset threshold, or the loss function converges.

[0120] It can be understood that the encoding model, decoding model and target model in the present application are all deep learning models. The deep learning models have model parameters. Training these models is actually constantly adjusting the values ​​of the model parameters of each model so that the values ​​of the model parameters of each model tend to an ideal solution.

[0121] Through the training scheme shown in S301 to S306, the training of the target model, encoding model and decoding model that can output short text equivalent to the training text is realized. The training text is encoded to obtain vector data of the training text, and the vector data is decoded to obtain a decoded text (third text). The decoded text is equivalent to a text that restores or recovers the training text based on the vector data of the training text. The decoded text may be semantically identical or similar to the training text, but may be different from the training text in terms of expression.

[0122] The vector data of the decoded text and the training text are fused or concatenated and input into the target model to be trained. The decoded text is input in plain text, which is equivalent to input in context. By combining the text context and the vector data of the text, the inference model (target model to be trained) is trained to train a model that can output short text equivalent to the training text, thereby ensuring the accuracy of the training of the inference model and providing a guarantee for the practical application of the model. By combining the context of the text and the vector data of the text, the encoding model and the decoding model are trained to ensure the accuracy of the training of the encoding model and the decoding model and improve the training efficiency. The training of the encoding model and the decoding model also provides an auxiliary role for the accurate training of the inference model.

[0123] If the training text includes text that needs to be kept confidential, the decoded text obtained based on the vector data of the training text and the text output by the inference model can both be texts obtained after certain confidentiality processing of the text that needs to be kept confidential, which can effectively avoid the leakage of private information and achieve privacy protection for specific words in the text.

[0124] The above is an explanation of the training scheme or training stage of the present application. From the above content, in the training stage and the model application stage of the present application, the text is encoded to obtain or generate a vector of the text, thereby achieving compression of the text. Among them, in the model application stage, the compressed text (a part of the original text) and another part of the original text are input into the reasoning model. Compared with the scheme of directly inputting the original text into the reasoning model, the input amount of the reasoning model can be effectively reduced, and the reasoning model can achieve effective output of short texts equivalent to the original text, reduce the input amount of the large language model, and effectively solve the problem of too long input text of the large language model.

[0125] During the model application stage, if the original text contains text that needs to be kept confidential, the text that needs to be kept confidential will be vectored and will not appear in plain text. In the short text equivalent to the original text obtained through reasoning of the inference model, the information that needs to be kept confidential will be integrated or processed in a confidential manner, thereby avoiding the leakage of confidential information.

[0126] The encoding model trained in this application can be used to encode confidential information or private information by inputting it into the encoding model to obtain the vector data of the text, so as to prevent the confidential information or private information from being leaked. In some application scenarios where sensitive information needs to be stored in a confidential form, such text can be encoded in advance through the encoding model to generate vector data of the text, and the generated vector data can be saved to achieve secure storage of private information or confidential information. When needed later, it can be read from it.

[0127] The present application provides a text processing device, such as Figure 5 As shown, the device comprises:

[0128] A first obtaining unit 501 is used to obtain a first text to be processed;

[0129] A division unit 502, configured to divide the first to-be-processed text into a first text and a second text based on a preset condition;

[0130] A second obtaining unit 503 is used to encode the first text to obtain target encoding data of the first text;

[0131] The third obtaining unit 504 is used to input the second text and the target coding data into the target model to obtain a second text to be processed; wherein the semantics of the second text to be processed are partially the same as those of the first text to be processed, and the amount of characters in the second text to be processed is less than that in the first text to be processed;

[0132] The fourth obtaining unit 505 is used to input the second text to be processed into the large language model to obtain a target result of the large language model performing natural language processing on the second text to be processed; the target result is used to reply the first text to be processed.

[0133] As an implementation manner, the dividing unit 502 is used to:

[0134] Detecting whether the first to-be-processed text contains text that needs to be kept confidential and / or needs to have a reduced amount of text;

[0135] In response to detecting that the first to-be-processed text contains text that needs to be kept confidential and / or needs to have a reduced amount of text,

[0136] The text that needs to be kept confidential and / or has to be reduced in length should be the first text;

[0137] The text in the first to-be-processed text other than the text that needs to be kept confidential and / or needs to have a reduced amount of words is used as the second text.

[0138] As an implementation manner, the second obtaining unit 503 is configured to:

[0139] Inputting the first text into a coding model to obtain output data of the coding model;

[0140] Using the output data of the encoding model as target encoding data of the first text;

[0141] The third obtaining unit 504 is used to input the second text and the target coding data into the target model to obtain a second text to be processed; wherein the confidentiality of the second text to be processed to the text that needs to be kept confidential is stronger than the confidentiality of the first text to be processed to the text that needs to be kept confidential.

[0142] As an implementation mode, the device further includes a training unit, which is used to:

[0143] Obtain training data, wherein the training data includes N training texts and actual labels of each training text, wherein N is a positive integer greater than or equal to 2;

[0144] Input N training texts into the encoding model to be trained to obtain encoding data of each training text;

[0145] Inputting the encoded data of each training text into the decoding model to be trained to obtain N third texts;

[0146] Inputting the encoded data of each training text and the N third texts into the target model to be trained to obtain N fourth texts;

[0147] Inputting the N fourth texts into the large language model to obtain N prediction results of natural language prediction performed by the large language model on the N fourth texts;

[0148] Based on the actual labels of the training texts and the N prediction results, the encoding model to be trained, the decoding model to be trained and the target model to be trained are trained.

[0149] As an implementation method, the training unit is further used for:

[0150] Determine, among the coded data of the N training texts and the N third texts, the coded data and the third texts belonging to the same training text;

[0151] Concatenate the encoded data belonging to the same training text and the third text to obtain N concatenated data;

[0152] The N concatenated data are input into the target model to be trained to obtain N fourth texts.

[0153] As an implementation method, the training unit is further used for:

[0154] Performing a mask operation on each spliced ​​data to obtain unmasked data and masked data in each spliced ​​data;

[0155] The target model to be trained obtains N fourth texts by predicting the masked data using the unmasked data.

[0156] As an implementation method, the training unit is further used for:

[0157] Based on the actual labels of each training document and N prediction results, the loss value of the loss function is obtained;

[0158] According to the loss value of the loss function, the encoding model to be trained, the decoding model to be trained, and the target model to be trained are trained.

[0159] It should be noted that the text processing device of the embodiment of the present application solves the problem based on a principle similar to that of the aforementioned text processing method. Therefore, the implementation process and implementation principle of the text processing device can refer to the description of the implementation process and implementation principle of the aforementioned method, and the repeated parts will not be repeated.

[0160] The text processing device provided in the embodiment of the present application can be implemented as an electronic device, which can be a server or a terminal device.

[0161] The server may be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal device may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. The terminal device and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0162] According to an embodiment of the present application, the present application also provides an electronic device and a non-transitory computer-readable storage medium storing computer instructions.

[0163] The electronic device includes at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the text processing method of the present application. In a non-transitory computer-readable storage medium storing computer instructions, the computer instructions are used to perform the text processing method of the present application.

[0164] The present application also provides a computer program product, including a computer program / instruction, which implements the text processing method of the present application when executed by a processor.

[0165] Figure 6 A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0166] like Figure 6As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0167] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0168] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as text processing methods. For example, in some embodiments, the text processing method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the text processing method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the text processing method in any other appropriate manner (e.g., by means of firmware).

[0169] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0170] The program code for implementing the method of the present application can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, implements the functions / operations specified in the flow chart and / or block diagram. The program code can be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0171] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0172] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0173] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0174] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0175] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A text processing method, comprising: Get the first text to be processed; dividing the first to-be-processed text into a first text and a second text based on a preset condition; Encoding the first text to obtain target encoded data of the first text; Inputting the second text and the target coded data into the target model to obtain a second text to be processed; wherein the semantics of the second text to be processed are partially the same as those of the first text to be processed, and the amount of characters in the second text to be processed is less than that in the first text to be processed; The second text to be processed is input into a large language model to obtain a target result of the large language model performing natural language processing on the second text to be processed; the target result is used to reply to the first text to be processed.

2. The method according to claim 1, wherein dividing the first to-be-processed text into the first text and the second text based on a preset condition comprises: Detecting whether the first to-be-processed text contains text that needs to be kept confidential and / or needs to have a reduced amount of text; In response to detecting that the first to-be-processed text contains text that needs to be kept confidential and / or needs to have a reduced amount of text, The text that needs to be kept confidential and / or has to be reduced in length should be the first text; The text in the first to-be-processed text other than the text that needs to be kept confidential and / or needs to have a reduced amount of words is used as the second text.

3. The method according to claim 1 or 2, further comprising: Inputting the first text into a coding model to obtain output data of the coding model; Using the output data of the encoding model as target encoding data of the first text; The second text and the target coded data are input into the target model to obtain a second text to be processed; wherein the confidentiality of the second text to be processed to the text that needs to be kept confidential is stronger than the confidentiality of the first text to be processed to the text that needs to be kept confidential.

4. The method according to claim 1, further comprising: Obtain training data, wherein the training data includes N training texts and actual labels of each training text, wherein N is a positive integer greater than or equal to 2; Input N training texts into the encoding model to be trained to obtain encoding data of each training text; Inputting the encoded data of each training text into the decoding model to be trained to obtain N third texts; Inputting the encoded data of each training text and the N third texts into the target model to be trained to obtain N fourth texts; Inputting the N fourth texts into the large language model to obtain N prediction results of natural language prediction performed by the large language model on the N fourth texts; Based on the actual labels of the training texts and the N prediction results, the encoding model to be trained, the decoding model to be trained and the target model to be trained are trained.

5. The method according to claim 4, wherein the step of inputting the encoded data of each training text and the N third texts into the target model to be trained to obtain the N fourth texts comprises: Determine, among the coded data of the N training texts and the N third texts, the coded data and the third texts belonging to the same training text; Concatenate the encoded data belonging to the same training text and the third text to obtain N concatenated data; The N concatenated data are input into the target model to be trained to obtain N fourth texts.

6. The method according to claim 5, wherein the step of inputting the N concatenated data into the target model to be trained to obtain the N fourth texts comprises: Performing a mask operation on each spliced ​​data to obtain unmasked data and masked data in each spliced ​​data; The target model to be trained obtains N fourth texts by predicting the masked data using the unmasked data.

7. The method according to claim 4, 5 or 6, wherein the training of the encoding model to be trained, the decoding model to be trained and the target model to be trained based on the actual labels of each training text and the N prediction results comprises: Based on the actual labels of each training document and N prediction results, the loss value of the loss function is obtained; According to the loss value of the loss function, the encoding model to be trained, the decoding model to be trained, and the target model to be trained are trained.

8. A text processing device, comprising: A first obtaining unit, used for obtaining a first text to be processed; A division unit, configured to divide the first to-be-processed text into a first text and a second text based on a preset condition; A second obtaining unit, used for encoding the first text to obtain target encoding data of the first text; a third obtaining unit, configured to input the second text and the target coding data into a target model to obtain a second text to be processed; wherein the semantics of the second text to be processed are partially the same as those of the first text to be processed, and the amount of characters in the second text to be processed is less than that in the first text to be processed; The fourth obtaining unit is used to input the second text to be processed into the large language model to obtain a target result of the large language model performing natural language processing on the second text to be processed; the target result is used to reply the first text to be processed.

9. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to execute the method according to any one of claims 1-7.

Images