Text processing model training method, electronic device and program product

Abstract

The invention relates to artificial intelligence, and provides a training method of a text processing model, electronic equipment and a program product. The training method comprises the steps that each statement is reordered based on a code of each character of each statement in a first text, and a second text comprises a plurality of statements; selecting a first preset number of first statements in a second text according to the position sequence of the plurality of statements to construct a text list; selecting a second preset number of first statements from the text list as second statements; according to the similarity between third statements in the second text and the second statements, the third statements with the similarity with the second statements smaller than a preset threshold value are selected and added to the text list to obtain an updated text list, vocabulary replacement is conducted on the statements in the updated text list, and the first model is trained based on the text after vocabulary replacement. The method can improve the generalization ability of the model.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and specifically to a training method, electronic device, and program product for a text processing model. Background Art

[0002] Currently, the Internet is filled with a vast amount of text, and contains a large amount of repeated text. If there is a large amount of repeated text on the Internet, it will not only reduce the overall text quality, but also lead to a waste of storage resources.

[0003] To improve text quality and save storage resources, it is necessary to deduplicate text content on the internet. Related technologies typically compare multiple texts pairwise. However, this method is inefficient for deduplication, and the resulting text is not conducive to model training. Summary of the Invention

[0004] The present application provides a training method, electronic device and program product for a text processing model to solve the technical problem that deduplication efficiency is low and the obtained text is not conducive to the model training process.

[0005] In a first aspect, an embodiment of the present application provides a method for training a text processing model, the method comprising: reordering each sentence in a first text based on the encoding of each character of each sentence to obtain a second text, wherein the second text includes multiple sentences, and similar sentences in the multiple sentences are arranged adjacent to each other; selecting a first preset number of first sentences in the second text according to the position order of the multiple sentences to construct a text list; selecting a second preset number of first sentences in the text list as second sentences; selecting a third sentence in the second text whose similarity to the second sentence is less than a preset threshold and adding it to the text list to obtain an updated text list based on the similarity between the third sentence and the second sentence, wherein the third sentence is different from the first sentence; constructing a first prompt text based on a preset field and a text example including the vocabulary corresponding to the preset field; based on the first prompt text, using a preset language model, determining the vocabulary corresponding to the preset field from the sentences in the updated text list; replacing the vocabulary based on the field identifier corresponding to the preset field to obtain the text after the vocabulary is replaced; and training a first model based on the text after the vocabulary is replaced.

[0006] According to a second aspect of an embodiment of the present application, there is provided a training device for a text processing model, the device comprising: a sorting unit for reordering each sentence in a first text based on the encoding of each character of each sentence to obtain a second text, wherein the second text comprises multiple sentences, and similar sentences in the multiple sentences are arranged adjacently; a construction unit for selecting a first preset number of first sentences in the second text according to the position order of the multiple sentences to construct a text list; a determination unit for selecting a second preset number of first sentences in the text list as second sentences; a selection unit for selecting, based on the similarity between the third sentence in the second text and the second sentence, the third sentence having a similarity with the second sentence less than a preset threshold, and adding the third sentence to the text list to obtain an updated text list, wherein the third sentence is different from the first sentence; constructing a first prompt text based on a preset field and a text example including a vocabulary corresponding to the preset field; determining the vocabulary corresponding to the preset field from the sentences in the updated text list using a preset language model based on the first prompt text; replacing the vocabulary based on the field identifier corresponding to the preset field to obtain a text after the vocabulary is replaced; and training a first model based on the text after the vocabulary is replaced.

[0007] A third aspect of an embodiment of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method provided in the first aspect when executing the computer program.

[0008] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps in the method provided in the first aspect are implemented.

[0009] A fifth aspect of an embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the steps in the method provided in the first aspect.

[0010] In the training method of the text processing model of this embodiment, by reordering the characters in the first text, similar sentences in the second text can be made adjacent, and then by sorting the positions of the first sentences in the text list, the second sentence is determined from the first sentence, and the second sentence that is relatively similar to the third sentence can be initially screened out. Based on the similarity between the third sentence and the second sentence, the third sentence that is dissimilar to the second sentence can be determined, and then the third sentence that is dissimilar to the second sentence is added to the text list to obtain an updated text list. By performing vocabulary replacement on the sentences in the updated text list and training the first model based on the text after vocabulary replacement, it can be ensured that the text after vocabulary replacement can cover texts in most fields, avoiding the first model's dependence on texts in specific fields, thereby improving the model's generalization ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0012] Figure 1 This is a schematic diagram of an application scenario of a text processing model training method provided in an embodiment of the present application; Figure 2 This is a flowchart of a text deduplication method provided by an embodiment of the present application; Figure 3 This is a flowchart of a method for training a text processing model provided in an embodiment of the present application; Figure 4 Schematic diagram of a determination framework of a sentence after vocabulary replacement provided in an embodiment of the present application; Figure 5 This is a flowchart of another text processing model training method provided in an embodiment of the present application; Figure 6 This is a flowchart of a method for applying a text processing model provided in an embodiment of the present application; Figure 7 Schematic diagram of a determination framework of the second reply text provided in an embodiment of the present application; Figure 8 This is a functional module diagram of a text processing model training device provided in an embodiment of the present application; Figure 9 It is a structural diagram of an electronic device for implementing a training method for a text processing model provided in an embodiment of the present application. DETAILED DESCRIPTION

[0013] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0014] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or design schemes. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way.

[0015] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application belongs. The terms used in the specification of this application are for the purpose of describing specific embodiments only and are not intended to limit this application. It should be understood that, unless otherwise specified in this application, " / " means or. For example, A / B can mean A or B. "And / or" in this application is merely a way to describe the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. "At least one" means one or more. "Multiple" means two or more than two. For example, at least one of a, b or c can mean: a, b, c, a and b, a and c, b and c, a, b and c.

[0016] Some terminology explanations: Large Language Model (LLM): A large language model is a deep learning model trained using large amounts of text data. It can generate natural language text or understand the meaning of text. It can handle a variety of natural language tasks, such as text classification, question answering, and conversation.

[0017] Currently, the Internet is filled with a vast amount of text, and contains a large amount of repeated text. If there is a large amount of repeated text on the Internet, it will not only reduce the overall text quality, but also lead to a waste of storage resources.

[0018] To improve text quality and save storage resources, it is necessary to deduplicate text content on the internet. Related technologies typically compare multiple texts pairwise. However, this method is inefficient for deduplication, and the resulting text is not conducive to model training.

[0019] Based on the above problems, in order to improve the efficiency of deduplication and enhance the generalization ability of model training, an embodiment of the present application provides a training method for a text processing model.

[0020] See also Figure 1 , Figure 1 Schematic diagram of an application scenario of a text processing model training method provided in an embodiment of the present application. The scenario may include various electronic devices 100 and a server 200.

[0021] The electronic device 100 can be a mobile phone, a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal digital assistant (PDA), an augmented reality (AR) device, a virtual reality (VR) device, an artificial intelligence (AI) device, a wearable device, an in-vehicle device, a smart home device and / or a smart city device. The embodiments of the present application do not impose any special restrictions on the specific type of the electronic device 100.

[0022] Server 200 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, i.e., Content Delivery Network (CDN), as well as big data and artificial intelligence platforms, but is not limited to these.

[0023] It should be noted that the method in the embodiment of the present application can be executed separately by the electronic device 100 or the server 200, or can be executed jointly by the server 200 and the electronic device 100. When executed separately by the electronic device 100 or the server 200, the entire text processing process (text deduplication, model training and application process) can be implemented separately by the electronic device 100 or the server 200. For example, the first model can be trained on the electronic device 100 to obtain a trained first model. Accordingly, after training, the electronic device 100 can use the trained first model to obtain the first reply text to the user's question. For example, the second model can be trained on the electronic device 100 to obtain a trained second model. Accordingly, after training, the electronic device 100 can use the trained second model to determine the label corresponding to the first reply text. The above process can also be executed separately by the server 200. When the server 200 and the electronic device 100 jointly execute the process, the server 200 can train the first model and / or the second model and then deploy the trained first model and / or the trained second model to the electronic device 100. Part of the text processing process implemented by the electronic device 100 can be implemented by the electronic device 100, and part of the process can be implemented by the server 200. The two work together to implement the entire text processing process. In actual application, specific configurations can be made according to the situation and are not specifically limited here.

[0024] It should be noted that when the training process of the entire text processing model provided in the embodiment of the present application is performed separately by the server 200 or the electronic device 100, the above-mentioned application scenario may also only include any single device in the server 200 or the electronic device 100, or the server 200 and the electronic device 100 may also be considered to be the same device. In actual application, when the entire text processing process provided in the embodiment of the present application is jointly performed by the server 200 and the electronic device 100, the server 200 and the electronic device 100 may also be the same device, that is, the server 200 and the electronic device 100 may be different functional modules of the same device, or virtual devices virtualized from the same physical device.

[0025] In one possible implementation, the user may provide conversation data within a first preset time period and a fourth text within a second preset time period through the electronic device 100, and the server 200 may use the training method of an embodiment of the present application to determine the second reply text of the fourth text and return it to the electronic device 100 for presentation.

[0026] In the embodiment of the present application, the electronic device 100 and the server 200 can be directly or indirectly connected to each other through one or more networks. The network can be a wired network or a wireless network. For example, the wireless network can be a mobile cellular network or a Wireless Fidelity (Wi-Fi) network. Of course, it can also be other possible networks, and the embodiment of the present application does not limit this. It should be noted that Figure 1 The examples shown are just for illustration. In fact, the number of terminal devices and servers is not limited and is not specifically limited in the embodiments of this application.

[0027] The following describes the training method of the text processing model provided by the embodiment of the present application in combination with the application scenarios described above and with reference to the accompanying drawings. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principles of the embodiment of the present application, and the embodiment of the present application is not limited in this respect.

[0028] like Figure 2 FIG. 1 is a flowchart of a text deduplication method provided by an embodiment of the present application. The text deduplication method is applied to electronic devices, for example, Figure 1 The electronic device 100. According to different requirements, the order of the steps in the flowchart can be changed, and some steps can be omitted.

[0029] S201 : reorder each sentence in the first text based on the encoding of each character of each sentence to obtain a second text.

[0030] In at least one embodiment of the present application, training data may be obtained to train the first model. For example, conversation data from a question-and-answer scenario may be used as training data. The training data may include, but is not limited to, conversation data between a customer and an agent. The present embodiment does not limit the form of the conversation data; for example, the conversation data may be presented in the form of text, video, or voice.

[0031] In at least one embodiment of the present application, the first text may be the speech of any user in a question-and-answer scenario. In one example, in order for the first model to learn the agent's speech, the electronic device may extract the agent's speech from the conversation data between the customer and the agent, and use the extracted speech as the first text.

[0032] In at least one embodiment of the present application, in order to place similar sentences in a first text adjacent to each other, the electronic device may reorder each sentence in the first text based on the encoding of each character of each sentence in the first text to obtain a second text, in which similar sentences are arranged adjacent to each other.

[0033] In this embodiment, a mapping table can be provided, which includes characters and corresponding code point values. The electronic device can determine the encoding of each character in each sentence in the first text based on the mapping table. This embodiment uses the mapping table to quickly determine the encoding corresponding to each character.

[0034] In this embodiment, the electronic device can use the sorted() function to sort the codes corresponding to each sentence in the first text to obtain a string. According to the order of the codes in the string, each sentence in the first text is reordered to obtain a second text. The second text includes multiple sentences, and similar sentences in the multiple sentences are arranged adjacently. In this embodiment, by sorting the codes corresponding to each sentence in the first text, sentences with similar meanings can be arranged adjacently, thereby achieving similar sentences in the multiple sentences in the second text.

[0035] S202 : Selecting a first preset number of first sentences in the second text according to the position sequence of the plurality of sentences to construct a text list.

[0036] In at least one embodiment of the present application, in order to perform deduplication processing on the second text, a text list (also referred to as a "deduplication set") may be constructed.

[0037] In an embodiment of the present application, the electronic device may use at least one sentence in the second text as the first sentence.

[0038] In one example, the electronic device may use at least one sentence in the second text as the first sentence. For example, if the second text is: sentence A, sentence B, sentence C, sentence D, then the first sentence may include: sentence A, sentence B.

[0039] In another example, the electronic device may use the first sentence in the second text as the first sentence. For example, if the second text is: sentence A, sentence B, sentence C, sentence D, then the first sentence may include: sentence A.

[0040] In an embodiment of the present application, the electronic device can select a first preset number of first sentences in the second text to construct a text list based on the position order of the first sentences in the second text. The position order of the first sentences in the text list is the same as the position order of the first sentences in the second text. For example, if the second text is: sentence A, sentence B, sentence C, sentence D, and the first sentence can include: sentence A, sentence B, then the position order of the first sentence in the second text is: sentence A-sentence B. After construction, a text list is obtained, wherein the order of the first sentences in the text list is: sentence A-sentence B.

[0041] The embodiment of the present application constructs a text list according to the position order of the first sentences in the second text, thereby preventing the first sentences from being disrupted in the text list, thereby ensuring that first sentences with similar meanings can be arranged adjacent to each other in the text list.

[0042] S203: Select a second preset number of first sentences in the text list as second sentences.

[0043] In at least one embodiment of the present application, in order to reduce resource usage, a second preset number can be set. For example, the second preset number can be set to 1, 2, etc. Setting a larger second preset number can improve the deduplication effect of the second text, and setting a smaller second preset number can improve the deduplication efficiency of the second text. This application is not limited to this. The electronic device can select the second preset number of first sentences as the second sentence.

[0044] In at least one embodiment of the present application, the electronic device selecting a second preset number of first sentences from a text list as second sentences includes: selecting a second preset number of first sentences arranged at the end of the text list from the text list as second sentences. For example, if the second text is: sentence A, sentence B, sentence C, sentence D; three first sentences are selected from the second text to construct a text list, wherein the text list includes sentence B, sentence C, and sentence D. If two first sentences are selected from the text list as second sentences, then the second sentences include sentence C and sentence D.

[0045] In an embodiment of the present application, since similar sentences in the second text are arranged adjacent to each other, a second preset number of first sentences are selected as second sentences based on the position order of the first sentences in the second text and the position order of the first sentences in the text list. This can make the determined second sentence more similar to the third sentence in the second text, thereby avoiding invalid detection of the remaining sentences in the first sentence.

[0046] S204 : According to the similarity between the third sentence and the second sentence in the second text, select a third sentence whose similarity to the second sentence is less than a preset threshold and add it to the text list.

[0047] In at least one embodiment of the present application, the third statement is different from the first statement. For example, if the second text is: statement A, statement B, statement C, statement D, and the first statement is: statement A, then the third statement may include: statement B, statement C, statement D.

[0048] In at least one embodiment of the present application, in order to improve the accuracy of the similarity, the similarity between the third sentence and the second sentence may be determined based on multiple dimensions.

[0049] In an embodiment of the present application, the electronic device can calculate the character similarity between the third sentence and the second sentence, calculate the semantic similarity between the third sentence and the second sentence, and determine the similarity between the third sentence and the second sentence based on the character similarity and the semantic similarity.

[0050] In an embodiment of the present application, the electronic device can calculate the character similarity between the third sentence and the second sentence based on the edit distance formula. Specifically, the electronic device determines the minimum number of edit operations required to convert the third sentence into the second sentence, and determines the character similarity based on the minimum number of edit operations. The calculation formula for character similarity can be expressed as: ,in, It can represent character similarity. Can indicate the minimum number of editing operations, It can be set according to actual needs, for example, Can be set to any positive real number.

[0051] In an embodiment of the present application, the electronic device determines the semantic similarity between the third sentence and the second sentence using a preset language model, wherein the preset language model may include a large language model, a deep language model, etc. Specifically, the electronic device may construct a preset prompt text, which may include but is not limited to: identity information indicating that the preset language model is used to evaluate the semantics of the text, task information instructing the preset language model to output the semantic similarity between the third sentence and the second sentence, and examples for reference by the preset language model. In this embodiment of the present application, the preset language model can be used to quickly determine semantic similarity.

[0052] In another embodiment, the electronic device may determine a first coding feature corresponding to the third sentence and a second coding feature corresponding to the second sentence, and use the similarity between the first coding feature and the second coding feature as the semantic similarity.

[0053] In an embodiment of the present application, the electronic device can calculate the weighted sum of the character similarity and the semantic similarity to obtain the similarity between the third sentence and the second sentence. The weights corresponding to the character similarity and the semantic similarity can be set and adjusted according to actual needs.

[0054] The embodiment of the present application calculates the similarity between the third statement and the second statement by comprehensively considering multiple dimensions, thereby improving the accuracy of the similarity.

[0055] In at least one embodiment of the present application, a preset threshold may be set to identify repeated sentences in the second text. The preset threshold may be set and adjusted according to actual needs. The electronic device may determine that a third sentence having a similarity greater than or equal to the preset threshold is a third sentence similar to the second sentence. To avoid repeated sentences in the text list, the electronic device does not add the third sentence similar to the second sentence to the text list.

[0056] In an embodiment of the present application, the electronic device may determine that the third sentence whose similarity is less than a preset threshold is a third sentence that is dissimilar to the second sentence. To avoid missing information in the second text in the text list, the electronic device may add the third sentence that is dissimilar to the second sentence to the text list.

[0057] In an embodiment of the present application, in order to avoid the third statement from being repeatedly involved in the detection, the electronic device may delete the third statement involved in the detection.

[0058] For example, assuming the second sentence includes sentence A, and the third sentence includes sentence B, sentence C, and sentence D, the electronic device can calculate the similarity between sentence B and sentence A. Assuming the similarity between sentence B and sentence A is greater than or equal to a preset threshold, it can be indicated that sentence B and sentence A are similar sentences. Therefore, to avoid similar sentences in the text list, sentence B will not be added to the text list, and to prevent sentence B from being repeatedly detected, the electronic device can delete sentence B. In this case, the third sentence includes sentence C and sentence D.

[0059] Next, the electronic device can calculate the similarity between sentence C and sentence A. If the similarity between sentence C and sentence A is less than a preset threshold, it can be indicated that sentence C is not similar to sentence A. Therefore, sentence C can be added to the text list. To prevent sentence C from being repeatedly detected, the electronic device can delete sentence C. In this case, the third sentence includes sentence D. The above process is repeated until the third sentence does not include any sentences.

[0060] In multiple embodiments of the present application, by reordering the characters in the first text, similar sentences in the second text can be made adjacent, and then by sorting the positions of the first sentences in the text list, the second sentence can be determined from the first sentence, and the second sentence that is relatively similar to the third sentence can be initially screened out. Based on the similarity between the third sentence and the second sentence, the third sentence that is dissimilar to the second sentence can be determined, and then the third sentence that is dissimilar to the second sentence can be added to the text list. Since there is no need to compare the third sentence with all the first sentences, the updating efficiency of the text list can be improved, thereby improving the deduplication efficiency of the first text.

[0061] like Figure 3 FIG. 1 is a flowchart of a text processing model training method provided by an embodiment of the present application. The text processing model training method is applied to electronic devices, for example, Figure 1 The electronic device 100. According to different requirements, the order of the steps in the flowchart can be changed, and some steps can be omitted.

[0062] S301 : reorder each sentence based on the encoding of each character of each sentence in the first text to obtain a second text.

[0063] S302 : Select a first preset number of first sentences in the second text according to the position sequence of the plurality of sentences to construct a text list.

[0064] S303: Select a second preset number of first sentences in the text list as second sentences.

[0065] S304 : According to the similarity between the third sentence and the second sentence in the second text, select a third sentence whose similarity to the second sentence is less than a preset threshold and add it to the text list.

[0066] The details of steps S301 to S304 can be found above. Figure 2 The detailed description of steps S201 to S204 is not repeated here.

[0067] S305: Construct a first prompt text based on the preset field and the text example including the vocabulary corresponding to the preset field.

[0068] In at least one embodiment of the present application, the conversation data in a question-and-answer scenario typically includes corresponding variable information. For example, in a telemarketing scenario, the conversation data typically includes the customer's name and gender. Telemarketing scenarios may also include variable information related to specific activities, such as the available withdrawal limit and interest rate.

[0069] In at least one embodiment of the present application, since the variable information in the dialogue data corresponding to different question-and-answer scenarios is different, in order to reduce the first model's dependence on the dialogue data of a specific scenario, the electronic device can perform vocabulary replacement on the variable information in the dialogue data.

[0070] In at least one embodiment of the present application, during the vocabulary replacement process, the electronic device constructs a first prompt text based on a preset field and a text example including the vocabulary corresponding to the preset field.

[0071] In an embodiment of the present application, the preset fields may include but are not limited to: fields corresponding to variable information such as amount, name, activity information, date, etc. that changes due to different question-and-answer scenarios.

[0072] In an embodiment of the present application, the first prompt text may include: identity information indicating the preset language model as a text annotation expert. For example, you are now a text annotation expert, and your task is to annotate relevant fields in the text given the required annotation fields and examples.

[0073] The first prompt text may also include task information instructing a preset language model to output vocabulary corresponding to a preset field. For example, the required annotation fields include surname, gender, mobile phone number, and application date. The annotation rule is to identify relevant fields in the text and annotate the fields according to the following rules: surname: ,gender: 、Mobile phone number: 、Application date: .

[0074] The first prompt text may also include text examples that include vocabulary corresponding to the preset field (also referred to as "examples for reference by the preset language model"). Examples for reference by the preset language model may include, but are not limited to, examples that require vocabulary replacement and examples that do not require vocabulary replacement. Examples for reference by the preset language model may include, but are not limited to, text, thought processes, and output annotation results.

[0075] For example, examples that require lexical replacement could be: Text: "Yes, I am not mistaken sir. You applied for it yesterday on the mini program. I also saw that your credit limit has been approved. You can use it directly, okay?"

[0076] Thinking process: The sentence 'Yes, that's right, Mr.' mentions gender: 'Mr.', and the sentence 'You applied yesterday on the mini program' mentions application date: 'yesterday'. Therefore, the text involves the annotation fields: Gender: 'Mr.', Application date: 'yesterday'. Replace 'Mr.' in the text with ' ', 'yesterday' is replaced by ' '. The result of annotation is: "Yes, I am not mistaken. , here you are Well, you applied on the mini program, and I can see that your credit limit has been approved. You can use it directly, okay?"

[0077] Output annotation process: "Yes, I am not wrong. , here you are Well, you applied on the mini program, and I can see that your credit limit has been approved. You can use it directly, okay?"

[0078] For example, an example that does not require lexical replacement could be: Text: "Oh, so you are not satisfied with the interest?"

[0079] Thinking process: The sentence "Oh, so you're not satisfied with the interest rate?" does not mention the characters related to surname, gender, mobile phone number, or application date. Therefore, no annotation is required, and the result is "No annotation required." The annotation result is: No annotation required.

[0080] Output annotation process: No annotation is required.

[0081] S306: Based on the first prompt text, using a preset language model, determine the vocabulary corresponding to the preset field from the sentences in the updated text list.

[0082] In at least one embodiment of the present application, an electronic device inputs a first prompt text into a preset language model to obtain a first feedback text output by the preset language model. When the first feedback text indicates that the preset language model has learned the text example in the first prompt text, a sentence in an updated text list is input into the preset language model to obtain a vocabulary corresponding to a preset field. The vocabulary corresponding to the preset field may be variable information corresponding to the preset field in the sentence in the updated text list. For example, assuming the preset field is gender, the vocabulary corresponding to the preset field may be: Mr.

[0083] For example, if the first feedback text output by the preset language model is "Okay, I learned it, I will directly output the annotation result", it can mean that the first feedback text indicates that the preset language model has learned the example in the first prompt text.

[0084] In other embodiments, when the first feedback text indicates that the preset language model has not learned the examples in the first prompt text, the electronic device may add new examples in the first prompt text for reference by the preset language model.

[0085] S307: Replace the vocabulary based on the field identifier corresponding to the preset field to obtain a text after the vocabulary is replaced.

[0086] In at least one embodiment of the present application, the field identifier corresponding to the preset field can be set and adjusted according to actual needs. For example, the field identifier corresponding to the preset field "gender" can be set as: , the field identifier corresponding to the preset field "date" can be set as: .

[0087] In at least one embodiment of the present application, the electronic device inputs the sentences in the updated text list into a preset language model to obtain vocabulary corresponding to the preset field. The electronic device invokes the preset language model to replace the vocabulary corresponding to the preset field with the field identifier corresponding to the preset field, and uses the second feedback text output by the preset language model as the text after the vocabulary replacement.

[0088] Combine Figure 4Explain the process of determining the sentence after vocabulary replacement, such as Figure 4 As shown, the deduplicated data (also called "sentences in the updated text list") and scenario variable information (also called "preset fields") are input into the large model (also called the "preset language model"). The large model replaces the relevant variables in the deduplicated data (also called "vocabulary corresponding to the preset fields") to obtain the template speech (also called "text after vocabulary replacement").

[0089] The embodiment of the present application can assist the preset language model in learning to mark out the words corresponding to the preset fields in the sentences in the updated text list through the first prompt text, so that the text after the vocabulary replacement can be quickly obtained.

[0090] For example, suppose the conversation data of one scenario is "Hey, because our current product F has a maximum limit of 200,000, then your current available limit is 2,000 yuan, right?". Suppose the conversation data of another scenario is "Hey, because our current product Q has a maximum limit of 200,000, then your current available limit is 5,000 yuan, right?". In order to reduce the dependence of the first model on the conversation data of a specific scenario, the electronic device can perform vocabulary replacement on the variable information in the conversation data. For example, the conversation data after vocabulary replacement is "Hey, because our current product Q has a maximum limit of 200,000, then your current available limit is 5,000 yuan, right?". Well, the maximum limit has been updated to 200,000, so what is your current available limit? Yuan, right? ".

[0091] S308: Training a first model based on the text after vocabulary replacement.

[0092] In at least one embodiment of the present application, the electronic device trains a first model based on the text after vocabulary replacement. In this embodiment, because the text after vocabulary replacement covers conversation data in most fields, using the text after vocabulary replacement to train the first model can reduce the first model's reliance on training data, thereby improving the first model's generalization ability.

[0093] In multiple embodiments of the present application, by reordering the characters in the first text, similar sentences in the second text can be made adjacent, and then by sorting the positions of the first sentences in the text list, the second sentence can be determined from the first sentence, and the second sentence that is relatively similar to the third sentence can be initially screened out. Based on the similarity between the third sentence and the second sentence, the third sentence that is dissimilar to the second sentence can be determined, and then the third sentence that is dissimilar to the second sentence can be added to the text list. Since there is no need to compare the third sentence with all the first sentences, the updating efficiency of the text list can be improved, thereby improving the training efficiency of the first model. In addition, by performing vocabulary replacement on the sentences in the updated text list and training the first model based on the text after vocabulary replacement, it can be ensured that the text after vocabulary replacement can cover texts in most fields, avoiding the first model's dependence on texts in specific fields, thereby improving the model's generalization ability.

[0094] like Figure 5 FIG. 1 is a flowchart of another text processing model training method provided by an embodiment of the present application. The text processing model training method is applied to electronic devices, for example, Figure 1 The electronic device 100. According to different requirements, the order of the steps in the flowchart can be changed, and some steps can be omitted.

[0095] S501 : reorder each sentence in the first text based on the encoding of each character of each sentence to obtain a second text.

[0096] S502 : Select a first preset number of first sentences in the second text according to the position sequence of the plurality of sentences to construct a text list.

[0097] S503: Select a second preset number of first sentences in the text list as second sentences.

[0098] S504 : According to the similarity between the third sentence and the second sentence in the second text, select a third sentence whose similarity to the second sentence is less than a preset threshold and add it to the text list.

[0099] The details of steps S501 to S504 can be found above. Figure 2 The detailed description of steps S201 to S204 is not repeated here.

[0100] S505 , clustering the sentences in the updated text list to obtain sentences corresponding to multiple categories.

[0101] In at least one embodiment of the present application, the electronic device encodes the sentences in the updated text list to obtain the coding features corresponding to each sentence. At least two coding features are randomly selected from a plurality of coding features as the centroid, the distance between the coding feature corresponding to each sentence and the centroid is calculated, and each sentence is assigned to the category corresponding to the centroid closest to the distance. Based on the coding features of all sentences in the category corresponding to the centroid, the updated centroid is determined. If the change value between the distance between the coding feature corresponding to each sentence and the updated centroid and the distance between the coding feature corresponding to each sentence and the centroid is greater than or equal to a set threshold, all sentences in the category corresponding to the centroid are reallocated until the change value between the distance between the coding feature corresponding to each sentence and the updated centroid and the distance between the coding feature corresponding to each sentence and the centroid is less than a set threshold. If the change value between the distance between the coding feature corresponding to each sentence and the updated centroid and the distance between the coding feature corresponding to each sentence and the centroid is less than a set threshold, sentences corresponding to multiple categories are determined based on the sentences in the category corresponding to the updated centroid.

[0102] In an embodiment of the present application, in the process of determining the updated centroid, the electronic device may perform bit-by-bit calculations on the coding features of all sentences in the category corresponding to the centroid to obtain the updated centroid. In one example, the electronic device may calculate the average value of the corresponding feature elements in all coding features to obtain the updated centroid. For example, a category includes sentence A and sentence B, the coding features of sentence A are [a1, a2, a3, …, an], and the coding features of sentence B are [b1, b2, b3, …, bn]. After calculation, the updated centroid can be obtained as [(a1+b1) / 2, (a2+b2) / 2, (a3+b3) / 2, …, (an+bn) / 2]. In another example, the electronic device may calculate the weighted sum of the corresponding feature elements in all coding features to obtain the updated centroid.

[0103] In an embodiment of the present application, the electronic device can use a similarity calculation formula to calculate the distance between the coding feature corresponding to each sentence and the updated centroid, and calculate the distance between the coding feature corresponding to each sentence and the centroid. The present application does not impose any specific restrictions on the similarity calculation formula.

[0104] S506 , sorting the sentences corresponding to each category based on the distance between the sentences corresponding to each category and the centroid corresponding to each category to obtain a third text.

[0105] In at least one embodiment of the present application, in order to make adjacent sentences more similar, the electronic device can sort the sentences corresponding to each category in ascending order based on the distance between the sentences corresponding to each category and the centroid corresponding to each category to obtain a third text.

[0106] For example, suppose a category includes sentence A and sentence B, the encoding features of sentence A are [a1, a2, a3, ..., an], the encoding features of sentence B are [b1, b2, b3, ..., bn], and the centroid is [(a1+b1) / 2, (a2+b2) / 2, (a3+b3) / 2, ..., (an+bn) / 2]. After calculation, the encoding features of sentence A [a1, a2, a3, ..., an] and the centroid are [(a1+b1) / 2, (a2+b2) / 2, (a3+b3) / 2, ..., (an+bn) / 2]. The distance between the centroid [(a1+b1) / 2,(a2+b2) / 2,(a3+b3) / 2,…,(an+bn) / 2] is 3, and the distance between the encoded features of sentence B [b1,b2,b3,…,bn] and the centroid [(a1+b1) / 2,(a2+b2) / 2,(a3+b3) / 2,…,(an+bn) / 2] is 2, then the third text can be obtained as {sentence B, sentence A}.

[0107] This embodiment sorts the sentences corresponding to each category, so that adjacent sentences in the third text are made more similar, and the divergence between sentences is reduced, which is helpful for model training.

[0108] S507: Construct a second prompt text based on the sentences in the third text for which label recognition has been completed.

[0109] In at least one embodiment of the present application, sentences in the third text for which label recognition has been completed may be provided for reference by a preset language model, so that the preset language model learns how to perform label recognition on the sentences.

[0110] In at least one embodiment of the present application, the electronic device constructs a second prompt text based on the sentences for which label recognition has been completed in the third text, including: determining multiple sentences from the sentences for which label recognition has been completed based on a preset time window; and constructing the second prompt text based on the multiple sentences and the corresponding text labels.

[0111] In an embodiment of the present application, to improve the tag recognition performance of a preset language model, the electronic device may set a preset time window. The electronic device may determine the recognition time of sentences for which tag recognition has been completed, and use sentences whose recognition time falls within the preset time window as multiple sentences for reference by the preset language model.

[0112] In an embodiment of the present application, the electronic device may use the determined multiple sentences and corresponding text labels as examples in the second prompt text for reference by a preset language model.

[0113] For example, the example for reference of the preset language model in the second prompt text may be: Text: "Hello Mr. Ren, I am a staff member of XX. I am calling to confirm with you that you are the one using YY, right?"

[0114] Thinking process: First, we divide the text into the following segments according to semantics: "1. Hello Mr. Ren, I am a staff member of XX. 2. I am calling to confirm with you that you are the one using YY, right?"

[0115] So, we end up with two labels! Then, we summarize each sentence one by one: 'Hello, Mr. Ren, I'm from XX.' This sentence introduces me, so the first label is: "Self-introduction." 'I'm calling to confirm that you're using YY, right?' This sentence confirms that the customer is actually using the product, so the second label is: "Confirm account user."

[0116] Output tag: "Hello, Mr. Ren, I'm a staff member of XX," belongs to "Self-introduction." "I'm calling to confirm that you are the one using YY, right?" belongs to "Confirm account user."

[0117] The second prompt text may also include: identity information indicating the preset language model as an operations expert. For example, you are a senior telephone sales operations expert, and your current task is to analyze a given marketing text by referring to background information and give corresponding speech tags.

[0118] The second prompt text may also include task information instructing the preset language model to output vocabulary corresponding to the speech label. For example, when sorting out the speech labels, the labels given should be as complete as possible, and the speech labels typed out should represent the overall meaning of the text. If the text is short, summarizing it with a single speech label is sufficient. When sorting out the speech labels, it is necessary to appropriately split the speech labels according to the semantic relationship of the words and the logical relationship between the preceding and following sentences. The speech labels are then summarized for the split short sentences in turn, and the speech labels can be appropriately referenced from historical conversations. The label content should be summarized as concisely and clearly as possible, but key information should not be omitted. Variable information such as specific time, date, and amount should be abstracted, and specific values ​​should not be directly output. The sorted speech labels should include specific conversation details, such as the main content of the activity and the specific questions that are addressed by the customer.

[0119] The embodiment of the present application determines multiple sentences from the sentences that have completed label recognition through a preset time window, which can avoid using all the sentences that have completed label recognition as the second prompt text, thereby improving the recognition efficiency of the preset language model for the second prompt text. In addition, based on the sentences that have completed label recognition and the corresponding text labels, the second prompt text is constructed, which can improve the recognition ability of the preset recognition model for text labels.

[0120] S508: Based on the second prompt text, using a preset language model, identify the text label corresponding to the sentence of each category.

[0121] In at least one embodiment of the present application, the electronic device inputs the second prompt text into a preset language model to obtain a third feedback text output by the preset language model. When the third feedback text indicates that the preset language model has learned the examples in the second prompt text, sentences of each category are input into the preset language model to obtain text labels corresponding to the sentences of each category.

[0122] For example, if the third feedback text output by the preset language model is "Okay, I learned it. I will directly output the speech tags. Let's get started.", it can mean that the third feedback text indicates that the preset language model has learned the example in the second prompt text.

[0123] In other embodiments, when the third feedback text indicates that the preset language model has not learned the examples in the first prompt text, the electronic device can increase the window length of the preset time window and add new examples for reference by the preset language model in the second prompt text.

[0124] S509: Training a second model based on the sentences of each category and the corresponding text labels.

[0125] In at least one embodiment of the present application, an electronic device can construct training samples based on multiple categories of sentences and corresponding text labels, and the training samples can include positive samples and negative samples. Positive samples can include multiple categories of sentences and corresponding text labels, and negative samples can include multiple categories of sentences and other labels, and the other labels can be text labels in the training samples that are different from the positive samples. For example, for sentence A, the text label corresponding to sentence A is label Z. Assuming that the text labels corresponding to multiple categories of sentences include: label X, label Y, label Z, the electronic device can construct a positive sample based on sentence A and label Z, a negative sample based on sentence A and label X, and another negative sample based on sentence A and label Y.

[0126] In at least one embodiment of the present application, the electronic device uses the second model to obtain a prediction result based on the sentences in the training sample. The electronic device calculates a loss value of the second model based on the similarity between the text labels in the positive sample and the prediction result and the similarity between the text labels in the negative sample and the prediction result.

[0127] In the embodiment of the present application, the calculation formula of the loss value of the second model can be expressed as: .in, Can represent the loss value of the second model, Can represent the encoding features of the prediction results, It can represent the encoding features of the text labels in the positive samples, It can represent the similarity between the text label in the positive sample and the predicted result. It can represent the encoding features of the text labels in the negative samples, It can represent the similarity between the text label in the negative sample and the predicted result. It can be set and adjusted according to actual needs, for example, It can be set to 0.05, and this application does not impose any specific limitation on this.

[0128] In at least one embodiment of the present application, the electronic device adjusts the second model based on the loss value of the second model until the loss value reaches a first preset condition. The first preset condition may include, but is not limited to: the loss value no longer decreases, the loss value is less than or equal to a preset loss threshold, and the preset loss threshold can be set and adjusted as needed.

[0129] In another embodiment, the electronic device may further adjust the second model based on the loss value until the number of adjustments or the learning rate reaches a second preset condition, wherein the second preset condition may include but is not limited to: the number of adjustments reaching a preset number or the learning rate reaching a preset value.

[0130] In multiple embodiments of the present application, by clustering the sentences in the updated second list and sorting the sentences corresponding to each category to obtain a third text, adjacent sentences in the third text can be made more similar, thereby improving the recognition effect of the preset language model on the sentences in the third text. By constructing a second prompt text through the sentences in the third text that have completed label recognition, the preset language model can be assisted in learning the examples in the second prompt text, thereby improving the prediction accuracy of the text label. The second model is trained by the sentences of each category and the corresponding text labels. Since there is no need to manually mark the text labels corresponding to the sentences, the training efficiency of the second model can be improved. In addition, by training the second model, without changing the first model, it is possible to support the business party to quickly iterate the strategy words according to the labels.

[0131] like Figure 6 FIG. 1 is a flowchart of a method for applying a text processing model provided by an embodiment of the present application. The training method of the text processing model is applied to electronic devices, for example, Figure 1 The electronic device 100. According to different requirements, the order of the steps in the flowchart can be changed, and some steps can be omitted.

[0132] S601, based on the conversation data within the first preset time period and the fourth text within the second preset time period, using the first model, obtain a first reply text of the fourth text.

[0133] In at least one embodiment of the present application, the second preset time period is after the first preset time period. The electronic device inputs the conversation data and the fourth text into the first model to obtain a first reply text to the fourth text.

[0134] S602: Based on the first reply text, use the second model to determine the label corresponding to the first reply text.

[0135] In at least one embodiment of the present application, the electronic device inputs the first reply text into the second model to obtain a label corresponding to the first reply text.

[0136] S603: Determine the type of the fourth text based on the label corresponding to the first reply text.

[0137] In at least one embodiment of the present application, a preset tag may be set to determine the type of the fourth text. For example, the preset tag may be set to "interest answer." The electronic device compares the tag corresponding to the first reply text with the preset tag. If the tag corresponding to the first reply text is the preset tag, the type of the fourth text may be determined to be the preset type. If the tag corresponding to the first reply text is not the preset tag, the type of the fourth text may be determined to be not the preset type.

[0138] S604: Determine a second reply text for the fourth text based on the type of the fourth text.

[0139] In at least one embodiment of the present application, if the type of the fourth text is a preset type, the electronic device determines a second reply text of the fourth text based on the type of the fourth text, including: determining a text library matching the fourth text based on the label corresponding to the first reply text; calculating a first similarity between the preset text in the text library and the fourth text; determining the preset text whose first similarity is greater than a first threshold as the target text; and obtaining a reply text corresponding to the target text from the text library as the second reply text.

[0140] In the embodiment of the present application, the label corresponding to the first reply text corresponds to the text library. The electronic device can determine the text library that matches the fourth text based on the label corresponding to the first reply text. In order to prevent the second reply text from involving sensitive information such as personal privacy, the multiple question-answer pairs included in the text library do not include variable information corresponding to the preset fields. The vocabulary corresponding to the preset fields in the multiple question-answer pairs is presented with a preset identifier. For example, the answer in the text library can be set to: "Hey, because we are now Well, the maximum limit has been updated to 200,000, so what is your current available limit? Yuan, right? ". Each question-answer pair includes a preset text and a corresponding answer text.

[0141] In the embodiment of the present application, the electronic device can calculate the first similarity between the preset text in the text library and the fourth text using a similarity calculation formula. The embodiment of the present application does not impose any specific limitation on the similarity calculation formula.

[0142] In an embodiment of the present application, a first threshold value may be set to select a target text that matches the fourth text from the preset texts. The value of the first threshold value may be set and adjusted based on actual needs. In one example, the electronic device determines the preset text whose first similarity is greater than the first threshold value as the target text. In another example, the electronic device may determine the preset text with the greatest first similarity as the target text.

[0143] In the embodiment of the present application, when the type of the fourth text is a preset type, the text library that matches the fourth text can be accurately determined based on the label corresponding to the first reply text. A target text similar to the fourth text can be screened based on the first similarity between the preset text in the text library and the fourth text, and the second reply text can be accurately determined based on the reply text in the text library that corresponds to the target text.

[0144] In at least one embodiment of the present application, if the type of the fourth text is not a preset type, the electronic device determines the second reply text of the fourth text based on the type of the fourth text, and also includes: obtaining a candidate reply text corresponding to the label of the first reply text from a preset library; calculating the second similarity between the candidate reply text and the fourth text; and determining the candidate reply text whose second similarity is greater than a second threshold as the second reply text.

[0145] In an embodiment of the present application, to prevent the second reply text from involving sensitive information such as personal privacy, the multiple reply texts included in the preset library do not include variable information corresponding to the preset fields, and the vocabulary corresponding to the preset fields in the multiple reply texts is presented with a preset identifier. The preset library includes multiple reply texts and corresponding tags. Based on the tag corresponding to the first reply text, the electronic device can obtain a reply text corresponding to the tag of the first reply text from the preset library as a candidate reply text.

[0146] In the embodiment of the present application, the electronic device can calculate the second similarity between the candidate reply text and the fourth text using a similarity calculation formula. The embodiment of the present application does not impose any specific limitation on the similarity calculation formula.

[0147] In an embodiment of the present application, in order to filter out the second reply text that matches the fourth text from the candidate reply texts, a second threshold value can be set, and the value of the second threshold value can be set and adjusted according to actual needs. In one example, the electronic device determines the candidate reply text whose second similarity is greater than the second threshold value as the second reply text. In another example, the electronic device can determine the candidate reply text with the greatest second similarity as the second reply text.

[0148] In the embodiment of the present application, when the type of the fourth text is not the preset type, a candidate reply text of the fourth text can be obtained from a preset library based on the label of the first reply text, and then the second reply text of the fourth text can be accurately determined through the second similarity between the candidate reply text and the fourth text.

[0149] Combine Figure 7 To illustrate the process of determining the second reply text, the electronic device inputs the historical conversation (also referred to as "conversation data within the first preset time period") and the user's current question (also referred to as "the fourth text within the second preset time period") into a large conversation model (also referred to as the "first model"). The large conversation model outputs the answer to the current question (also referred to as the "first reply text"). The electronic device inputs the answer to the current question into a retrieval model (also referred to as the "second model"). The retrieval model outputs the optimal label (also referred to as the "label corresponding to the first reply text") and determines whether the optimal label is a feature label (also referred to as the "preset label"). If the optimal label is a special label, the electronic device uses the retrieval model to retrieve the special reply script (also referred to as the "second reply text") corresponding to the user's current question from a special label question and answer library (also referred to as the "text library matching the fourth text"). If the optimal label is not a special label, the electronic device retrieves the script corresponding to the optimal label from a script library (also referred to as the "preset library") as the optimal script (also referred to as the "second reply text").

[0150] In at least one embodiment of the present application, the electronic device determines the field information corresponding to the preset field from the second reply text, replaces the field information with the corresponding variable information, obtains the target reply text, and sends the target reply text to the user terminal.

[0151] In the training method of this embodiment, the first model is used to determine the first reply text of the fourth text, and the second model is used to determine the label corresponding to the first reply text. Based on the label corresponding to the first reply text, the type of the fourth text can be determined, and then based on the type of the fourth text, the second reply text of the fourth text can be determined, which can improve the accuracy of the second reply text.

[0152] like Figure 8 , which is a functional module diagram of a text processing model training device provided by an embodiment of the present application. The text processing model training device 81 includes a sorting unit 810, a construction unit 811, a determination unit 812, a selection unit 813, a replacement unit 814, a training unit 815, a clustering unit 816, and a recognition unit 817. The module / unit referred to in this application refers to a type of training unit that can be processed by a processor (e.g., Figure 9A series of computer-readable instruction segments are obtained by the processor 901 shown in FIG. 1 and are capable of completing fixed functions, which are stored in a memory (eg, Figure 9 902).

[0153] In one embodiment, a sorting unit 810 is used to reorder each sentence in the first text based on the encoding of each character of each sentence to obtain a second text; a construction unit 811 is used to select a first preset number of first sentences in the second text according to the position order of the multiple sentences to construct a text list; a determination unit 812 is used to select a second preset number of first sentences in the text list as second sentences; and a selection unit 813 is used to select a third sentence whose similarity to the second sentence is less than a preset threshold and add it to the text list based on the similarity between the third sentence and the second sentence in the second text, and the third sentence is different from the first sentence.

[0154] In one embodiment, the determining unit 812 is specifically configured to select, from the text list, a second preset number of first sentences arranged at the end of the text list as the second sentences.

[0155] In one embodiment, the selection unit 813 is specifically used to: calculate the character similarity between the third sentence and the second sentence; calculate the semantic similarity between the third sentence and the second sentence; determine the similarity between the third sentence and the second sentence based on the character similarity and the semantic similarity; determine the third sentence whose similarity is less than a preset threshold as a third sentence that is dissimilar to the second sentence, and add the third sentence that is dissimilar to the second sentence to the text list.

[0156] In one embodiment, the construction unit 811 is further used to construct a first prompt text based on a preset field and a text example including vocabulary corresponding to the preset field; the determination unit 812 is further used to determine the vocabulary corresponding to the preset field from the sentences in the updated text list based on the first prompt text using a preset language model; the replacement unit 814 is used to replace the vocabulary based on the field identifier corresponding to the preset field to obtain the text after the vocabulary is replaced; the training unit 815 is used to train the first model based on the text after the vocabulary is replaced.

[0157] In one embodiment, the clustering unit 816 is used to cluster the sentences in the updated text list to obtain sentences corresponding to multiple categories; the sorting unit 810 is also used to sort the sentences corresponding to each category based on the distance between the sentences corresponding to each category and the centroid corresponding to each category to obtain a third text; the construction unit 811 is also used to construct a second prompt text based on the sentences in the third text that have completed label recognition; the recognition unit 817 is used to identify the text labels corresponding to the sentences in each category based on the second prompt text using a preset language model; the training unit 815 is also used to train the second model based on the sentences in each category and the corresponding text labels.

[0158] In one embodiment, the constructing unit 811 is specifically configured to: determine, based on a preset time window, a plurality of sentences from the sentences for which tag recognition has been completed; and construct a second prompt text based on the plurality of sentences and the corresponding text tags.

[0159] In one embodiment, the determination unit 812 is also used to obtain a first reply text of the fourth text using the first model based on the conversation data within the first preset time period and the fourth text within the second preset time period, and the second preset time period is after the first preset time period; the determination unit 812 is also used to determine a label corresponding to the first reply text based on the first reply text using the second model; the determination unit 812 is also used to determine the type of the fourth text based on the label corresponding to the first reply text; the determination unit 812 is also used to determine the second reply text of the fourth text based on the type of the fourth text.

[0160] In one embodiment, the determination unit 812 is specifically used to: if the type of the fourth text is a preset type, determine a text library that matches the fourth text based on the label corresponding to the first reply text; calculate the first similarity between the preset text in the text library and the fourth text; determine the preset text whose first similarity is greater than the first threshold as the target text; and obtain the reply text corresponding to the target text from the text library as the second reply text.

[0161] In one embodiment, the determination unit 812 is specifically used to: if the type of the fourth text is not a preset type, obtain a candidate reply text corresponding to the label of the first reply text from a preset library; calculate the second similarity between the candidate reply text and the fourth text; and determine the candidate reply text whose second similarity is greater than the second threshold as the second reply text.

[0162] In multiple embodiments of the present application, by reordering the characters in the first text, similar sentences in the second text can be made adjacent, and then by sorting the positions of the first sentences in the text list, the second sentence can be determined from the first sentence, and the second sentence that is relatively similar to the third sentence can be initially screened out. Based on the similarity between the third sentence and the second sentence, the third sentence that is dissimilar to the second sentence can be determined, and then the third sentence that is dissimilar to the second sentence can be added to the text list. Since there is no need to compare the third sentence with all the first sentences, the updating efficiency of the text list can be improved, thereby improving the deduplication efficiency of the first text.

[0163] Figure 9 is a schematic diagram of the structure of an electronic device for implementing a training method for a text processing model provided in an embodiment of the present application. Figure 9 The electronic device 100 is used to perform Figures 2 to 7 The method shown.

[0164] The electronic device 100 includes at least one processor 901 , a memory 902 , and at least one network interface 903 .

[0165] The processor 901 is, for example, a general-purpose central processing unit (CPU), a network processor (NP), a graphics processing unit (GPU), a neural-network processing unit (NPU), a data processing unit (DPU), a microprocessor, or one or more integrated circuits for implementing the solution of the present application. For example, the processor 901 includes an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD is, for example, a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0166] The memory 902 may be, for example, a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, a random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, an optical disc storage (including a compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto. Optionally, the memory 902 exists independently and is connected to the processor 901 via the internal connection 904. Alternatively, the memory 902 and the processor 901 may be integrated together.

[0167] The network interface 903 uses any transceiver-like device for communicating with other devices or communication networks. For example, the network interface 903 includes at least one of a wired network interface and a wireless network interface. For example, the wired network interface is an Ethernet interface. For example, the Ethernet interface is an optical interface, an electrical interface, or a combination thereof. For example, the wireless network interface is a wireless local area network (WLAN) interface, a cellular network interface, or a combination thereof.

[0168] In some embodiments, the processor 901 includes one or more CPUs, such as Figure 9 CPU0 and CPU1 are shown in the figure.

[0169] In some embodiments, the electronic device 100 optionally includes multiple processors, such as Figure 9 901 and processor 905 are shown in FIG. Each of these processors is, for example, a single-CPU or a multi-CPU. A processor herein optionally refers to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0170] In some embodiments, electronic device 100 further includes internal connections 904. Processor 901, memory 902, and at least one network interface 903 are connected via internal connections 904. Internal connections 904 include pathways for transmitting information between the aforementioned components. Internal connections 904 may optionally be a single board or bus. Internal connections 904 may optionally be classified as an address bus, a data bus, a control bus, and the like.

[0171] In some embodiments, the electronic device 100 further includes an input / output interface 906 . The input / output interface 906 is connected to the internal connection 904 .

[0172] Optionally, the processor 901 implements the method in the above embodiment by reading the program code 910 stored in the memory 902, or the processor 901 implements the method in the above embodiment by internally stored program code. In the case where the processor 901 implements the method in the above embodiment by reading the program code 910 stored in the memory 902, the memory 902 stores the program code that implements the method provided in the embodiment of the present application.

[0173] For more details on how the processor 901 implements the above functions, please refer to the descriptions in the previous method embodiments, which will not be repeated here.

[0174] This embodiment also provides a computer storage medium, which stores computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the training method in the above-mentioned embodiment.

[0175] This embodiment also provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes the above-mentioned related steps to implement the training method in the above-mentioned embodiment.

[0176] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor can execute the computer-executable instructions stored in the memory to enable the chip to execute the training method in the above-mentioned method embodiments.

[0177] Among them, the electronic device, computer storage medium, computer program product or chip provided in this embodiment is used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above, and will not be repeated here.

[0178] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0179] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0180] The units described as separate components may or may not be physically separate, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0181] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0182] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a device (which can be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program code.

[0183] The above are only specific embodiments of the present application, but the scope of protection of the present application is not limited thereto. Any changes or replacements within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A training method for a text processing model, characterized in that: The method comprises: reordering each sentence in the first text based on the encoding of each character of the sentence to obtain a second text, wherein the second text includes a plurality of sentences, and similar sentences in the plurality of sentences are arranged adjacently; Selecting a first preset number of first sentences in the second text according to the position order of the plurality of sentences to construct a text list; selecting a second preset number of first sentences in the text list as second sentences; selecting, based on a similarity between a third sentence in the second text and the second sentence, the third sentence having a similarity with the second sentence less than a preset threshold, and adding the third sentence to the text list to obtain an updated text list, wherein the third sentence is different from the first sentence; Constructing a first prompt text based on a preset field and a text example including vocabulary corresponding to the preset field; Based on the first prompt text, using a preset language model, determining the vocabulary corresponding to the preset field from the sentences in the updated text list; Replacing the vocabulary based on the field identifier corresponding to the preset field to obtain a text after the vocabulary is replaced; A first model is trained based on the text after the vocabulary replacement.

2. The training method of the text processing model according to claim 1, characterized in that: The selecting a second preset number of first sentences in the text list as second sentences includes: The second preset number of first sentences arranged at the end of the text list are selected in the text list as the second sentences.

3. The training method of the text processing model according to claim 1, characterized in that: The selecting, based on the similarity between the third sentence in the second text and the second sentence, the third sentence having a similarity with the second sentence less than a preset threshold and adding the third sentence to the text list comprises: Calculating character similarity between the third sentence and the second sentence; Calculating semantic similarity between the third sentence and the second sentence; Determining similarity between the third sentence and the second sentence based on the character similarity and the semantic similarity; The third sentence having a similarity less than the preset threshold is determined as a third sentence that is dissimilar to the second sentence, and the third sentence that is dissimilar to the second sentence is added to the text list.

4. The training method of the text processing model according to claim 1, characterized in that: After selecting, based on the similarity between the third sentence in the second text and the second sentence, the third sentence having a similarity with the second sentence less than a preset threshold and adding the third sentence to the text list, the method further includes: Delete the third statement in the second text.

5. The training method of the text processing model according to claim 1, characterized in that: The method further comprises: Clustering the sentences in the updated text list to obtain sentences corresponding to multiple categories; sorting the sentences corresponding to each category based on the distance between the sentences corresponding to each category and the centroid corresponding to each category to obtain a third text; Constructing a second prompt text based on the sentences in the third text for which label recognition has been completed; Based on the second prompt text, using a preset language model, identifying text labels corresponding to the sentences of each category; A second model is trained based on the sentences of each category and the corresponding text labels.

6. The text processing model training method according to claim 5, characterized in that: The step of constructing a second prompt text based on the sentence in the third text for which label recognition has been completed includes: Determining, based on a preset time window, a plurality of sentences from the sentences for which tag recognition has been completed; The second prompt text is constructed based on the multiple sentences and corresponding text tags.

7. The method for training a text processing model according to any one of claims 1 to 6, characterized in that: The method comprises: Obtaining a first reply text to the fourth text using a first model based on conversation data within a first preset time period and a fourth text within a second preset time period, wherein the second preset time period is after the first preset time period; Based on the first reply text, using the second model, determining a label corresponding to the first reply text; Determining the type of the fourth text based on the label corresponding to the first reply text; A second reply text to the fourth text is determined based on the type of the fourth text.

8. The text processing model training method according to claim 7, characterized in that: The determining, based on the type of the fourth text, a second reply text for the fourth text includes: If the type of the fourth text is a preset type, determining a text library matching the fourth text based on the label corresponding to the first reply text; Calculating a first similarity between a preset text in the text library and the fourth text; Determining a preset text having a first similarity greater than a first threshold as a target text; A reply text corresponding to the target text is obtained from the text library as the second reply text.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the training method for the text processing model according to any one of claims 1 to 8 is implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, which, when executed by a processor, implements the method for training a text processing model according to any one of claims 1 to 8.

Images