Text processing model training method, electronic device, and program product

Abstract

This application relates to artificial intelligence and provides a training method, electronic device, and program product for a text processing model. The training method includes: reordering each sentence based on the encoding of each character in each sentence of a first text, wherein the second text includes multiple sentences; constructing a text list by selecting a first preset number of first sentences from the second text according to the positional order of the multiple sentences; selecting a second preset number of first sentences from the text list as second sentences; adding third sentences with a similarity less than a preset threshold to the second sentences in the second text to the text list to obtain an updated text list; training a first model by performing word replacement on the sentences in the updated text list and based on the word-replaced text. This method can improve the generalization ability of the model.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to a training method for a text processing model, an electronic device, and a program product. Background Technology

[0002] Currently, the internet is filled with massive amounts of text, including a large amount of duplicate text. The presence of a large amount of duplicate text on the internet not only reduces the overall quality of the text but also leads to a waste of storage resources.

[0003] To improve text quality and save storage resources, it is necessary to deduplicate text content on the Internet. Related technologies typically involve pairwise comparisons of multiple texts; however, this method is inefficient and the resulting text is detrimental to model training. Summary of the Invention

[0004] This application provides a training method, electronic device, and program product for a text processing model to solve the technical problem of low deduplication efficiency and the resulting text being unfavorable to the model training process.

[0005] The first aspect of this application provides a method for training a text processing model. The method includes: reordering each statement based on the encoding of each character in a first text to obtain a second text, wherein the second text includes multiple statements, and similar statements are arranged adjacently in the multiple statements; selecting a first preset number of first statements in the second text according to the positional order of the multiple statements to construct a text list; selecting a second preset number of first statements in the text list as second statements; adding third statements with a similarity of less than a preset threshold to the second statements in the second text to the text list to obtain an updated text list, wherein the third statements are different from the first statements; constructing a first prompt text based on a preset field and text examples including words corresponding to the preset field; determining the words corresponding to the preset field from the statements in the updated text list based on the first prompt text and using a preset language model; replacing the words based on the field identifier corresponding to the preset field to obtain text with the replaced words; and training a first model based on the text with the replaced words.

[0006] A second aspect of this application provides a training apparatus for a text processing model. The apparatus includes: a sorting unit, configured to reorder each statement based on the encoding of each character in each statement in a first text to obtain a second text, wherein the second text includes multiple statements, and similar statements are arranged adjacently in the multiple statements; a construction unit, configured to select a first preset number of first statements in the second text according to the positional order of the multiple statements to construct a text list; a determination unit, configured to select a second preset number of first statements in the text list as second statements; a selection unit, configured to select third statements in the second text whose similarity to the second statements is less than a preset threshold and add them to the text list to obtain an updated text list, wherein the third statements are different from the first statements; constructing a first prompt text based on a preset field and text examples including words corresponding to the preset field; determining words corresponding to the preset field from statements in the updated text list based on the first prompt text using a preset language model; replacing the words based on the field identifier corresponding to the preset field to obtain text with the replaced words; and training a first model based on the text with the replaced words.

[0007] A third aspect of this application provides an electronic device, the electronic device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method provided in the first aspect above.

[0008] A fourth aspect of this application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the method provided in the first aspect above.

[0009] A fifth aspect of this application provides a computer program product, the computer program product including a computer program, which, when executed by a processor, implements the steps in the method provided in the first aspect above.

[0010] In the training method of the text processing model in this embodiment, by reordering the characters in the first text, similar sentences in the second text can be made adjacent. Then, based on the position of the first sentence in the text list, the second sentence is determined from the first sentence, initially screening out second sentences that are relatively similar to the third sentence. Based on the similarity between the third sentence and the second sentence, the third sentences that are dissimilar to the second sentence can be determined. These dissimilar third sentences are then added to the text list to obtain an updated text list. By performing word substitution on the sentences in the updated text list, and based on the word-substituted text, the first model is trained. This ensures that the word-substituted text can cover text from most domains, avoiding the first model's dependence on text from specific domains, thereby improving the model's generalization ability. Attached Figure Description

[0011] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a schematic diagram illustrating an application scenario of a text processing model training method provided in an embodiment of this application;

[0013] Figure 2 This is a flowchart of a text deduplication method provided in an embodiment of this application;

[0014] Figure 3 This is a flowchart illustrating a training method for a text processing model provided in an embodiment of this application;

[0015] Figure 4 This is a schematic diagram of the framework for determining the word-replaced sentence provided in the embodiments of this application;

[0016] Figure 5 This is a flowchart of another text processing model training method provided in the embodiments of this application;

[0017] Figure 6 This is a flowchart illustrating a method for applying a text processing model, as provided in an embodiment of this application.

[0018] Figure 7 This is a schematic diagram illustrating the framework for determining the second response text provided in the embodiments of this application;

[0019] Figure 8 This is a functional block diagram of a training device for a text processing model provided in an embodiment of this application;

[0020] Figure 9 This is a schematic diagram of the structure of an electronic device that implements the training method of a text processing model according to an embodiment of this application. Detailed Implementation

[0021] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0023] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used in this application's specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. It should be understood that, unless otherwise stated, " / " in this application means "or". For example, A / B can mean A or B. "And / or" in this application is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. "At least one" refers to one or more. "More than one" refers to two or more. For example, at least one of a, b, or c can represent: a, b, c, a and b, a and c, b and c, and a, b, and c (seven cases).

[0024] Explanation of some terms:

[0025] Large Language Model (LLM): A large language model is a deep learning model trained on a large amount of text data. It can generate natural language text or understand the meaning of language text. Large language models can handle various natural language tasks, such as text classification, question answering, and dialogue.

[0026] Currently, the internet is filled with massive amounts of text, including a large amount of duplicate text. The presence of a large amount of duplicate text on the internet not only reduces the overall quality of the text but also leads to a waste of storage resources.

[0027] To improve text quality and save storage resources, it is necessary to deduplicate text content on the Internet. Related technologies typically involve pairwise comparisons of multiple texts; however, this method is inefficient and the resulting text is detrimental to model training.

[0028] To address the aforementioned issues and improve deduplication efficiency and enhance the generalization ability of model training, this application provides a method for training a text processing model.

[0029] Please see Figure 1 , Figure 1 This is a schematic diagram illustrating an application scenario for a text processing model training method provided in an embodiment of this application. The scenario may include various electronic devices 100 and servers 200.

[0030] Electronic device 100 may be a mobile phone, tablet computer, desktop computer, laptop computer, handheld computer, notebook computer, ultra-mobile personal computer (UMPC), netbook, as well as cellular phone, personal digital assistant (PDA), augmented reality (AR) device, virtual reality (VR) device, artificial intelligence (AI) device, wearable device, in-vehicle device, smart home device and / or smart city device. This application embodiment does not impose any special restrictions on the specific type of electronic device 100.

[0031] Server 200 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms, but it is not limited to these.

[0032] It should be noted that the methods in this embodiment can be executed by either electronic device 100 or server 200 alone, or by both server 200 and electronic device 100. When executed by either electronic device 100 or server 200 alone, the entire text processing process (text deduplication, model training, and application) can be implemented by either electronic device 100 or server 200 independently. For example, a first model can be trained on electronic device 100 to obtain a trained first model. Correspondingly, after training, electronic device 100 can use the trained first model to obtain the first response text to the user's question. Similarly, a second model can be trained on electronic device 100 to obtain a trained second model. Correspondingly, after training, electronic device 100 can use the trained second model to determine the tag corresponding to the first response text. The above process can also be executed by server 200 alone. When executed jointly by server 200 and electronic device 100, server 200 can train the first model and / or the second model, and then deploy the trained first model and / or the trained second model to electronic device 100. Part of the text processing can be implemented by electronic device 100, and part can be implemented by server 200; the two cooperate to achieve the entire text processing process. In practical applications, specific configurations can be made according to the circumstances, and no specific limitations are made here.

[0033] It should be noted that when the entire text processing model training process provided in this application embodiment is executed solely by server 200 or electronic device 100, the above application scenario may also include only either server 200 or electronic device 100, or server 200 and electronic device 100 may be considered as the same device. In practical applications, when the entire text processing process provided in this application embodiment is jointly executed by server 200 and electronic device 100, server 200 and electronic device 100 may also be the same device, that is, server 200 and electronic device 100 may be different functional modules of the same device, or virtual devices virtualized by the same physical device.

[0034] In one possible implementation, the user can provide dialogue data within a first preset time period and a fourth text within a second preset time period through the electronic device 100. The server 200 can then use the training method of the present application embodiment to determine the second response text of the fourth text and return it to the electronic device 100 for presentation.

[0035] In this embodiment, the electronic device 100 and the server 200 can communicate directly or indirectly through one or more networks. This network can be a wired network or a wireless network; for example, the wireless network can be a mobile cellular network or a Wireless-Fidelity (Wi-Fi) network, and of course, it can be other possible networks. This embodiment does not limit this. It should be noted that... Figure 1 The examples shown are merely illustrative; in reality, the number of terminal devices and servers is unlimited and is not specifically limited in the embodiments of this application.

[0036] The following describes the training method of the text processing model provided in the embodiments of this application, in conjunction with the application scenarios described above and with reference to the accompanying drawings. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principle of the embodiments of this application, and the embodiments of this application are not limited in any way in this respect.

[0037] like Figure 2 The diagram shown is a flowchart of a text deduplication method provided in an embodiment of this application. This text deduplication method is applied in electronic devices, for example... Figure 1 Electronic device 100. Depending on different needs, the order of steps in this flowchart can be changed, and some steps can be omitted.

[0038] S201, based on the encoding of each character in each statement in the first text, reorder each statement to obtain the second text.

[0039] In at least one embodiment of this application, training data can be acquired to train the first model. For example, dialogue data can be acquired from a question-and-answer scenario as training data. The training data may include, but is not limited to, dialogue data between a customer and an agent. This application does not limit the format of the dialogue data; for example, the dialogue data may be presented in the form of text, video, or voice.

[0040] In at least one embodiment of this application, the first text can be the dialogue of any user in a question-and-answer scenario. In one example, in order for the first model to learn the agent's dialogue, the electronic device can extract the agent's dialogue from the dialogue data between the customer and the agent, and use the extracted dialogue as the first text.

[0041] In at least one embodiment of this application, in order for similar statements in a first text to be adjacent to each other, an electronic device can reorder each statement in the first text based on the encoding of each character of each statement in the first text to obtain a second text in which similar statements are arranged adjacently.

[0042] In this embodiment, a mapping table can be set up, which can include characters and their corresponding codepoint values. The electronic device can determine the encoding of each character in each statement of the first text based on the mapping table. This embodiment uses the mapping table to quickly determine the encoding corresponding to each character.

[0043] In this embodiment, the electronic device can use the `sorted()` function to sort the codes corresponding to each statement in the first text, obtaining a string. Based on the order of the codes in the string, each statement in the first text is reordered to obtain the second text. The second text includes multiple statements, with similar statements arranged adjacently. This embodiment, by sorting the codes corresponding to each statement in the first text, allows sentences with similar meanings to be arranged adjacently, thereby enabling similar statements in the second text to be arranged adjacently.

[0044] S202, in the second text, select a first preset number of first statements according to the positional order of multiple statements to construct a text list.

[0045] In at least one embodiment of this application, in order to deduplicatize the second text, a text list (also referred to as a "deduplicated set") can be constructed.

[0046] In this embodiment of the application, the electronic device may use at least one statement in the second text as the first statement.

[0047] In one example, the electronic device may use at least one statement from the second text as the first statement. For example, if the second text is: statement A, statement B, statement C, statement D, then the first statement may include: statement A, statement B.

[0048] In another example, the electronic device may use the statement that appears first in the second text as the first statement. For example, if the second text is: statement A, statement B, statement C, statement D, then the first statement may include: statement A.

[0049] In this embodiment, the electronic device can select a first preset number of first statements in the second text to construct a text list based on the positional order of the first statements in the second text. The positional order of the first statements in the text list is the same as the positional order of the first statements in the second text. For example, if the second text consists of statements A, B, C, and D, the first statements may include statements A and B. Then, the positional order of the first statements in the second text is statement A-statement B. After construction, a text list is obtained, wherein the order of the first statements in the text list is statement A-statement B.

[0050] This application embodiment constructs a text list based on the positional order of the first statement in the second text, which can prevent the first statement from being disordered in the text list, thereby ensuring that first statements with similar meanings in the text list can be arranged adjacently.

[0051] S203, Select the first statement of the second preset quantity from the text list as the second statement.

[0052] In at least one embodiment of this application, in order to reduce resource consumption, a second preset quantity can be set. For example, the second preset quantity can be set to 1, 2, etc. Setting a larger second preset quantity can improve the deduplication effect of the second text, while setting a smaller second preset quantity can improve the deduplication efficiency of the second text. This application does not limit this. The electronic device can select the first statement of the second preset quantity as the second statement.

[0053] In at least one embodiment of this application, the electronic device selecting a second preset number of first statements as second statements from a text list includes: selecting a second preset number of first statements arranged at the end of the text list as second statements. For example, if the second text consists of: statement A, statement B, statement C, and statement D; selecting three first statements from the second text to construct a text list, wherein the text list includes statement B, statement C, and statement D. Selecting two first statements from the text list as second statements means that the second statements include statement C and statement D.

[0054] In this embodiment of the application, since similar statements in the second text are arranged adjacently, selecting a second preset number of first statements as the second statements based on the position order of the first statements in the second text and the position order of the first statements in the text list can make the determined second statements more similar to the third statements in the second text, thus avoiding invalid detection of the remaining statements in the first statements.

[0055] S204, based on the similarity between the third statement in the second text and the second statement, select the third statement whose similarity to the second statement is less than a preset threshold and add it to the text list.

[0056] In at least one embodiment of this application, the third statement is different from the first statement. For example, if the second text is: statement A, statement B, statement C, statement D, and the first statement is: statement A, then the third statement may include: statement B, statement C, statement D.

[0057] In at least one embodiment of this application, in order to improve the accuracy of similarity, the similarity between the third statement and the second statement can be determined based on multiple dimensions.

[0058] In this embodiment of the application, the electronic device can calculate the character similarity between the third statement and the second statement, calculate the semantic similarity between the third statement and the second statement, and determine the similarity between the third statement and the second statement based on the character similarity and semantic similarity.

[0059] In this embodiment, the electronic device calculates the character similarity between the third statement and the second statement based on an edit distance formula. Specifically, the electronic device determines the minimum number of editing operations required to convert the third statement into the second statement, and determines the character similarity based on the minimum number of editing operations. The formula for calculating character similarity can be expressed as: ,in, It can represent character similarity. It can represent the minimum number of editing operations. It can be set according to actual needs, for example, It can be set to any positive real number.

[0060] In this embodiment, the electronic device uses a preset language model to determine the semantic similarity between the third statement and the second statement. The preset language model may include a large language model, a deep language model, etc. Specifically, the electronic device can construct preset prompt text, which may include, but is not limited to: information instructing the preset language model to act as identity information for evaluating the semantics of the text; task information instructing the preset language model to output the semantic similarity between the third statement and the second statement; and examples for the preset language model to reference. This embodiment utilizes a preset language model to quickly determine semantic similarity.

[0061] In another embodiment, the electronic device can determine a first encoded feature corresponding to the third statement and a second encoded feature corresponding to the second statement. The electronic device uses the similarity between the first encoded feature and the second encoded feature as semantic similarity.

[0062] In this embodiment, the electronic device can calculate a weighted sum of character similarity and semantic similarity to obtain the similarity between the third statement and the second statement. The weights corresponding to character similarity and semantic similarity can be set and adjusted according to actual needs.

[0063] The embodiments of this application improve the accuracy of similarity by calculating the similarity between the third statement and the second statement by comprehensively considering multiple dimensions.

[0064] In at least one embodiment of this application, in order to identify duplicate statements in the second text, a preset threshold can be set. The preset threshold can be set and adjusted according to actual needs. The electronic device can determine a third statement with a similarity greater than or equal to the preset threshold as a third statement similar to the second statement. To avoid duplicate statements in the text list, the electronic device will not add third statements similar to the second statement to the text list.

[0065] In this embodiment, the electronic device can determine a third statement with a similarity less than a preset threshold as a third statement that is dissimilar to the second statement. To avoid omitting information from the second text in the text list, the electronic device can add the third statement that is dissimilar to the second statement to the text list.

[0066] In this embodiment of the application, in order to avoid the third statement from participating in the detection repeatedly, the electronic device can delete the third statement that is participating in the detection.

[0067] For example, suppose the second statement includes statement A, and the third statement includes statements B, C, and D. The electronic device can calculate the similarity between statement B and statement A. If the similarity between statement B and statement A is greater than or equal to a preset threshold, it can be considered that statement B and statement A are similar statements. Therefore, to avoid the existence of similar statements in the text list, statement B will not be added to the text list, and to avoid statement B participating in the detection repeatedly, the electronic device can delete statement B. In this case, the third statement includes statements C and D.

[0068] Next, the electronic device can calculate the similarity between statement C and statement A. If the similarity between statement C and statement A is less than a preset threshold, it indicates that statement C and statement A are not similar. Therefore, statement C can be added to the text list. To avoid statement C participating in the detection repeatedly, the electronic device can delete statement C. At this point, the third statement includes statement D. This process is repeated until the third statement does not contain any statements.

[0069] In several embodiments of this application, by reordering the characters in the first text, similar sentences in the second text can be made adjacent. Then, by sorting the first sentences in the text list, the second sentences can be determined from the first sentences, and second sentences that are similar to the third sentences can be initially screened out. Based on the similarity between the third sentences and the second sentences, third sentences that are not similar to the second sentences can be determined, and then the third sentences that are not similar to the second sentences can be added to the text list. Since it is not necessary to compare the third sentences with all the first sentences, the update efficiency of the text list can be improved, thereby improving the deduplication efficiency of the first text.

[0070] like Figure 3 The diagram shown is a flowchart of a text processing model training method provided in an embodiment of this application. This text processing model training method is applied to electronic devices, for example... Figure 1 Electronic device 100. Depending on different needs, the order of steps in this flowchart can be changed, and some steps can be omitted.

[0071] S301, based on the encoding of each character in each statement in the first text, reorder each statement to obtain the second text.

[0072] S302, select a first preset number of first statements from the second text according to the positional order of multiple statements, and construct a text list.

[0073] S303, Select the first statement of the second preset quantity from the text list as the second statement.

[0074] S304, based on the similarity between the third statement in the second text and the second statement, select the third statement whose similarity to the second statement is less than a preset threshold and add it to the text list.

[0075] For details on steps S301-S304, please refer to the above text. Figure 2 The detailed descriptions of steps S201-S204 in the previous section will not be repeated here.

[0076] S305, construct the first prompt text based on the preset fields and text examples including the words corresponding to the preset fields.

[0077] In at least one embodiment of this application, the dialogue data in a question-and-answer scenario typically includes relevant variable information. For example, in a telemarketing scenario, the dialogue data typically includes the customer's name and gender. Telemarketing scenarios may also include variable information related to specific activities, such as withdrawable limits and interest rates.

[0078] In at least one embodiment of this application, since the variable information in the dialogue data corresponding to different question-and-answer scenarios is different, in order to reduce the dependence of the first model on the dialogue data of a specific scenario, the electronic device can perform word substitution on the variable information in the dialogue data.

[0079] In at least one embodiment of this application, during the word replacement process, the electronic device constructs a first prompt text based on a preset field and a text example including the words corresponding to the preset field.

[0080] In this application embodiment, the preset fields may include, but are not limited to, fields corresponding to variable information that change due to different question and answer scenarios, such as amount, name, activity information, and date.

[0081] In this embodiment of the application, the first prompt text may include: information indicating the identity of the preset language model as a text annotation expert, for example, you are now a text annotation expert, and your task is to annotate the relevant fields in the text given the required annotation fields and examples.

[0082] The initial prompt text may also include task information instructing the preset language model to output vocabulary corresponding to preset fields. For example, the required labeled fields may include surname, gender, last four digits of mobile phone number, and application date. The labeling rule is to identify relevant fields in the text and label the fields according to the following rules: Surname: ,gender: Mobile phone number ending in: Application date: .

[0083] The first prompt text may also include: text examples containing words corresponding to the preset fields (also known as "examples for the preset language model to reference"). Examples for the preset language model to reference may include, but are not limited to: examples that require word replacement and examples that do not require word replacement. Examples for the preset language model to reference may include, but are not limited to: text, thought process, and output annotation results.

[0084] For example, an example that requires word replacement could be:

[0085] Text: "Yes, sir, there's no mistake. You applied through the mini-program yesterday, and we see that your credit limit has been approved. You can use it directly, okay?"

[0086] Thought process: The statement "Hmm, no mistake, sir" mentions gender: "sir," and the statement "You applied yesterday, on the mini-program?" mentions application date: "yesterday." Therefore, the relevant fields in this text are: gender: "sir," and application date: "yesterday." Replace "sir" in the text with... Replace 'yesterday' with ' The annotation result is: "Hmm, no mistake." Are you here? "Well, I applied through the mini-program, and we can see that your credit limit has been approved. You can use it directly, okay?"

[0087] Output annotation process: "Hmm, no mistake." Are you here? "Well, I applied through the mini-program, and we can see that your credit limit has been approved. You can use it directly, okay?"

[0088] For example, an example that does not require word substitution can be:

[0089] Text: "Oh, so you're not satisfied with the interest rate?"

[0090] Thought process: The statement "Oh, so you're not satisfied with the interest rate?" does not mention any information related to surname, gender, last four digits of mobile phone number, or application date. Therefore, no annotation is needed, and the result is: No annotation required.

[0091] Output annotation process: No annotation required.

[0092] S306, based on the first prompt text, using a preset language model, determine the words corresponding to the preset fields from the statements in the updated text list.

[0093] In at least one embodiment of this application, the electronic device inputs a first prompt text into a preset language model to obtain a first feedback text output by the preset language model. When the first feedback text indicates that the preset language model has learned the text example in the first prompt text, the electronic device inputs statements from an updated text list into the preset language model to obtain vocabulary corresponding to a preset field. The vocabulary corresponding to the preset field can be variable information in the statements of the updated text list that corresponds to the preset field. For example, assuming the preset field is gender, the vocabulary corresponding to the preset field could be: Mr. / Ms.

[0094] For example, if the first feedback text output by the preset language model is "Okay, I've learned it, I will directly output the annotation results", it can indicate that the first feedback text indicates that the preset language model has learned the example in the first prompt text.

[0095] In other embodiments, if the first feedback text indicates that the preset language model has not learned the example in the first prompt text, the electronic device may add an example for the preset language model to refer to in the first prompt text.

[0096] S307: Based on the field identifier corresponding to the preset field, replace the words to obtain the text after word replacement.

[0097] In at least one embodiment of this application, the field identifier corresponding to the preset field can be set and adjusted according to actual needs. For example, the field identifier corresponding to the preset field "gender" can be set as follows: The field identifier corresponding to the default field "Date" can be set as follows: .

[0098] In at least one embodiment of this application, the electronic device inputs statements from the updated text list into a preset language model to obtain words corresponding to preset fields. The electronic device then calls the preset language model to replace the words corresponding to the preset fields with field identifiers corresponding to the preset fields, and uses the second feedback text output by the preset language model as the text after word replacement.

[0099] Combination Figure 4Explain the process of determining the sentence after word substitution, such as... Figure 4 As shown, the deduplicated data (also known as "statements in the updated text list") and scenario variable information (also known as "preset fields") are input into the large model (also known as "preset language model"). The large model can replace the relevant variables in the deduplicated data (also known as "vocabularies corresponding to the preset fields") to obtain the template speech (also known as "text after vocabulary replacement").

[0100] This application embodiment uses a first prompt text to assist a preset language model in learning to annotate words in the updated text list that correspond to preset fields, thereby enabling the rapid acquisition of text after word replacement.

[0101] For example, suppose the dialogue data for one scenario is: "Well, because our current product F has updated its maximum credit limit to 200,000, your current available credit limit is 2,000, right?". Suppose the dialogue data for another scenario is: "Well, because our current product Q has updated its maximum credit limit to 200,000, your current available credit limit is 5,000, right?". To reduce the first model's reliance on dialogue data specific to a particular scenario, electronic devices can perform word substitution on the variable information in the dialogue data. For example, the dialogue data after word substitution might be: "Well, because our current..." Well, the maximum credit limit has been updated to 200,000. So, what is your current available credit limit? Yuan, right?

[0102] S308, based on the text after word substitution, trains the first model.

[0103] In at least one embodiment of this application, the electronic device trains a first model based on the word-replaced text. In this embodiment, since the word-replaced text covers dialogue data from most domains, training the first model using the word-replaced text can reduce the first model's dependence on training data, thereby improving the generalization ability of the first model.

[0104] In several embodiments of this application, by reordering the characters in the first text, similar sentences in the second text can be made adjacent. Then, by sorting the first sentences in the text list, the second sentences can be determined from the first sentences, initially screening out second sentences that are relatively similar to the third sentence. Based on the similarity between the third and second sentences, third sentences that are dissimilar to the second sentences can be determined, and these dissimilar third sentences are added to the text list. Since it is not necessary to compare the third sentences with all first sentences, the efficiency of updating the text list can be improved, thereby improving the training efficiency of the first model. Furthermore, by performing word substitution on the sentences in the updated text list and training the first model based on the word-substituted text, it can be ensured that the word-substituted text can cover text from most domains, avoiding the first model's dependence on text from specific domains, thereby improving the model's generalization ability.

[0105] like Figure 5 The diagram shown is a flowchart of another text processing model training method provided in this application embodiment. This text processing model training method is applied to electronic devices, for example... Figure 1 Electronic device 100. Depending on different needs, the order of steps in this flowchart can be changed, and some steps can be omitted.

[0106] S501, based on the encoding of each character in each statement in the first text, reorder each statement to obtain the second text.

[0107] S502, select a first preset number of first statements from the second text according to the positional order of multiple statements, and construct a text list.

[0108] S503, Select the first statement of the second preset quantity from the text list as the second statement.

[0109] S504, based on the similarity between the third statement in the second text and the second statement, select the third statement whose similarity to the second statement is less than a preset threshold and add it to the text list.

[0110] For details on steps S501-S504, please refer to the above text. Figure 2 The detailed descriptions of steps S201-S204 in the previous section will not be repeated here.

[0111] S505, cluster the statements in the updated text list to obtain statements corresponding to multiple categories.

[0112] In at least one embodiment of this application, the electronic device encodes statements in an updated text list to obtain encoded features corresponding to each statement. At least two encoded features are randomly selected as centroids from multiple encoded features. The distance between the encoded feature corresponding to each statement and the centroid is calculated, and each statement is assigned to the category corresponding to the nearest centroid. Based on the encoded features of all statements in the category corresponding to the centroid, the updated centroid is determined. If the change between the distance between the encoded feature corresponding to each statement and the updated centroid and the distance between the encoded feature corresponding to each statement and the centroid is greater than or equal to a set threshold, all statements in the category corresponding to the centroid are reassigned until the change between the distance between the encoded feature corresponding to each statement and the updated centroid and the distance between the encoded feature corresponding to each statement and the centroid is less than the set threshold. If the change between the distance between the encoded feature corresponding to each statement and the updated centroid and the distance between the encoded feature corresponding to each statement and the centroid is less than the set threshold, statements corresponding to multiple categories are determined based on the statements in the category corresponding to the updated centroid.

[0113] In this embodiment, during the determination of the updated centroid, the electronic device can perform bitwise calculations on the encoded features of all statements in the category corresponding to the centroid to obtain the updated centroid. In one example, the electronic device can calculate the average value of the corresponding feature elements in all encoded features to obtain the updated centroid. For example, a category includes statements A and B, where the encoded features of statement A are [a1,a2,a3,…,an] and the encoded features of statement B are [b1,b2,b3,…,bn]. The updated centroid can be calculated as [(a1+b1) / 2,(a2+b2) / 2,(a3+b3) / 2,…,(an+bn) / 2]. In another example, the electronic device can calculate the weighted sum of the corresponding feature elements in all encoded features to obtain the updated centroid.

[0114] In the embodiments of this application, the electronic device can use the similarity calculation formula to calculate the distance between the encoded features corresponding to each statement and the updated centroid, and to calculate the distance between the encoded features corresponding to each statement and the centroid. This application does not impose specific limitations on the similarity calculation formula.

[0115] S506, based on the distance between the statement corresponding to each category and the centroid corresponding to each category, sort the statements corresponding to each category to obtain the third text.

[0116] In at least one embodiment of this application, in order to make adjacent sentences more similar, the electronic device can sort the sentences in ascending order based on the distance between the sentences corresponding to each category and the centroid corresponding to each category to obtain a third text.

[0117] For example, suppose a category includes statements A and B. The encoding features of statement A are [a1,a2,a3,…,an], and the encoding features of statement B are [b1,b2,b3,…,bn], with centroids [(a1+b1) / 2,(a2+b2) / 2,(a3+b3) / 2,…,(an+bn) / 2]. Calculations show that the encoding features [a1,a2,a3,…,an] of statement A correspond to the centroids [(a1+b1) / 2,(a2+b2) / 2,(a3+b3) / 2,…,(an+bn) / 2]. The distance between the centroid [(a1+b1) / 2,(a2+b2) / 2,(a3+b3) / 2,…,(an+bn) / 2] and the centroid [(a1+b1) / 2,(a2+b2) / 2,(a3+b3) / 2,…,(an+bn) / 2] is 3, and the distance between the encoded feature [b1,b2,b3,…,bn] of statement B and the centroid [(a1+b1) / 2,(a2+b2) / 2,(a3+b3) / 2,…,(an+bn) / 2] is 2. Therefore, the third text can be obtained as {statement B, statement A}.

[0118] This embodiment sorts the sentences corresponding to each category, making adjacent sentences in the third text more similar and reducing the divergence between sentences, which helps with model training.

[0119] S507, construct a second prompt text based on the statements in the third text that have already completed tag recognition.

[0120] In at least one embodiment of this application, the statements in the third text that have completed tag recognition can be used as a reference for a preset language model, so that the preset language model can learn how to perform tag recognition on the statements.

[0121] In at least one embodiment of this application, the electronic device constructs a second prompt text based on statements in a third text that have completed tag recognition, including: determining multiple statements from the statements that have completed tag recognition based on a preset time window; and constructing the second prompt text based on the multiple statements and their corresponding text tags.

[0122] In this embodiment of the application, in order to improve the tag recognition effect of the preset language model, the electronic device can set a preset time window. The electronic device can determine the recognition time of the statements that have completed tag recognition, and use the statements whose recognition time is within the preset time window as multiple statements for reference by the preset language model.

[0123] In this embodiment of the application, the electronic device can use the determined multiple statements and their corresponding text tags as examples in the second prompt text for reference by the preset language model.

[0124] For example, the example provided in the second prompt text for the preset language model to reference could be:

[0125] Text: "Hello Mr. Ren, this is a staff member from XX. I'm calling to confirm that you are the one using YY, right?"

[0126] Thought process: First, we break down the text according to its semantics, which can be divided into the following segments: "1. Hello Mr. Ren, this is XX staff member, 2. I'm calling to confirm that you are the one using YY, right?"

[0127] Therefore, there are ultimately two labels! Let's summarize each sentence in turn: "Hello Mr. Ren, this is XX staff member," this sentence is a self-introduction, so the first label can be summarized as: Self-introduction. "The call is to confirm that you are the one using YY, right?" This sentence is confirming whether the customer is indeed using the product, so the second label is: Confirming Account User.

[0128] Output script tags: "Hello Mr. Ren, this is XX staff member," belongs to "self-introduction." "The call is to confirm that you are the one using YY, right?" belongs to "confirming account user."

[0129] The second prompt text may also include: information indicating the identity of the preset language model as an operations expert, for example, you are a senior telephone sales operations expert, and your current task is to analyze the given marketing text with reference to the background information and give the corresponding wording tags.

[0130] The second prompt text may also include task information instructing the preset language model to output the vocabulary corresponding to the dialogue tags. For example, when sorting out dialogue tags, the provided tags should be as comprehensive as possible, and the output dialogue tags should represent the overall meaning of the text. If the text is short, a single dialogue tag is sufficient. When sorting out dialogue tags, it is necessary to appropriately break down the dialogue according to the semantic relationships and logical connections between sentences, and then summarize the dialogue tags for each of the broken sentences in turn, referring appropriately to historical dialogues. The tag content should be summarized using concise and clear language, but key information should not be omitted. Specific variables such as time, date, and amount should be abstracted, and specific values ​​should not be directly output. The sorted dialogue tags should include specific dialogue details, such as the main content of the activity and what specific questions the customer needs to be addressed.

[0131] This application embodiment uses a preset time window to determine multiple statements from statements that have completed tag recognition, which can avoid using all statements that have completed tag recognition as the second prompt text, thereby improving the recognition efficiency of the preset language model for the second prompt text. In addition, constructing the second prompt text based on statements that have completed tag recognition and their corresponding text tags can improve the recognition ability of the preset recognition model for text tags.

[0132] S508, based on the second prompt text, uses a preset language model to identify the text tags corresponding to each category of statements.

[0133] In at least one embodiment of this application, the electronic device inputs a second prompt text into a preset language model to obtain a third feedback text output by the preset language model. When the third feedback text indicates that the preset language model has learned the examples in the second prompt text, statements of each category are input into the preset language model to obtain text tags corresponding to the statements of each category.

[0134] For example, if the third feedback text output by the preset language model is "Okay, I've learned it. I will directly output the word tag. Let's get started," it can indicate that the third feedback text indicates that the preset language model has learned the example in the second prompt text.

[0135] In other embodiments, if the third feedback text indicates that the preset language model has not learned the example in the first prompt text, the electronic device may increase the window length of the preset time window and add an example for the preset language model to refer to in the second prompt text.

[0136] S509, based on the statements and corresponding text labels of each category, train the second model.

[0137] In at least one embodiment of this application, an electronic device can construct training samples based on multiple categories of statements and their corresponding text tags. The training samples can include positive samples and negative samples. Positive samples can include multiple categories of statements and their corresponding text tags, while negative samples can include multiple categories of statements and other tags, where the other tags can be text tags in the training samples that are different from the positive samples. For example, for statement A, the text tag corresponding to statement A is tag Z. Assuming that the text tags corresponding to multiple categories of statements include: tag X, tag Y, and tag Z, the electronic device can construct positive samples based on statement A and tag Z, construct negative samples based on statement A and tag X, and can also construct another negative sample based on statement A and tag Y.

[0138] In at least one embodiment of this application, the electronic device obtains a prediction result based on sentences in training samples using a second model. The electronic device calculates the loss value of the second model based on the similarity between the text labels in positive samples and the prediction result, and the similarity between the text labels in negative samples and the prediction result.

[0139] In this embodiment, the formula for calculating the loss value of the second model can be expressed as:

[0140] .in, This can represent the loss value of the second model. The encoded features that can represent the prediction results It can represent the encoding features of text labels in positive samples. It can represent the similarity between the text labels in the positive samples and the predicted results. It can represent the encoded features of text labels in negative samples. It can represent the similarity between the text labels in the negative samples and the predicted results. It can be set and adjusted according to actual needs, for example, It can be set to 0.05, and this application does not impose specific restrictions on it.

[0141] In at least one embodiment of this application, the electronic device adjusts the second model based on the loss value of the second model until the loss value reaches a first preset condition. The first preset condition may include, but is not limited to: the loss value no longer decreases, or the loss value is less than or equal to a preset loss threshold. The preset loss threshold can be set and adjusted as needed.

[0142] In another embodiment, the electronic device may further adjust the second model based on the loss value until the number of adjustments or the learning rate reaches a second preset condition. The second preset condition may include, but is not limited to, a preset number of adjustments or a preset value for the learning rate.

[0143] In several embodiments of this application, by clustering the statements in the updated second list and sorting the statements corresponding to each category to obtain the third text, adjacent statements in the third text can be made more similar, thereby improving the recognition effect of the preset language model on the statements in the third text. Using the statements in the third text that have already undergone tag recognition, a second prompt text is constructed, which can assist the preset language model in learning the examples in the second prompt text, thereby improving the prediction accuracy of text tags. The second model is trained using the statements of each category and their corresponding text tags. Since there is no need for manual annotation of the text tags corresponding to the statements, the training efficiency of the second model can be improved. Furthermore, by training the second model, without changing the first model, business users can quickly iterate their strategy scripts based on tags.

[0144] like Figure 6 The diagram shown is a flowchart of a method for applying a text processing model according to an embodiment of this application. This text processing model training method is applied to electronic devices, for example... Figure 1 Electronic device 100. Depending on different needs, the order of steps in this flowchart can be changed, and some steps can be omitted.

[0145] S601, based on the dialogue data within the first preset time period and the fourth text within the second preset time period, the first response text of the fourth text is obtained using the first model.

[0146] In at least one embodiment of this application, the second preset time period is after the first preset time period. The electronic device inputs the dialogue data and the fourth text into the first model to obtain the first response text of the fourth text.

[0147] S602, Based on the first response text, use the second model to determine the tag corresponding to the first response text.

[0148] In at least one embodiment of this application, the electronic device inputs the first response text into the second model to obtain the tag corresponding to the first response text.

[0149] S603, determine the type of the fourth text based on the tag corresponding to the first response text.

[0150] In at least one embodiment of this application, a preset tag can be set to determine the type of the fourth text, for example, the preset tag can be set to "interest answer". The electronic device compares the tag corresponding to the first reply text with the preset tag. If the tag corresponding to the first reply text is the preset tag, the type of the fourth text can be determined to be the preset type. If the tag corresponding to the first reply text is not the preset tag, the type of the fourth text can be determined to be the preset type.

[0151] S604, based on the type of the fourth text, determine the second response text of the fourth text.

[0152] In at least one embodiment of this application, if the type of the fourth text is a preset type, the electronic device determines the second response text of the fourth text based on the type of the fourth text, including: determining a text library that matches the fourth text based on the tag corresponding to the first response text; calculating the first similarity between the preset text in the text library and the fourth text; determining the preset text with the first similarity greater than a first threshold as the target text; and obtaining the response text corresponding to the target text from the text library as the second response text.

[0153] In this embodiment, the tags corresponding to the first response text have a corresponding relationship with the text library. The electronic device can determine the text library that matches the fourth text based on the tags corresponding to the first response text. To avoid the second response text involving sensitive information such as personal privacy, the multiple question-and-answer pairs included in the text library do not include variable information corresponding to preset fields. The words corresponding to the preset fields in the multiple question-and-answer pairs are presented with preset identifiers. For example, the answer in the text library can be set as: "Eh because we are now..." Well, the maximum credit limit has been updated to 200,000. So, what is your current available credit limit? "Yuan, right?" Each question-and-answer pair includes a pre-set text and a corresponding response text.

[0154] In this embodiment, the electronic device can use a similarity calculation formula to calculate the first similarity between a preset text in the text library and a fourth text. This embodiment does not impose specific limitations on the similarity calculation formula.

[0155] In this embodiment, to filter out target text that matches the fourth text from preset text, a first threshold can be set. The value of the first threshold can be set and adjusted according to actual needs. In one example, the electronic device determines preset text with a first similarity greater than the first threshold as target text. In another example, the electronic device can determine preset text with the highest first similarity as target text.

[0156] In this embodiment of the application, when the type of the fourth text is a preset type, the text library matching the fourth text can be accurately determined based on the tags corresponding to the first response text. By using the first similarity between the preset text in the text library and the fourth text, target texts similar to the fourth text can be filtered out. Then, by using the response texts corresponding to the target texts in the text library, the second response text can be accurately determined.

[0157] In at least one embodiment of this application, if the type of the fourth text is not a preset type, the electronic device determines the second response text of the fourth text based on the type of the fourth text, and further includes: obtaining candidate response texts corresponding to the tags of the first response text from a preset library; calculating the second similarity between the candidate response texts and the fourth text; and determining the candidate response texts with the second similarity greater than the second threshold as the second response texts.

[0158] In this embodiment, to avoid the second response text involving sensitive information such as personal privacy, the multiple response texts included in the preset library do not include variable information corresponding to preset fields, and the words corresponding to the preset fields in the multiple response texts are presented with preset identifiers. The preset library includes multiple response texts and corresponding tags. The electronic device can obtain the response text corresponding to the tag of the first response text from the preset library as a candidate response text based on the tag corresponding to the first response text.

[0159] In this embodiment, the electronic device can use a similarity calculation formula to calculate the second similarity between the candidate response text and the fourth text. This embodiment does not impose specific limitations on the similarity calculation formula.

[0160] In this embodiment, to filter out the second response text that matches the fourth text from the candidate response texts, a second threshold can be set. The value of the second threshold can be set and adjusted according to actual needs. In one example, the electronic device determines the candidate response text with a second similarity greater than the second threshold as the second response text. In another example, the electronic device can determine the candidate response text with the highest second similarity as the second response text.

[0161] In this embodiment of the application, when the type of the fourth text is not a preset type, candidate response texts of the fourth text can be obtained from a preset library based on the tags of the first response text. Then, by using the second similarity between the candidate response texts and the fourth text, the second response text of the fourth text can be accurately determined.

[0162] Combination Figure 7 The process of determining the second response text is as follows: The electronic device inputs historical dialogues (also referred to as "dialogue data within the first preset time period") and the user's current question (also referred to as "the fourth text within the second preset time period") into a large dialogue model (also referred to as "the first model"). The large dialogue model outputs the answer to the current question (also referred to as "the first response text"). The electronic device then inputs the answer to the current question into a retrieval model (also referred to as "the second model"). The retrieval model outputs the optimal label (also referred to as "the label corresponding to the first response text") and determines whether the optimal label is a feature label (also referred to as "the preset label"). If the optimal label is a special label, the electronic device uses the retrieval model to retrieve the special response script (also referred to as "the text library matching the fourth text") corresponding to the user's current question from the special label question-and-answer library (also referred to as "the second response text"). If the optimal label is not a special label, the electronic device retrieves the script corresponding to the optimal label from the script library (also referred to as "the preset library") as the optimal script (also referred to as "the second response text").

[0163] In at least one embodiment of this application, the electronic device determines the field information corresponding to the preset field from the second response text, replaces the field information with the corresponding variable information to obtain the target response text, and sends the target response text to the user terminal.

[0164] In the training method of this embodiment, the first model is used to determine the first response text of the fourth text, and the second model is used to determine the label corresponding to the first response text. Based on the label corresponding to the first response text, the type of the fourth text can be determined, and thus the second response text of the fourth text can be determined based on the type of the fourth text, which can improve the accuracy of the second response text.

[0165] like Figure 8 The diagram shown is a functional block diagram of a text processing model training device provided in an embodiment of this application. The text processing model training device 81 includes a sorting unit 810, a construction unit 811, a determination unit 812, a selection unit 813, a replacement unit 814, a training unit 815, a clustering unit 816, and a recognition unit 817. The module / unit referred to in this application refers to a module / unit that can be processed by a processor (e.g., ...). Figure 9The processor 901 shown acquires a series of computer-readable instruction segments that are capable of performing a fixed function, and these segments are stored in memory (e.g., a memory processor 901). Figure 9 In the memory 902 shown.

[0166] In one embodiment, the sorting unit 810 is used to reorder each statement based on the encoding of each character in each statement in the first text to obtain a second text; the construction unit 811 is used to select a first preset number of first statements in the second text according to the positional order of the plurality of statements to construct a text list; the determining unit 812 is used to select a second preset number of first statements in the text list as second statements; and the selection unit 813 is used to select third statements in the second text whose similarity to the second statements is less than a preset threshold and add them to the text list, wherein the third statements are different from the first statements.

[0167] In one embodiment, the determining unit 812 is specifically configured to: select the second preset number of first statements arranged at the end of the text list as the second statement.

[0168] In one embodiment, the selection unit 813 is specifically used for: calculating the character similarity between the third statement and the second statement; calculating the semantic similarity between the third statement and the second statement; determining the similarity between the third statement and the second statement based on the character similarity and semantic similarity; determining the third statement with a similarity less than a preset threshold as a third statement that is not similar to the second statement, and adding the third statement that is not similar to the second statement to the text list.

[0169] In one embodiment, the construction unit 811 is further configured to construct a first prompt text based on a preset field and text examples including words corresponding to the preset field; the determination unit 812 is further configured to determine the words corresponding to the preset field from the statements in the updated text list based on the first prompt text and using a preset language model; the replacement unit 814 is configured to replace the words based on the field identifier corresponding to the preset field to obtain the text after word replacement; and the training unit 815 is configured to train a first model based on the text after word replacement.

[0170] In one embodiment, clustering unit 816 is used to cluster the statements in the updated text list to obtain statements corresponding to multiple categories; sorting unit 810 is also used to sort the statements corresponding to each category based on the distance between the statements corresponding to each category and the centroids corresponding to each category to obtain third text; construction unit 811 is also used to construct second prompt text based on the statements in the third text that have completed tag recognition; recognition unit 817 is used to recognize the text tags corresponding to the statements of each category based on the second prompt text and using a preset language model; training unit 815 is also used to train a second model based on the statements of each category and their corresponding text tags.

[0171] In one embodiment, the construction unit 811 is specifically used to: determine multiple statements from statements that have completed tag recognition based on a preset time window; and construct a second prompt text based on the multiple statements and their corresponding text tags.

[0172] In one embodiment, the determining unit 812 is further configured to, based on the dialogue data within a first preset time period and the fourth text within a second preset time period, use a first model to obtain a first response text for the fourth text, wherein the second preset time period is after the first preset time period; the determining unit 812 is further configured to, based on the first response text, use a second model to determine the tag corresponding to the first response text; the determining unit 812 is further configured to, based on the tag corresponding to the first response text, determine the type of the fourth text; and the determining unit 812 is further configured to, based on the type of the fourth text, determine a second response text for the fourth text.

[0173] In one embodiment, the determining unit 812 is specifically used for: if the type of the fourth text is a preset type, determining a text library that matches the fourth text based on the tag corresponding to the first response text; calculating the first similarity between the preset text in the text library and the fourth text; determining the preset text with the first similarity greater than a first threshold as the target text; and obtaining the response text corresponding to the target text from the text library as the second response text.

[0174] In one embodiment, the determining unit 812 is specifically used for: if the type of the fourth text is not a preset type, obtaining candidate response texts corresponding to the tags of the first response text from a preset library; calculating the second similarity between the candidate response texts and the fourth text; and determining the candidate response texts with a second similarity greater than a second threshold as the second response texts.

[0175] In several embodiments of this application, by reordering the characters in the first text, similar sentences in the second text can be made adjacent. Then, by sorting the first sentences in the text list, the second sentences can be determined from the first sentences, and second sentences that are similar to the third sentences can be initially screened out. Based on the similarity between the third sentences and the second sentences, third sentences that are not similar to the second sentences can be determined, and then the third sentences that are not similar to the second sentences can be added to the text list. Since it is not necessary to compare the third sentences with all the first sentences, the update efficiency of the text list can be improved, thereby improving the deduplication efficiency of the first text.

[0176] Figure 9 This is a schematic diagram of the structure of an electronic device that implements the training method for a text processing model according to an embodiment of this application. Optionally, Figure 9 The electronic device 100 shown is used to perform Figures 2 to 7 The method shown.

[0177] Electronic device 100 includes at least one processor 901, memory 902, and at least one network interface 903.

[0178] Processor 901 is, for example, a general-purpose central processing unit (CPU), a network processor (NP), a graphics processing unit (GPU), a neural-network processing unit (NPU), a data processing unit (DPU), a microprocessor, or one or more integrated circuits for implementing the embodiments of this application. For example, processor 901 includes application-specific integrated circuits (ASICs), programmable logic devices (PLDs), or combinations thereof. A PLD is, for example, a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0179] Memory 902 may be, for example, read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions; random access memory (RAM) or other types of dynamic storage devices capable of storing information and instructions; electrically erasable programmable read-only memory (EEPROM); compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed discs, laser discs, optical discs, digital universal discs, Blu-ray discs, etc.); magnetic disk storage media or other magnetic storage devices; or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. Optionally, memory 902 exists independently and is connected to processor 901 via internal connection 904. Alternatively, memory 902 and processor 901 may be integrated together.

[0180] Network interface 903 uses any transceiver-like device for communicating with other devices or communication networks. Network interface 903 includes, for example, at least one of a wired network interface or a wireless network interface. The wired network interface is, for example, an Ethernet interface. The Ethernet interface is, for example, an optical interface, an electrical interface, or a combination thereof. The wireless network interface is, for example, a wireless local area network (WLAN) interface, a cellular network interface, or a combination thereof.

[0181] In some embodiments, processor 901 includes one or more CPUs, such as Figure 9 CPU0 and CPU1 are shown in the diagram.

[0182] In some embodiments, the electronic device 100 may optionally include a plurality of processors, such as Figure 9 The processors 901 and 905 are shown. Each of these processors is, for example, a single-core processor (CPU) or a multi-core processor (CPU). Here, "processor" may optionally refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0183] In some embodiments, the electronic device 100 further includes an internal connection 904. The processor 901, memory 902, and at least one network interface 903 are connected via the internal connection 904. The internal connection 904 includes pathways for transmitting information between the aforementioned components. Optionally, the internal connection 904 is a single board or a bus. Optionally, the internal connection 904 may be divided into an address bus, a data bus, a control bus, etc.

[0184] In some embodiments, the electronic device 100 further includes an input / output interface 906. The input / output interface 906 is connected to an internal connection 904.

[0185] Optionally, the processor 901 implements the method in the above embodiments by reading the program code 910 stored in the memory 902, or the processor 901 implements the method in the above embodiments by internally stored program code. When the processor 901 implements the method in the above embodiments by reading the program code 910 stored in the memory 902, the memory 902 stores program code that implements the method provided in the embodiments of this application.

[0186] For more details on how processor 901 implements the above functions, please refer to the descriptions in the previous method embodiments, which will not be repeated here.

[0187] This embodiment also provides a computer storage medium storing computer instructions. When the computer instructions are executed on an electronic device, the electronic device performs the aforementioned related method steps to implement the training method in the above embodiment.

[0188] This embodiment also provides a computer program product that, when run on an electronic device, causes the electronic device to perform the aforementioned related steps to implement the training method described in the above embodiment.

[0189] In addition, embodiments of this application also provide an apparatus, which may specifically be a chip, component, or module. The apparatus may include a connected processor and a memory; wherein the memory is used to store computer execution instructions, and when the apparatus is running, the processor may execute the computer execution instructions stored in the memory to cause the chip to execute the training methods in the above-described method embodiments.

[0190] In this embodiment, the electronic device, computer storage medium, computer program product or chip are all used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects of the corresponding method provided above, and will not be repeated here.

[0191] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0192] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0193] The unit described as a separate component may or may not be physically separate. The component shown as a unit can be one physical unit or multiple physical units, that is, it can be located in one place or distributed in multiple different places. Some or all of the units can be selected to achieve the purpose of the solution in this embodiment according to actual needs.

[0194] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0195] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially or in other words, the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0196] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A training method for a text processing model, characterized in that, The method includes: Based on the encoding of each character in each statement in the first text, each statement is reordered to obtain a second text, wherein the second text includes multiple statements, and similar statements are arranged adjacently among the multiple statements; In the second text, a text list is constructed by selecting a first preset number of first statements according to the positional order of the plurality of statements; Select a second preset number of first statements from the text list as the second statements; Based on the similarity between the third statement in the second text and the second statement, select the third statement whose similarity to the second statement is less than a preset threshold and add it to the text list to obtain an updated text list. The third statement is different from the first statement. Based on preset fields and text examples including words corresponding to the preset fields, a first prompt text is constructed; Based on the first prompt text, using a preset language model, the vocabulary corresponding to the preset field is determined from the statements in the updated text list; Based on the field identifier corresponding to the preset field, the word is replaced to obtain the text after word replacement; The first model is trained based on the text after the word replacement.

2. The training method for the text processing model according to claim 1, characterized in that, The step of selecting a second preset number of first statements as second statements from the text list includes: Select the first statement, which is the second preset number of statements arranged at the end of the text list, as the second statement.

3. The training method for the text processing model according to claim 1, characterized in that, The step of selecting third statements in the second text that have a similarity to the second statement less than a preset threshold and adding them to the text list based on the similarity between the third statement in the second text and the second statement includes: Calculate the character similarity between the third statement and the second statement; Calculate the semantic similarity between the third statement and the second statement; Based on the character similarity and the semantic similarity, the similarity between the third statement and the second statement is determined; The third statement with a similarity less than the preset threshold is identified as a third statement that is not similar to the second statement, and the third statement that is not similar to the second statement is added to the text list.

4. The training method for the text processing model according to claim 1, characterized in that, After selecting third statements in the second text whose similarity to the second statement is less than a preset threshold and adding them to the text list based on the similarity between the third statement in the second text and the second statement, the method further includes: Delete the third statement in the second text.

5. The training method for the text processing model according to claim 1, characterized in that, The method further includes: Cluster the statements in the updated text list to obtain statements corresponding to multiple categories; Based on the distance between the statement corresponding to each category and the centroid corresponding to each category, the statements corresponding to each category are sorted to obtain the third text; Based on the statements in the third text that have already completed tag recognition, construct a second prompt text; Based on the second prompt text, a preset language model is used to identify the text tags corresponding to the statements in each category; A second model is trained based on the statements and corresponding text labels of each category.

6. The training method for the text processing model according to claim 5, characterized in that, The second prompt text is constructed based on the statements in the third text that have already undergone tag recognition, including: Based on a preset time window, multiple statements are determined from the statements that have completed tag recognition; Based on the multiple statements and their corresponding text tags, the second prompt text is constructed.

7. The training method for the text processing model according to any one of claims 1 to 6, characterized in that, The method includes: Based on the dialogue data within a first preset time period and the fourth text within a second preset time period, the first response text of the fourth text is obtained using the first model, wherein the second preset time period is after the first preset time period. Based on the first response text, the second model is used to determine the tags corresponding to the first response text; The type of the fourth text is determined based on the tags corresponding to the first response text; Based on the type of the fourth text, determine the second response text of the fourth text.

8. The training method for the text processing model according to claim 7, characterized in that, The step of determining the second response text based on the type of the fourth text includes: If the type of the fourth text is a preset type, a text library matching the fourth text is determined based on the tags corresponding to the first response text; Calculate the first similarity between the preset text in the text library and the fourth text; Text with a first similarity greater than a first threshold is identified as the target text. The response text corresponding to the target text is obtained from the text library and used as the second response text.

9. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements a training method for the text processing model as described in any one of claims 1 to 8.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the training method of the text processing model as described in any one of claims 1 to 8.

Images