Text processing method, training data set construction method, dialogue method and related products

Abstract

The invention discloses a text processing method, a training data set construction method, a dialogue method and related products. The method comprises the steps of obtaining a to-be-processed text set, wherein the to-be-processed text set comprises two or more to-be-processed texts; one to-be-processed text is selected from the to-be-processed text set to serve as an initial text, and to-be-processed texts, different from the initial text, in the to-be-processed text set serve as candidate texts; under the condition that the initial text and the candidate text are spliced to serve as a random event and the occurrence probability of the random event is a target probability, whether the initial text and the candidate text are spliced or not is determined; under the condition that it is determined that the initial text and the candidate text are not spliced, taking the initial text as a target text; when it is determined that the initial text and the candidate text are spliced, splicing the initial text and the candidate text to obtain a spliced text; and when the data volume of the spliced text is greater than or equal to a first threshold value, taking the spliced text as a target text.

Description

Technical Field

[0001] The present application relates to the technical field of natural language processing, and in particular to a text processing method, a training data set construction method, a dialogue method and related products. Background Art

[0002] With the development of natural language processing technology, there is an increasing demand for using language models to perform tasks based on text. Before using language models to perform tasks based on text, it is necessary to train the language models with a large amount of text. Therefore, how to obtain a text set that can be used to train language models is of great significance. Summary of the invention

[0003] The present application provides a text processing method, a training data set construction method, a dialogue method and related products to obtain a target text in a text set, wherein the related products include a text processing device, a training data set construction device, a dialogue device, an electronic device, a computer-readable storage medium and a computer program product.

[0004] In a first aspect, a text processing method is provided, the method being used to construct a text set, the text set being a set of target texts, the method comprising:

[0005] Acquire a text set to be processed, wherein the text set to be processed includes two or more texts to be processed, and the data volume of the texts to be processed is less than a first threshold;

[0006] Selecting one of the to-be-processed texts from the to-be-processed text set as an initial text, and the to-be-processed texts in the to-be-processed text set that are different from the initial text are candidate texts;

[0007] In the case where splicing the initial text with the candidate text is regarded as a random event and the probability of occurrence of the random event is a target probability, determining whether to splice the initial text with the candidate text;

[0008] In the case where it is determined not to splice the initial text with the candidate text, taking the initial text as the target text;

[0009] In the case where it is determined to splice the initial text with the candidate text, splicing the initial text with the candidate text to obtain a spliced ​​text;

[0010] When the data volume of the concatenated text is greater than or equal to the first threshold, the concatenated text is used as the target text.

[0011] In combination with any implementation manner of the present application, after obtaining the concatenated text, the method further includes:

[0012] When the data volume of the concatenated text is smaller than the first threshold, the concatenated text is used as the initial text.

[0013] In combination with any implementation manner of the present application, the text to be processed includes a first text and a second text, the data volume of the first text is greater than or equal to a second threshold, the data volume of the second text is less than the second threshold, and the second threshold is less than the first threshold;

[0014] A difference between the quantity of the second text and the quantity of the first text is greater than or equal to a third threshold.

[0015] In combination with any implementation manner of the present application, the step of obtaining a text set to be processed includes:

[0016] Acquire an original text set, where the original text set is a set of the first text and the second text, and the difference between the number of the second text and the number of the first text is greater than or equal to a fourth threshold, and the fourth threshold is greater than the third threshold;

[0017] The to-be-processed text set is obtained by copying the first text in the original text set.

[0018] In combination with any implementation manner of the present application, the step of obtaining the to-be-processed text set by copying the long text in the original text set includes:

[0019] Copying the first text in the original text set to obtain a copy text;

[0020] Adding the copy text to the original text set to obtain an expanded text set;

[0021] Add serial numbers to the texts in the expanded text set to obtain the to-be-processed text set, wherein the serial numbers are the order in which the texts in the to-be-processed text set are selected, and the difference between the serial numbers of any two of the copy texts is greater than or equal to a fifth threshold.

[0022] In combination with any implementation manner of the present application, the target field includes the field involved in the content of the first text and the field involved in the content of the second text, and the target field includes two or more different fields.

[0023] In combination with any embodiment of the present application, when splicing the initial text with the candidate text is regarded as a random event and the probability of occurrence of the random event is a target probability, determining whether to splice the initial text with the candidate text includes:

[0024] Sampling a reference value from a target distribution, wherein the target distribution is a distribution within a preset interval, and the target distribution obeys a uniform distribution within the preset interval;

[0025] Determine the proportion of numbers within the preset interval that are less than or equal to the reference value within the preset interval as a sampling probability;

[0026] When the sampling probability is greater than or equal to the target probability, determining not to splice the initial text with the candidate text;

[0027] When the sampling probability is less than the target probability, it is determined to concatenate the initial text with the candidate text.

[0028] In combination with any implementation manner of the present application, when the sampling probability is greater than or equal to the target probability, determining to splice the initial text with the candidate text includes:

[0029] When the sampling probability is greater than or equal to the target probability, determining a target similarity between the content in the initial text and the content in the candidate text;

[0030] When the target similarity is less than or equal to a sixth threshold, it is determined to concatenate the initial text with the candidate text.

[0031] In a second aspect, a method for constructing a training data set is provided, the method being used to construct a training data set, the training data set being a collection of training data, the training data comprising a first prompt word and a first label, the first prompt word being used to guide a language model to perform a task based on a target text, the first label comprising a result of performing the task based on the target text, the method comprising:

[0032] Acquire a labeled data set, the labeled data set including two or more labeled data, the labeled data including a second prompt word and a second label, the second prompt word is used to guide the language model to perform a task based on the text to be processed, the second label includes a result of performing the task based on the text to be processed, and the data volume of the text to be processed is less than a first threshold;

[0033] Selecting one of the labeled data from the labeled data set as initial data, and the labeled data in the labeled data set that is different from the initial data is candidate data;

[0034] In a case where splicing the initial data with the candidate data is regarded as a random event and the probability of occurrence of the random event is a target probability, determining whether to splice the initial data with the candidate data;

[0035] In a case where it is determined not to splice the initial data with the candidate data, using the initial data as the training data;

[0036] In the case where it is determined to splice the initial data with the candidate data, the initial data and the candidate data are spliced ​​to obtain spliced ​​data, the spliced ​​data includes a third prompt word and a third label, the third prompt word is used to guide the language model to perform a task based on the spliced ​​text, the third label includes a result of performing the task based on the spliced ​​text, the spliced ​​text is obtained by splicing the to-be-processed text of the initial data and the to-be-processed text of the candidate data, and the third label of the spliced ​​data includes the second label of the initial data and the third label of the candidate data;

[0037] When the data volume of the concatenated text is greater than or equal to the first threshold, the third prompt word is used as the first prompt word, and the third label is used as the first label to obtain the training data.

[0038] In combination with any implementation manner of the present application, after obtaining the spliced ​​data, the method further includes:

[0039] When the data volume of the concatenated text is smaller than the first threshold, the concatenated data is used as the initial data.

[0040] In combination with any implementation manner of the present application, the text to be processed includes a first text and a second text, the data volume of the first text is greater than or equal to a second threshold, the data volume of the second text is less than the second threshold, and the second threshold is less than the first threshold;

[0041] A difference between the amount of the labeled data including the second text and the amount of the labeled data including the first text is greater than or equal to a third threshold.

[0042] In combination with any implementation manner of the present application, the step of obtaining a labeled data set includes:

[0043] Acquire an original data set, the original data set being a collection of long data and short data; the long data comprising a fourth prompt word and a fourth label, the fourth prompt word being used to guide the language model to perform a task based on the first text, and the fourth label comprising a result of performing the task based on the first text; the short data comprising a fifth prompt word and a fifth label, the fifth prompt word being used to guide the language model to perform a task based on the second text, and the fifth label comprising a result of performing the task based on the second text; the difference between the amount of the short data and the amount of the long data is greater than or equal to a fourth threshold, and the fourth threshold is greater than the third threshold;

[0044] The labeled data set is obtained by copying the long data in the original data set.

[0045] In combination with any implementation manner of the present application, the step of obtaining the labeled data set by copying the long data in the original data set includes:

[0046] Copying the long data in the original data set to obtain copy data;

[0047] Adding the copy data to the original data set to obtain an expanded data set;

[0048] Add serial numbers to the data in the expanded data set to obtain the labeled data set, where the serial numbers are the order in which the data in the labeled data set are selected, and the difference between the serial numbers of any two of the copy data is greater than or equal to a fifth threshold.

[0049] In combination with any implementation manner of the present application, the target field includes the field involved in the content of the first text and the field involved in the content of the second text, and the target field includes two or more different fields.

[0050] In combination with any embodiment of the present application, when splicing the initial data with the candidate data is regarded as a random event and the probability of occurrence of the random event is a target probability, determining whether to splice the initial data with the candidate data includes:

[0051] Sampling a reference value from a target distribution, wherein the target distribution obeys a uniform distribution within a preset interval;

[0052] Determine the proportion of numbers within the preset interval that are less than or equal to the reference value within the preset interval as a sampling probability;

[0053] When the sampling probability is less than the target probability, determining not to splice the initial data with the candidate data;

[0054] When the sampling probability is greater than or equal to the target probability, it is determined to splice the initial data with the candidate data.

[0055] In combination with any implementation manner of the present application, when the sampling probability is greater than or equal to the target probability, determining to splice the initial data with the candidate data includes:

[0056] In a case where the sampling probability is greater than or equal to the target probability, determining a reference similarity between the content in the initial data and the content in the candidate data;

[0057] When the reference similarity is less than or equal to a sixth threshold, it is determined to splice the initial data with the candidate data.

[0058] A third aspect provides a dialogue method, the method comprising:

[0059] Obtain target questions and reference texts;

[0060] The reference text is input into a target language model so that the target language model answers the target question based on the reference text and obtains a target answer; the target language model is trained using a text set, and the target text in the text set is obtained according to the first aspect and any one of its embodiments; the target language model is also trained using a training data set, and the training data in the training data set is obtained according to the second aspect and any one of its embodiments.

[0061] In a fourth aspect, a text processing device is provided, the text processing device is used to construct a text set, the text set is a collection of target texts, and the text processing device includes:

[0062] An acquisition unit, used for acquiring a text set to be processed, wherein the text set to be processed includes two or more texts to be processed, and the data volume of the texts to be processed is less than a first threshold;

[0063] A processing unit, configured to select one of the to-be-processed texts from the to-be-processed text set as an initial text, and the to-be-processed texts in the to-be-processed text set that are different from the initial text are candidate texts;

[0064] A determination unit, configured to determine whether to splice the initial text with the candidate text when splicing the initial text with the candidate text is regarded as a random event and the probability of occurrence of the random event is a target probability;

[0065] The processing unit is configured to use the initial text as the target text when it is determined that the initial text and the candidate text are not to be spliced ​​together;

[0066] The splicing unit is used for splicing the initial text with the candidate text to obtain a spliced ​​text when it is determined that the initial text and the candidate text are to be spliced;

[0067] The processing unit is configured to use the concatenated text as the target text when the data volume of the concatenated text is greater than or equal to the first threshold.

[0068] In combination with any embodiment of the present application, the processing unit is further used for:

[0069] When the data volume of the concatenated text is smaller than the first threshold, the concatenated text is used as the initial text.

[0070] In combination with any implementation manner of the present application, the text to be processed includes a first text and a second text, the data volume of the first text is greater than or equal to a second threshold, the data volume of the second text is less than the second threshold, and the second threshold is less than the first threshold;

[0071] A difference between the quantity of the second text and the quantity of the first text is greater than or equal to a third threshold.

[0072] In combination with any implementation manner of the present application, the acquisition unit is specifically used to:

[0073] Acquire an original text set, where the original text set is a set of the first text and the second text, and the difference between the number of the second text and the number of the first text is greater than or equal to a fourth threshold, and the fourth threshold is greater than the third threshold;

[0074] The to-be-processed text set is obtained by copying the first text in the original text set.

[0075] In combination with any implementation manner of the present application, the acquisition unit is specifically used to:

[0076] Copying the first text in the original text set to obtain a copy text;

[0077] Adding the copy text to the original text set to obtain an expanded text set;

[0078] Add serial numbers to the texts in the expanded text set to obtain the to-be-processed text set, wherein the serial numbers are the order in which the texts in the to-be-processed text set are selected, and the difference between the serial numbers of any two of the copy texts is greater than or equal to a fifth threshold.

[0079] In combination with any implementation manner of the present application, the target field includes the field involved in the content of the first text and the field involved in the content of the second text, and the target field includes two or more different fields.

[0080] In combination with any implementation manner of the present application, the determining unit is specifically configured to:

[0081] Sampling a reference value from a target distribution, wherein the target distribution is a distribution within a preset interval, and the target distribution obeys a uniform distribution within the preset interval;

[0082] Determine the proportion of numbers within the preset interval that are less than or equal to the reference value within the preset interval as a sampling probability;

[0083] When the sampling probability is greater than or equal to the target probability, determining not to splice the initial text with the candidate text;

[0084] When the sampling probability is less than the target probability, it is determined to concatenate the initial text with the candidate text.

[0085] In combination with any implementation manner of the present application, the determining unit is specifically configured to:

[0086] When the sampling probability is greater than or equal to the target probability, determining a target similarity between the content in the initial text and the content in the candidate text;

[0087] When the target similarity is less than or equal to a sixth threshold, it is determined to concatenate the initial text with the candidate text.

[0088] In a fifth aspect, a device for constructing a training data set is provided, wherein the device for constructing a training data set is used to construct a training data set, wherein the training data set is a collection of training data, wherein the training data set includes a first prompt word and a first label, wherein the first prompt word is used to guide a language model to perform a task based on a target text, and the first label includes a result of performing the task based on the target text, wherein the device for constructing a training data set includes:

[0089] an acquisition unit, configured to acquire a labeled data set, wherein the labeled data set includes two or more labeled data, the labeled data includes a second prompt word and a second label, the second prompt word is used to guide the language model to perform a task based on the text to be processed, the second label includes a result of performing the task based on the text to be processed, and the data volume of the text to be processed is less than a first threshold;

[0090] A processing unit, configured to select one of the labeled data from the labeled data set as initial data, wherein the labeled data in the labeled data set that is different from the initial data is candidate data;

[0091] a determination unit, configured to determine whether to splice the initial data with the candidate data when splicing the initial data with the candidate data is regarded as a random event and the probability of occurrence of the random event is a target probability;

[0092] The processing unit is configured to use the initial data as the training data when it is determined that the initial data is not to be spliced ​​with the candidate data;

[0093] a splicing unit, configured to, when determining to splice the initial data with the candidate data, splice the initial data with the candidate data to obtain spliced ​​data, the spliced ​​data comprising a third prompt word and a third label, the third prompt word being used to guide the language model to perform a task based on the spliced ​​text, the third label comprising a result of performing the task based on the spliced ​​text, the spliced ​​text being obtained by splicing the to-be-processed text of the initial data and the to-be-processed text of the candidate data, the third label of the spliced ​​data comprising the second label of the initial data and the third label of the candidate data;

[0094] The processing unit is configured to, when the data volume of the concatenated text is greater than or equal to the first threshold, use the third prompt word as the first prompt word and the third label as the first label to obtain the training data.

[0095] In combination with any embodiment of the present application, the processing unit is further used for:

[0096] When the data volume of the concatenated text is smaller than the first threshold, the concatenated data is used as the initial data.

[0097] In combination with any implementation manner of the present application, the text to be processed includes a first text and a second text, the data volume of the first text is greater than or equal to a second threshold, the data volume of the second text is less than the second threshold, and the second threshold is less than the first threshold;

[0098] A difference between the amount of the labeled data including the second text and the amount of the labeled data including the first text is greater than or equal to a third threshold.

[0099] In combination with any implementation manner of the present application, the acquisition unit is specifically used to:

[0100] Acquire an original data set, the original data set being a collection of long data and short data; the long data comprising a fourth prompt word and a fourth label, the fourth prompt word being used to guide the language model to perform a task based on the first text, and the fourth label comprising a result of performing the task based on the first text; the short data comprising a fifth prompt word and a fifth label, the fifth prompt word being used to guide the language model to perform a task based on the second text, and the fifth label comprising a result of performing the task based on the second text; the difference between the amount of the short data and the amount of the long data is greater than or equal to a fourth threshold, and the fourth threshold is greater than the third threshold;

[0101] The labeled data set is obtained by copying the long data in the original data set.

[0102] In combination with any implementation manner of the present application, the acquisition unit is specifically used to:

[0103] Copying the long data in the original data set to obtain copy data;

[0104] Adding the copy data to the original data set to obtain an expanded data set;

[0105] Add serial numbers to the data in the expanded data set to obtain the labeled data set, where the serial numbers are the order in which the data in the labeled data set are selected, and the difference between the serial numbers of any two of the copy data is greater than or equal to a fifth threshold.

[0106] In combination with any implementation manner of the present application, the target field includes the field involved in the content of the first text and the field involved in the content of the second text, and the target field includes two or more different fields.

[0107] In combination with any implementation manner of the present application, the determining unit is specifically configured to:

[0108] Sampling a reference value from a target distribution, wherein the target distribution obeys a uniform distribution within a preset interval;

[0109] Determine the proportion of numbers within the preset interval that are less than or equal to the reference value within the preset interval as a sampling probability;

[0110] When the sampling probability is less than the target probability, determining not to splice the initial data with the candidate data;

[0111] When the sampling probability is greater than or equal to the target probability, it is determined to splice the initial data with the candidate data.

[0112] In combination with any implementation manner of the present application, the determining unit is specifically configured to:

[0113] In a case where the sampling probability is greater than or equal to the target probability, determining a reference similarity between the content in the initial data and the content in the candidate data;

[0114] When the reference similarity is less than or equal to a sixth threshold, it is determined to splice the initial data with the candidate data.

[0115] In a sixth aspect, a dialogue device is provided, the dialogue device comprising:

[0116] An acquisition unit, used to acquire target questions and reference texts;

[0117] An answering unit is used to input the reference text into a target language model so that the target language model answers the target question based on the reference text and obtains a target answer; the target language model is obtained by training using a text set, and the target text in the text set is obtained according to the first aspect and any one of its embodiments; the target language model is obtained by training using a training data set, and the training data in the training data set is obtained according to the second aspect and any one of its embodiments.

[0118] In the seventh aspect, an electronic device is provided, comprising: a processor and a memory, the memory being used to store computer program code, the computer program code comprising computer instructions, and when the processor executes the computer instructions, the electronic device executes the above-mentioned first aspect and any embodiment thereof, the electronic device either executes the above-mentioned second aspect and any embodiment thereof, or the electronic device either executes the technical solution of the above-mentioned third aspect.

[0119] In an eighth aspect, another electronic device is provided, comprising: a processor, a sending device, an input device, an output device and a memory, wherein the memory is used to store computer program code, and the computer program code includes computer instructions. When the processor executes the computer instructions, the electronic device executes the above-mentioned first aspect and any embodiment thereof, or the electronic device executes the above-mentioned second aspect and any embodiment thereof, or the electronic device executes the technical solution of the above-mentioned third aspect.

[0120] In the ninth aspect, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a processor, the processor is caused to execute the above-mentioned first aspect and any embodiment thereof, or the processor is caused to execute the above-mentioned second aspect and any embodiment thereof, or the processor is caused to execute the technical solution of the above-mentioned third aspect.

[0121] In the tenth aspect, a computer program product is provided, which includes a computer program or instructions. When the computer program or instructions are run on a computer, the computer executes the above-mentioned first aspect and any embodiment thereof, or the computer executes the above-mentioned second aspect and any embodiment thereof, or the computer executes the technical solution of the above-mentioned third aspect.

[0122] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application.

[0123] In the present application, the set of texts to be processed includes two or more texts to be processed, wherein the data volume of the texts to be processed is less than the first threshold value, that is, the set of texts to be processed includes two or more short texts. After obtaining the set of texts to be processed, the text processing device selects a text to be processed from the set of texts to be processed as the initial text, and the text to be processed that is different from the initial text in the set of texts to be processed is the candidate text. In the case where the splicing of the initial text and the candidate text is regarded as a random event, and the probability of the occurrence of the random event is the target probability, it is determined whether to splice the initial text with the candidate text, so that it is possible to splice the initial text with the candidate text, and it is also possible not to splice the initial text with the candidate text. In the case of determining not to splice the initial text with the candidate text, the initial text is used as the target text, thereby making the target text a short text, thereby making the text set include the short text. In the case of determining to splice the initial text with the candidate text, the initial text and the candidate text are spliced ​​to obtain the spliced ​​text. In the case where the data volume of the spliced ​​text is greater than or equal to the first threshold value, the spliced ​​text is used as the target text, thereby making the target text a long text, thereby making the text set include the long text. In general, by obtaining the target text through the embodiments of the present application, the target text can be either a short text or a long text, so that when a text set is a collection of target texts, the text set can include both long texts and short texts. BRIEF DESCRIPTION OF THE DRAWINGS

[0124] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the background technology, the drawings required for use in the embodiments of the present application or the background technology will be described below.

[0125] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0126] Figure 1 A flowchart of a text processing method provided in an embodiment of the present application;

[0127] Figure 2 A flowchart of another text processing method provided in an embodiment of the present application;

[0128] Figure 3 A flowchart of another text processing method provided in an embodiment of the present application;

[0129] Figure 4 A flowchart of another method for constructing a training data set provided in an embodiment of the present application;

[0130] Figure 5 A flowchart of a dialogue method provided in an embodiment of the present application;

[0131] Figure 6 A structural diagram of a text processing device provided in an embodiment of the present application;

[0132] Figure 7 A device for constructing a training data set provided in an embodiment of the present application;

[0133] Figure 8 A conversation device provided in an embodiment of the present application;

[0134] Fig. 9 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0135] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0136] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0137] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0138] The embodiment of the present application provides a text processing method, the execution subject of the text processing method is a text processing device, wherein the text processing device can be any electronic device that can execute the technical solution disclosed in the embodiment of the method of the present application. Optionally, the text processing device can be one of the following: a computer, a server.

[0139] It should be understood that the method embodiment of the present application can also be implemented by a processor executing a computer program code. The following describes the embodiment of the present application in conjunction with the drawings in the embodiment of the present application. Figure 1 , Figure 1 A flowchart of a text processing method provided in an embodiment of the present application.

[0140] 101. Obtain the text set to be processed.

[0141] In an embodiment of the present application, the set of texts to be processed includes two or more texts to be processed, wherein the data volume of the texts to be processed is less than a first threshold value. The first threshold value is a basis for measuring whether the data volume of a text is large or small. Specifically, if the data volume of a text is greater than or equal to the first threshold value, it means that the data volume of the text is large, and if the data volume of a text is less than the first threshold value, it means that the data volume of the text is small. For the convenience of expression, the text with a large data volume is referred to as a long text below, and the text with a small data volume is referred to as a short text. The texts to be processed in the set of texts to be processed are all short texts.

[0142] In an implementation of obtaining a text set to be processed, a text processing device receives the text set to be processed input by a user through an input component. The input component includes at least one of the following: a keyboard, a mouse, a touch screen, a touch pad, and an audio input device.

[0143] In another implementation of obtaining the to-be-processed text set, the text processing apparatus receives the to-be-processed text set sent by a terminal. The terminal may be any of the following: a mobile phone, a computer, a tablet computer, or a server.

[0144] 102. Select one of the above-mentioned to-be-processed texts from the above-mentioned to-be-processed text set as the initial text.

[0145] In the embodiment of the present application, the initial text is any text in the set of to-be-processed texts. The text processing device selects one of two or more to-be-processed texts in the set of to-be-processed texts as the initial text. The to-be-processed text in the set of to-be-processed texts that is different from the initial text is a candidate text.

[0146] 103. When splicing the initial text with the candidate text is regarded as a random event and the probability of occurrence of the random event is a target probability, determine whether to splice the initial text with the candidate text.

[0147] In the embodiment of the present application, the text processing device regards the splicing of the initial text and the candidate text as a random event, so the splicing of the initial text and the subsequent text may or may not occur. The probability of the occurrence of the random event is the target probability, that is, the text processing device regards the probability of the splicing of the initial text and the candidate text as the target probability. The text processing device can determine whether to splice the initial text and the candidate text by executing step 103.

[0148] It should be understood that determining whether the initial text and the candidate text are spliced ​​can occur once or more than once, and in the case of occurring once or more than once, the probability of splicing the initial text and the candidate text is determined to be the target probability. For example, the text processing device performs 10 judgments, wherein each judgment is to determine whether to splice the initial text and the candidate text. If the target probability is 0.8, then the result of 8 of the 10 judgments is that the initial text and the candidate text are spliced.

[0149] 104. When it is determined that the initial text is not to be concatenated with the candidate text, the initial text is used as the target text.

[0150] When the execution result of step 103 is to determine that the initial text is not to be spliced ​​with the candidate text, the text processing device uses the initial text as the target text. At this time, the data volume of the target text is less than the first threshold, that is, the target text is a short text.

[0151] 105. When it is determined that the initial text and the candidate text are to be spliced, the initial text and the candidate text are spliced ​​to obtain a spliced ​​text.

[0152] When the execution result of step 103 is to determine that the initial text and the candidate text are to be spliced, the text processing device splices the initial text and the candidate text to obtain a spliced ​​text, wherein the spliced ​​text includes the initial text and the candidate text.

[0153] 106. When the data volume of the concatenated text is greater than or equal to the first threshold, use the concatenated text as the target text.

[0154] If the data volume of the concatenated text is greater than or equal to the first threshold, it means that the concatenated text is a long text. Therefore, when the data volume of the concatenated text is greater than or equal to the first threshold, the text processing device uses the concatenated text as the target text, so that the target text can be a long text.

[0155] In the embodiment of the present application, the set of texts to be processed includes two or more texts to be processed, wherein the data volume of the texts to be processed is less than the first threshold value, that is, the set of texts to be processed includes two or more short texts. After obtaining the set of texts to be processed, the text processing device selects a text to be processed from the set of texts to be processed as the initial text, and the text to be processed that is different from the initial text in the set of texts to be processed is the candidate text. In the case where the splicing of the initial text and the candidate text is regarded as a random event, and the probability of the occurrence of the random event is the target probability, it is determined whether to splice the initial text with the candidate text, so that it is possible to splice the initial text with the candidate text, and it is also possible not to splice the initial text with the candidate text. In the case of determining not to splice the initial text with the candidate text, the initial text is used as the target text, thereby making the target text a short text, thereby making the text set include the short text. In the case of determining to splice the initial text with the candidate text, the initial text and the candidate text are spliced ​​to obtain the spliced ​​text. In the case where the data volume of the spliced ​​text is greater than or equal to the first threshold value, the spliced ​​text is used as the target text, thereby making the target text a long text, thereby making the text set include the long text. In general, by obtaining the target text through the embodiments of the present application, the target text can be either a short text or a long text, so that when a text set is a collection of target texts, the text set can include both long texts and short texts.

[0156] As an optional implementation, after obtaining the concatenated text, the text processing device further performs the following steps:

[0157] 2001. When the data volume of the concatenated text is less than the first threshold, the concatenated text is used as the initial text.

[0158] When the data volume of the concatenated text is less than the first threshold, the text processing device uses the concatenated text as the initial text, so that the text processing device updates the initial text and iteratively executes steps 103, 104, 105, and 106.

[0159] For the convenience of description, the initial text in step 102 is referred to as the initial text of the first iteration, the candidate text in step 102 is referred to as the candidate text of the first iteration, and the initial text obtained by executing step 2001 is referred to as the initial text of the second iteration. The candidate text corresponding to the initial text of the second iteration is referred to as the candidate text of the second iteration.

[0160] Since the concatenated text includes the initial text of the first iteration and the candidate text of the first iteration, when the initial text of the second iteration is obtained by executing step 2001, the initial text of the second iteration also includes the initial text of the first iteration and the candidate text of the first iteration. Because the initial text of the second iteration has changed compared to the initial text of the first iteration, the candidate text of the second iteration has also changed compared to the candidate text of the first iteration. Specifically, the candidate text of the second iteration is the text in the to-be-processed text set except the initial text of the first iteration and the candidate text of the first iteration.

[0161] For example, the set of texts to be processed includes text a to be processed, text b to be processed, and text c to be processed. The text processing device first selects text a to be processed as the initial text, that is, text a to be processed is the initial text of the first iteration. At this time, text b to be processed and text c to be processed are both candidate texts, that is, text b to be processed and text c to be processed are both candidate texts of the first iteration. If text a to be processed and text b to be processed are concatenated to obtain a concatenated text, and the concatenated text is used as the initial text, then the initial text of the second iteration includes text a to be processed and text b to be processed. At this time, text d to be processed is a candidate text, that is, text d to be processed is a candidate text of the second iteration.

[0162] In this embodiment, when the data volume of the concatenated text is less than the first threshold, the text processing device uses the concatenated text as the initial text, so that the text processing device can update the initial text and iteratively execute steps 103, 104, 105, and 106, thereby obtaining a target text with a data volume between the initial text of the first iteration and the long text. For example, the data volume of the initial text of the first iteration is 0.3K characters (tokens), where K represents one thousand. The first threshold is 32K tokens, so by executing step 2001, a target text with a data volume between 0.3K tokens and 32K tokens can be obtained. This can make the data volume of the target text more diversified, thereby making the data volume of the target text in the text set more diversified.

[0163] As an optional implementation, the text to be processed includes a first text and a second text, wherein the data volume of the first text is greater than or equal to a second threshold, the data volume of the second text is less than the second threshold, and the second threshold is less than the first threshold. In this implementation, the text to be processed is further divided into two categories: the data volume is less than the second threshold and the data volume is greater than or equal to the second threshold. For example, the first threshold is 32K tokens and the second threshold is 1K tokens.

[0164] The difference between the number of the second text and the number of the first text is greater than or equal to the third threshold. In the embodiment of the present application, the third threshold is a basis for measuring whether the difference is large or small. Specifically, a difference greater than or equal to the third threshold indicates that the difference is large, and a difference less than the third threshold indicates that the difference is small. In one possible implementation, the difference between the number of the second text and the number of the first text is the difference between the number of the second text and the number of the first text. For example, the third threshold is 400, and the difference between the number of the second text and the number of the first text is greater than or equal to 400. In another possible implementation, the difference between the number of the second text and the number of the first text is the ratio of the number of the second text to the number of the first text. For example, the third threshold is 20, and the ratio of the number of the second text to the number of the first text is greater than or equal to 20.

[0165] The data volume of the first text is greater than or equal to the second threshold, and the data volume of the second text is less than the second threshold, which means that the first text is a longer text in the set of texts to be processed (hereinafter referred to as the longer text), and the second text is a shorter text in the set of texts to be processed (hereinafter referred to as the shorter text). Since the longer text is not a text obtained by splicing, that is, the longer text is not a synthetic text, generating a long text based on the longer text can improve the authenticity of the long text and improve the efficiency of generating the long text.

[0166] The difference between the number of the second text and the number of the first text is greater than or equal to the third threshold, indicating that in the text set to be processed, the number of shorter texts differs greatly from the number of longer texts. At this time, based on the text processing method described above, the long text is obtained by using the text in the text set to be processed, and the long text can be obtained based on a small amount of longer text. This can not only improve the authenticity of the long text, but also improve the efficiency of generating the long text, and can also reduce the difficulty of generating the long text.

[0167] As an optional implementation, the text processing device performs the following steps during the process of executing step 101:

[0168] 3001. Obtain the original text set.

[0169] In the embodiment of the present application, the original text set is a set of the first text and the second text, the difference between the number of the second text and the number of the first text is greater than or equal to the fourth threshold, and the fourth threshold is greater than the third threshold. Since the fourth threshold is greater than the third threshold, the difference between the number of the second text and the number of the first text in the original text set is greater than the difference between the number of the second text and the number of the first text in the text set to be processed, that is, the proportion of the second text in the original text set is small.

[0170] 3002. Obtain the to-be-processed text set by copying the first text in the original text set.

[0171] In order to reduce the difference between the number of the second texts and the number of the first texts, the text processing device obtains the to-be-processed text set by copying the first text in the original text set.

[0172] In a possible implementation, the text processing device copies the first text in the original text set to obtain a copy text. The copy text is added to the original text set to obtain an extended text set. A sequence number is added to the text in the extended text set to obtain a to-be-processed text set, wherein the sequence number is the order in which the text in the to-be-processed text set is selected, and the difference between the sequence numbers of any two copy texts is greater than or equal to a fifth threshold.

[0173] The above-mentioned selected order refers to being selected as the initial text, or being selected as the candidate text to be spliced ​​with the initial text. For example, the set of texts to be processed includes text a to be processed, text b to be processed, and text c to be processed, wherein the order in which text a to be processed is selected is before the order in which text b to be processed is selected, and the order in which text b to be processed is selected is before the order in which text c to be processed is selected. Then the text processing device first selects text a to be processed as the initial text, and then selects text b to be processed as the candidate text. After splicing text a to be processed and text b to be processed to obtain a spliced ​​text, and using the spliced ​​text as the initial text, text c to be processed is selected as the candidate text, and it is determined whether to splice the initial text with the candidate text.

[0174] Considering that the contents of the transcript texts are all the same, if two identical transcript texts are selected from the set of texts to be processed, and the target text is obtained by splicing the two identical transcript texts, then the repetition of the content in the target text is likely to be high, thereby reducing the quality of the target text. Since the sequence number is the order in which the texts in the set of texts to be processed are selected, the difference in the sequence numbers of any two transcript texts is greater than or equal to the fifth threshold, so the interval between any two transcript texts being selected can be increased. This can reduce the probability of obtaining the target text by splicing two identical transcript texts, thereby reducing the repetition of the content in the target text, thereby improving the quality of the target text.

[0175] In this embodiment, the original text set is a set of a first text and a second text, the difference between the number of the second text and the number of the first text is greater than or equal to a fourth threshold, and the fourth threshold is greater than the third threshold. After obtaining the original text set, the text processing device can reduce the difference between the number of the second text and the number of the first text by copying the first text in the original text set, thereby obtaining the text set to be processed.

[0176] As an optional implementation, the target field includes the field involved in the content of the first text and the field involved in the content of the second text, wherein the target field includes two or more different fields. In this implementation, the target field includes two or more different fields, which can increase the diversity of the field involved in the content of the first text and the field involved in the content of the second text, and further increase the diversity of the fields involved in the texts in the text set to be processed.

[0177] For example, the original text set includes a first text a and a second text b, wherein the content of the first text a relates to the legal field, and the content of the second text b relates to the sports field.

[0178] Optionally, the first text is determined based on a preset category system, and the field involved in the first text matches the field in the preset category system.

[0179] Optionally, the preset category system includes primary categories and secondary categories, wherein the secondary categories are subcategories of the primary categories, and the secondary categories also include leaf nodes. The primary categories can be determined according to the purpose of the text. For example, if the text is used as training data to train a language model so that the language model performs a single-text task, or the language model performs a cross-text task, wherein the language model is a model with natural language processing capabilities, a single-text task refers to a language processing task based on one text, and a cross-text task refers to a language processing task based on two or more texts. Then the purpose of the text includes single-text tasks and cross-text tasks. At this time, the primary category includes single-text tasks and cross-text tasks.

[0180] Secondary categories can further refine primary categories. For example, for single-text tasks, they can be divided into covering answers and extracting summaries, where covering answers means answering questions with a text, and extracting summaries means extracting a summary of a text. In this case, the first-level category of single-text tasks includes the following second-level categories: covering answers and extracting summaries. For another example, for single-text tasks, they can include composite tasks, where composite tasks mean answering questions with two or more texts, and extracting summaries means extracting a summary of a text. In this case, the first-level category of single-text tasks includes the following second-level categories: covering answers and extracting summaries.

[0181] The leaf nodes of the secondary category can be used to represent the fields involved in the tasks in the secondary category. For example, the fields involved include plots, admission brochures, and judicial documents. Optionally, one leaf node represents one field, and there are 52 leaf nodes in total. Optionally, the content of the first text involves the fields represented by all leaf nodes, and the number of first texts corresponding to the fields represented by each leaf node is 20, which can make the fields involved in the content of the first text more diversified.

[0182] As an optional implementation, the text processing device performs the following steps during the process of executing step 103:

[0183] 4001. Sample from the target distribution to obtain a reference value.

[0184] In an embodiment of the present application, the target distribution is a distribution within a preset interval, and the target distribution obeys a uniform distribution within the preset interval. For example, if the preset interval is an interval from 0 to 1, then the target distribution is a distribution within 0 to 1, and the target distribution obeys a uniform distribution within the interval from 0 to 1. Since the target distribution is a distribution within a preset interval, sampling from the target distribution is sampling from the preset interval. And since the target distribution obeys a uniform distribution within the preset interval, when sampling from the target distribution, the probability of any value within the preset interval being sampled is the same. In step 401, the text processing device obtains a reference value by sampling from the target distribution.

[0185] 4002. Determine the proportion of numbers within the preset interval that are less than or equal to the reference value within the preset interval as the sampling probability.

[0186] Since the probability of any value in the preset interval being sampled is the same, the proportion of the numbers in the preset interval that are less than or equal to the reference value is the probability of sampling from the target distribution to obtain a value less than or equal to the reference value. In other words, the sampling probability is the probability of sampling from the target distribution to obtain a value less than or equal to the reference value.

[0187] 4003. When the sampling probability is greater than or equal to the target probability, determine not to splice the initial text with the candidate text.

[0188] 4004. When the sampling probability is less than the target probability, determine to concatenate the initial text with the candidate text.

[0189] Because the sampling probability is the proportion of the number of numbers less than or equal to the reference value in the preset interval, if the sampling probability is the target probability, the proportion of the number of numbers less than or equal to the reference value in the preset interval is the target probability. Therefore, if the sampling probability greater than or equal to the target probability is called a reference event, then the probability of the reference event occurring is: (1-target probability), and the probability of the reference event occurring is the target probability. For example, the target probability is 0.8, and the preset interval is an interval from 0 to 1. Then when the reference value sampled from the preset interval is 0.8, the sampling probability is 0.8, that is, the sampling probability is the target probability. When the reference value sampled from the preset interval is greater than or equal to 0.8, the sampling probability is greater than or equal to 0.8. The probability that the reference value sampled from the preset interval is greater than or equal to 0.8 is (1-0.8) = 0.2, so the probability of the reference event occurring is 0.2.

[0190] In step 4003, the text processing device determines not to splice the initial text with the candidate text when the sampling probability is greater than or equal to the target probability, that is, the text processing device determines not to splice the initial text with the candidate text when the reference event occurs. The probability of the reference event occurring is: (1-target probability), so the probability of not splicing the initial text with the candidate text is determined to be: (1-target probability). Conversely, in step 4004, the text processing device determines to splice the initial text with the candidate text when the sampling probability is less than the target probability, that is, the text processing device determines to splice the initial text with the candidate text when the reference event does not occur. The probability that the reference event does not occur is the target probability, so the probability of splicing the initial text with the candidate text is determined to be the target probability. Because the splicing of the initial text with the candidate text is determined to be a random event, the probability of the random event occurring is the target probability.

[0191] In one possible implementation, the text processing device determines the target similarity between the content in the initial text and the content in the candidate text when the sampling probability is greater than or equal to the target probability, and determines to splice the initial text with the candidate text when the target similarity is less than or equal to a sixth threshold.

[0192] In the embodiment of the present application, the sixth threshold is the basis for judging whether the target similarity is large or small. Specifically, if the target similarity is greater than the sixth threshold, it means that the target similarity is large. Conversely, if the target similarity is less than or equal to the sixth threshold, it means that the target similarity is small. If the target similarity is small, it means that the similarity between the content in the initial text and the content in the candidate text is low, which means that the difference between the content in the initial text and the content in the candidate text is large. At this time, by splicing the initial text with the candidate text, the span of the content in the spliced ​​text can be made larger, so that the spliced ​​text can better simulate the topic switching in the real dialogue scene. For example, the content in the initial text is about travel, and the content in the candidate text is about food. By splicing the initial text with the candidate text, the content in the spliced ​​text can be switched from travel to food, thereby simulating the real scene, switching the conversation topic from travel to food.

[0193] In this embodiment, the text processing device samples a reference value from a target distribution, wherein the target distribution is a distribution within a preset interval, and the target distribution obeys a uniform distribution within the preset interval. The proportion of numbers within the preset interval that are less than or equal to the reference value is determined as the sampling probability. When the sampling probability is greater than or equal to the target probability, it is determined not to splice the initial text with the candidate text. When the sampling probability is less than the target probability, it is determined to splice the initial text with the candidate text. In this way, the probability of splicing the initial text with the candidate text can be determined to be the target probability.

[0194] See also Figure 2 , Figure 2 A flowchart of another text processing method provided in an embodiment of the present application. Figure 2 As shown, the text set to be processed includes a first text and a second text, and the proportion of the first text in the text set to be processed is smaller than the proportion of the second text in the text set to be processed. In the first iteration, a text is first selected from the text set to be processed as the initial text, and then it is determined whether P~U(0,1)<0.8 is established, where P~U(0,1) means that when the preset interval is 0 to 1, P follows a uniform distribution between 0 and 1. 0.8 is the above-mentioned target probability, and P~U(0,1)<0.8 means that after sampling a reference value from the target distribution, the proportion of the number less than or equal to the reference value in the preset interval is determined as the sampling probability, and the sampling probability is less than 0.8. If the judgment result is no, that is, the sampling probability is greater than or equal to 0.8, then the initial text is added to the text set as the target text. If the judgment result is yes, that is, the sampling probability is less than 0.8, the initial text is spliced ​​with the candidate text. The spliced ​​text can be obtained by splicing. Then determine whether the concatenated text meets the following conditions: the data volume of the concatenated text is less than 32K, where 32K represents 32K tokens. If the judgment result is no, the concatenated text is used as the initial text, and the iterative execution continues to determine whether P~U(0,1)<0.8 is established. If the judgment result is yes, the concatenated text is added to the text set as the target text. Figure 2 The text processing method shown can make the text set include both target texts with a data volume greater than 32K and target texts with a data volume less than or equal to 32K, and can also enrich the diversity of the data volume of the target texts in the text set.

[0195] As an optional implementation, after obtaining a text set by the text processing method described above, the text set can be used to train a language model to improve the processing effect of the language model on the text. Specifically, the target text in the text set is annotated to obtain an annotated text set. Then, the annotated text set is used to train the language model. It should be understood that the device for performing the training of the language model using the annotated text set can be different from or the same as the text processing device.

[0196] The embodiment of the present application also provides a method for constructing a training data set, the method is used to construct a training data set, the training data set is a collection of training data, the training data includes a first prompt word and a first label, the first prompt word is used to guide the language model to perform a task based on the target text, and the first label includes the result of performing the task based on the target text. Optionally, the language model is a large language model (LLM).

[0197] It should be understood that the prompt words in the embodiments of the present application all include the text required for executing the task and the task to be executed. For example, the first prompt word includes the target text and the task to be executed.

[0198] In one possible implementation, the task is to answer a question, the first prompt word includes the target text and the question to be answered, and the first prompt word can be used to guide the language model to answer the question based on the target text. In this case, the task is to answer the question, and the first label includes the answer to the question based on the target text. For example, the first prompt word includes the target text and the following question: According to the target text, who swims faster, A or B? The first label is: A and B, they are both outstanding representatives in the sports world, but they play different sports. A is a swimmer who has outstanding performance in swimming. B is a basketball player who has outstanding performance on the basketball court. Therefore, A will be faster than B in swimming.

[0199] The execution subject of the training data set construction method is a training data set construction device (hereinafter referred to as a construction device), wherein the construction device can be any electronic device that can execute the technical solution disclosed in the method embodiment of the present application. Optionally, the construction device can be one of the following: a computer, a server.

[0200] See also Figure 3 , Figure 3 A flowchart of another text processing method provided in an embodiment of the present application.

[0201] 301. Obtain a labeled dataset.

[0202] In an embodiment of the present application, the labeled data set includes two or more labeled data, wherein the labeled data includes a second prompt word and a second label, and the second prompt word is used to guide the language model to perform a task based on the text to be processed, that is, the second prompt word includes the text to be processed and the task to be performed, and the second label includes the result of performing the task based on the text to be processed, wherein the data volume of the text to be processed is less than the first threshold. For the description of the text to be processed, please refer to the description of the text to be processed in step 101, which will not be repeated here.

[0203] As described in step 101, a text with a large amount of data is called a long text, and a text with a small amount of data is called a short text. For the convenience of expression, the data including the long text will be referred to as long data, and the data including the short text will be referred to as short data.

[0204] 302. Select one of the labeled data sets as initial data.

[0205] In the embodiment of the present application, the initial data is any data in the annotated data set. The construction device selects one of two or more to-be-processed texts in the to-be-processed text set as the initial text. The annotated data in the annotated data set that is different from the initial data is the candidate data.

[0206] 303. When splicing the initial data with the candidate data is regarded as a random event and the probability of occurrence of the random event is a target probability, determine whether to splice the initial data with the candidate data.

[0207] The implementation principle of this step is the same as that of step 103. Specifically, the initial data in this step corresponds to the initial text in step 103, and the candidate data in this step corresponds to the candidate text in step 103. By executing step 303, the construction device can determine whether to splice the initial data with the candidate data.

[0208] In the embodiment of the present application, splicing the initial data with the candidate data includes splicing the to-be-processed text in the initial data with the to-be-processed text in the candidate data, and splicing the second label of the initial data with the second label of the candidate data.

[0209] 304. When it is determined that the initial data is not to be concatenated with the candidate data, use the initial data as training data.

[0210] When the execution result of step 303 is to determine that the initial data and the candidate data are not to be spliced, the construction device uses the initial data as training data. At this time, the target text in the training data is the text to be processed in the initial data, the first label of the training data is the second label of the initial text, the data volume of the target text is less than the first threshold, and the target text is a short text.

[0211] 305. When it is determined that the initial data and the candidate data are to be spliced, the initial data and the candidate data are spliced ​​to obtain spliced ​​data.

[0212] When the execution result of step 303 is to determine that the initial data and the candidate data are to be spliced, the construction device splices the initial data and the candidate data to obtain spliced ​​data.

[0213] In an embodiment of the present application, the spliced ​​data includes a third prompt word and a third label, wherein the third prompt word is used to guide the language model to perform a task based on the spliced ​​text, the third label includes the result of performing the task based on the spliced ​​text, and the spliced ​​text is obtained by splicing the to-be-processed text of the initial data and the to-be-processed text of the candidate data, that is, the spliced ​​text includes the to-be-processed text in the initial data and the to-be-processed text in the candidate data, and the third label of the spliced ​​data includes the second label of the initial data and the third label of the candidate data.

[0214] For example, the second prompt word of the initial data includes the text to be processed a and task b, and the second label of the initial data is the result of executing task b based on the text to be processed a. The second prompt word of the candidate text includes the text to be processed c and task d, and the second label of the candidate data is the result of executing task d based on the text to be processed c. The third prompt word of the spliced ​​data includes the spliced ​​text and task e, wherein the spliced ​​text includes the text to be processed a and the text to be processed c, and the third label of the spliced ​​data includes the result of executing task b based on the text to be processed a and the result of executing task d based on the text to be processed c.

[0215] For another example, the second prompt word of the initial data includes the text to be processed a and task b, and the second label of the initial data is the result of executing task b based on the text to be processed a. The second prompt word of the candidate text includes the text to be processed c and task b, and the second label of the candidate data is the result of executing task b based on the text to be processed c. The third prompt word of the spliced ​​data includes the spliced ​​text and task b, wherein the spliced ​​text includes the text to be processed a and the text to be processed c, and the third label of the spliced ​​data includes the result of executing task b based on the text to be processed a and the result of executing task b based on the text to be processed c.

[0216] 306. When the data volume of the concatenated text is greater than or equal to the first threshold, use the third prompt word as the first prompt word and the third label as the first label to obtain the training data.

[0217] If the data volume of the concatenated text is greater than or equal to the first threshold, it means that the concatenated text is a long text, that is, the concatenated data is long data. Therefore, when the data volume of the concatenated text is greater than or equal to the first threshold, the construction device uses the concatenated data as training data, so that the training data can be long data. Specifically, the construction device uses the third prompt word as the first prompt word and the third label as the first label to obtain the training data.

[0218] In an embodiment of the present application, the labeled data set includes two or more labeled data, and the labeled data includes a second prompt word and a second label, wherein the second prompt word is used to guide the language model to perform a task based on the text to be processed, and the second label includes the result of performing the task based on the text to be processed, and the data volume of the text to be processed is less than the first threshold value, that is, the labeled data is short data, and the labeled data set includes two or more short data. After acquiring the labeled data set, the construction device selects a labeled data from the labeled data set as the initial data, and the labeled data different from the initial data in the labeled data set is the candidate data. In the case of splicing the initial data with the candidate data as a random event, and the probability of the random event occurring is the target probability, it is determined whether to splice the initial data with the candidate data, so that it is possible to splice the initial data with the candidate data, and it is also possible not to splice the initial data with the candidate data. In the case of determining not to splice the initial data with the candidate data, the initial data is used as training data, thereby making the training data short data, and then making the training data set include short data. In the case of determining to splice the initial data with the candidate data, the initial data and the candidate data are spliced ​​to obtain spliced ​​data. When it is determined that the initial data and the candidate data are to be spliced, the initial data and the candidate data are spliced ​​to obtain spliced ​​data, and when the amount of the spliced ​​text in the spliced ​​data is greater than or equal to the first threshold, the spliced ​​data is used as training data, thereby making the training data long data, and further making the training data set include long data. In general, by obtaining training data through the embodiments of the present application, the training data can be short data or long data, and thus when the training data set is a set of training data, the training data set can include both long data and short data.

[0219] Based on the technical solutions provided in steps 101 to 106 and the technical methods provided in steps 301 to 306, it can be seen that the text processing method and the training data set construction method provided in the embodiment of the present application are two technical solutions under one inventive concept.

[0220] As an optional implementation, after obtaining the splicing data, the construction device further performs the following steps:

[0221] 5001. When the data volume of the above-mentioned spliced ​​text is less than the above-mentioned first threshold, the above-mentioned spliced ​​data is used as the above-mentioned initial data.

[0222] When the data volume of the spliced ​​text is less than the first threshold, the construction device uses the spliced ​​data as initial data, so that the construction device can update the initial data and iteratively execute steps 303, 304, 305, and 306.

[0223] For the convenience of description, the initial data in step 302 is referred to as the initial data of the first iteration, the candidate data in step 102 is referred to as the candidate data of the first iteration, and the initial data obtained by executing step 2001 is referred to as the initial data of the second iteration. The candidate data corresponding to the initial data of the second iteration is referred to as the candidate data of the second iteration.

[0224] Since the spliced ​​data includes the initial data of the first iteration and the candidate data of the first iteration, when the initial data of the second iteration is obtained by executing step 5001, the initial data of the second iteration also includes the initial data of the first iteration and the candidate data of the first iteration. Because the initial data of the second iteration has changed compared to the initial data of the first iteration, the candidate data of the second iteration has also changed compared to the candidate data of the first iteration. Specifically, the candidate data of the second iteration is the data in the labeled data set except the initial data of the first iteration and the candidate data of the first iteration.

[0225] For example, the labeled data set includes labeled data a, labeled data b, and labeled data c. The data processing device first selects labeled data a as initial data, that is, labeled data a is the initial data of the first iteration. At this time, labeled data b and labeled data c are both candidate data, that is, labeled data b and labeled data c are both candidate data of the first iteration. If labeled data a and labeled data b are spliced ​​to obtain spliced ​​data, and the spliced ​​data is used as initial data, then the initial data of the second iteration includes labeled data a and labeled data b. At this time, labeled data d is candidate data, that is, labeled data d is candidate data of the second iteration.

[0226] In this embodiment, when the data volume of the concatenated text is less than the first threshold, the construction device uses the concatenated data as the initial data, so that the construction device can update the initial data and iteratively execute steps 303, 304, 305, and 306, thereby making the data volume of the target text in the training data volume between the text in the initial data of the first iteration and the long text. For example, the data volume of the text in the initial data of the first iteration is 0.3K tokens. The first threshold is 32K tokens, so by executing step 5001, a target text with a data volume between 0.3K tokens and 32K tokens can be obtained. In this way, the data volume of the target text can be more diversified, thereby making the data volume of the target text of the training data more diversified.

[0227] As an optional implementation, the text to be processed includes a first text and a second text, wherein the data volume of the first text is greater than or equal to a second threshold, the data volume of the second text is less than the second threshold, and the second threshold is less than the first threshold. In this implementation, the text to be processed is further divided into two categories: the data volume is less than the second threshold and the data volume is greater than or equal to the second threshold. For example, the first threshold is 32K tokens and the second threshold is 1K tokens.

[0228] The difference between the number of labeled data including the second text and the number of labeled data including the first text is greater than or equal to the third threshold. In the embodiment of the present application, the third threshold is the basis for measuring whether the difference is large or small. Specifically, the difference is greater than or equal to the third threshold, indicating that the difference is large, and the difference is less than the third threshold, indicating that the difference is small. In one possible implementation, the difference between the number of labeled data including the second text and the number of labeled data including the first text is the difference between the number of labeled data including the second text and the number of labeled data including the first text. For example, the third threshold is 400, at which time the difference between the number of labeled data including the second text and the number of labeled data including the first text is greater than or equal to 400. In another possible implementation, the difference between the number of labeled data including the second text and the number of labeled data including the first text is the ratio of the number of labeled data including the second text to the number of labeled data including the first text. For example, the third threshold is 20, at which time the ratio of the number of labeled data including the second text to the number of labeled data including the first text is greater than or equal to 20.

[0229] If the data volume of the first text is greater than or equal to the second threshold, and the data volume of the second text is less than the second threshold, it means that the first text is a longer text and the second text is a shorter text. Since the longer text is not a text obtained by splicing, that is, the longer text is not a synthetic text, generating a long text based on the longer text can improve the authenticity of the long text and improve the efficiency of generating the long text. For the convenience of expression, the labeled data including the longer text will be referred to as the longer data below, and the labeled data including the shorter text will be referred to as the shorter data below.

[0230] The difference between the number of annotated data including the second text and the number of annotated data including the first text is greater than or equal to the third threshold, indicating that in the annotated data set, the number of shorter data differs greatly from the number of longer data. At this time, based on the construction method of the training data set described above, the annotated data in the annotated data set is used to obtain long data, and it is possible to obtain long data based on a small amount of longer data. This can not only improve the authenticity of the long text in the long data, but also improve the efficiency of generating long data and reduce the difficulty of generating long data.

[0231] As an optional implementation manner, the construction device performs the following steps during the execution of step 301:

[0232] 6001. Obtain the original data set.

[0233] In the embodiment of the present application, the original data set is a collection of longer data and shorter data. The longer data includes a fourth prompt word and a fourth label, the fourth prompt word is used to guide the language model to perform a task based on the first text, and the fourth label includes the result of performing the task based on the first text. The shorter data includes a fifth prompt word and a fifth label, the fifth prompt word is used to guide the language model to perform a task based on the second text, and the fifth label includes the result of performing the task based on the second text. The difference between the number of shorter data and the number of longer data is greater than or equal to the fourth threshold, and the fourth threshold is greater than the third threshold.

[0234] Since the fourth threshold is greater than the third threshold, the difference between the number of shorter data and the number of longer data in the original data set is greater than the difference between the number of shorter data and the number of longer data in the labeled data set, that is, the proportion of longer data in the original data set is small.

[0235] 6002. Obtain the labeled data set by copying the longer data in the original data set.

[0236] In order to reduce the difference between the amount of shorter data and the amount of longer data, the construction device obtains the labeled data set by copying the longer data in the original data set.

[0237] In one possible implementation, the construction device copies the longer data in the original data set to obtain the copy data. The copy data is added to the original data set to obtain the extended data set. A sequence number is added to the data in the extended data set to obtain a labeled data set, wherein the sequence number is the order in which the data in the labeled data set is selected, and the difference between the sequence numbers of any two copy data is greater than or equal to the fifth threshold. The order in which the data in the labeled data set is selected has the same meaning as the order in which the text is selected in step 3002, and therefore, the meaning of the order in which the data in the labeled data set is selected will not be repeated here.

[0238] The above-mentioned selected order refers to being selected as initial data, or being selected as candidate data to be spliced ​​with initial data. For example, the labeled data set includes labeled data a, labeled data b, and labeled data c, wherein the order in which labeled data a is selected is before the order in which labeled data b is selected, and the order in which labeled data b is selected is before the order in which labeled data c is selected. Then the construction device first selects labeled data a as initial data, and then selects labeled data b as candidate data. After splicing labeled data a and labeled data b to obtain spliced ​​data, and using the spliced ​​data as initial data, labeled data c is selected as candidate data, and it is determined whether to splice the initial data with the candidate data.

[0239] Considering that the contents of the duplicate data are the same, if two identical duplicate data are selected from the labeled data set and the training data is obtained by splicing the two identical duplicate data, the content in the training data is likely to be highly repetitive, thereby reducing the quality of the training data. Since the sequence number is the order in which the data in the labeled data set is selected, the difference in the sequence numbers of any two duplicate data is greater than or equal to the fifth threshold, so the interval between any two duplicate data can be increased. This can reduce the probability of obtaining training data by splicing two identical duplicate data, thereby reducing the repetitiveness of the content in the training data, thereby improving the quality of the training data.

[0240] In this embodiment, the original data set is a collection of longer data and shorter data, the difference between the number of shorter data and the number of longer data is greater than or equal to the fourth threshold, and the fourth threshold is greater than the third threshold. After acquiring the original data set, the construction device can reduce the difference between the number of longer data and the number of shorter data by copying the longer data in the original data set, thereby obtaining a labeled data set.

[0241] As an optional implementation, the target domain includes the domain involved in the content of the first text and the domain involved in the content of the second text, wherein the target domain includes two or more different domains. In this implementation, the target domain includes two or more different domains, which can increase the diversity of the domain involved in the content of the first text and the domain involved in the content of the second text, and further increase the diversity of the domain involved in the text in the annotated data.

[0242] For example, the original data set includes a first text a and a second text b, wherein the content of the first text a relates to the legal field, and the content of the second text b relates to the sports field.

[0243] Optionally, the first text is determined based on a preset category system, and the field involved in the first text matches the field in the preset category system. The preset category system can be found in the above description.

[0244] As an optional implementation, the text processing device performs the following steps during the process of executing step 303:

[0245] 7001. Sample from the target distribution to obtain a reference value.

[0246] In an embodiment of the present application, the target distribution is a distribution within a preset interval, and the target distribution obeys a uniform distribution within the preset interval. For example, if the preset interval is an interval from 0 to 1, then the target distribution is a distribution within 0 to 1, and the target distribution obeys a uniform distribution within the interval from 0 to 1. Since the target distribution is a distribution within a preset interval, sampling from the target distribution is sampling from the preset interval. And since the target distribution obeys a uniform distribution within the preset interval, when sampling from the target distribution, the probability of any value within the preset interval being sampled is the same. In step 401, the text processing device obtains a reference value by sampling from the target distribution.

[0247] 7002. Determine the proportion of numbers within the preset interval that are less than or equal to the reference value within the preset interval as the sampling probability.

[0248] Since the probability of any value in the preset interval being sampled is the same, the proportion of the numbers in the preset interval that are less than or equal to the reference value is the probability of sampling from the target distribution to obtain a value less than or equal to the reference value. In other words, the sampling probability is the probability of sampling from the target distribution to obtain a value less than or equal to the reference value.

[0249] 7003. When the sampling probability is greater than or equal to the target probability, determine not to splice the initial data with the candidate data.

[0250] 7004. When the sampling probability is less than the target probability, determine to splice the initial data with the candidate data.

[0251] Because the sampling probability is the proportion of the number of numbers less than or equal to the reference value in the preset interval, if the sampling probability is the target probability, the proportion of the number of numbers less than or equal to the reference value in the preset interval is the target probability. Therefore, if the sampling probability greater than or equal to the target probability is called a reference event, then the probability of the reference event occurring is: (1-target probability), and the probability of the reference event occurring is the target probability. For example, the target probability is 0.8, and the preset interval is an interval from 0 to 1. Then when the reference value sampled from the preset interval is 0.8, the sampling probability is 0.8, that is, the sampling probability is the target probability. When the reference value sampled from the preset interval is greater than or equal to 0.8, the sampling probability is greater than or equal to 0.8. The probability that the reference value sampled from the preset interval is greater than or equal to 0.8 is (1-0.8) = 0.2, so the probability of the reference event occurring is 0.2.

[0252] In step 7003, the construction device determines not to splice the initial data with the candidate data when the sampling probability is greater than or equal to the target probability, that is, the construction device determines not to splice the initial data with the candidate data when the reference event occurs. The probability of the reference event occurring is: (1-target probability), so the probability of not splicing the initial data with the candidate data is determined to be: (1-target probability). On the contrary, in step 7004, the construction device determines to splice the initial data with the candidate data when the sampling probability is less than the target probability, that is, the construction device determines to splice the initial data with the candidate data when the reference event does not occur. The probability that the reference event does not occur is the target probability, so the probability of splicing the initial data with the candidate data is determined to be the target probability. Because the splicing of the initial data with the candidate data is determined to be a random event, the probability of the random event occurring is the target probability.

[0253] In one possible implementation, the construction device determines a reference similarity between the content in the initial data and the content in the candidate data when the sampling probability is greater than or equal to the target probability, and determines to splice the initial data with the candidate data when the reference similarity is less than or equal to a sixth threshold.

[0254] In the embodiment of the present application, the sixth threshold is the basis for judging whether the reference similarity is large or small. Specifically, if the reference similarity is greater than the sixth threshold, it means that the target similarity is large. Conversely, if the reference similarity is less than or equal to the sixth threshold, it means that the reference similarity is small. A small reference similarity indicates that the similarity between the content in the initial data and the content in the candidate data is low, which means that the difference between the content in the initial data and the content in the candidate data is large. At this time, by splicing the initial data and the candidate data, the span of the content in the spliced ​​data can be made larger, so that the spliced ​​data can better simulate the topic switching in the real dialogue scene. For example, the text in the initial data is about travel, and the text in the candidate data is about food. By splicing the initial data and the candidate data, the content in the spliced ​​data can be switched from travel to food, thereby simulating the real scene in which the conversation topic is switched from travel to food.

[0255] In this embodiment, the construction device samples the reference value from the target distribution, wherein the target distribution is a distribution within a preset interval, and the target distribution obeys a uniform distribution within the preset interval. The proportion of numbers within the preset interval that are less than or equal to the reference value within the preset interval is determined as the sampling probability. When the sampling probability is greater than or equal to the target probability, it is determined not to splice the initial data with the candidate data. When the sampling probability is less than the target probability, it is determined to splice the initial data with the candidate data. In this way, the probability of splicing the initial data with the candidate data can be determined to be the target probability.

[0256] See also Figure 4 , Figure 4 A flow chart of another method for constructing a training data set provided in an embodiment of the present application. Figure 4As shown, the labeled data set includes longer data and shorter data, and the proportion of longer data in the labeled data set is smaller than that of shorter data in the labeled data set. In the first iteration, a data is first selected from the labeled data set as the initial data, and then it is determined whether P~U(0,1)<0.8 holds, where P~U(0,1) means that when the preset interval is 0 to 1, P follows a uniform distribution between 0 and 1. 0.8 is the above target probability, and P~U(0,1)<0.8 means that after sampling the reference value from the target distribution, the proportion of the number in the preset interval that is less than or equal to the reference value is determined as the sampling probability, and the sampling probability is less than 0.8. If the judgment result is no, that is, the sampling probability is greater than or equal to 0.8, then the initial data is added to the training data set as training data. If the judgment result is yes, that is, the sampling probability is less than 0.8, the initial data is spliced ​​with the candidate data. Spliced ​​data can be obtained by splicing. Then determine whether the concatenated text in the concatenated data meets the following conditions: the amount of concatenated text data is less than 32K, where 32K represents 32K tokens. If the judgment result is no, the concatenated data is used as the initial data, and the iterative execution continues to determine whether P~U(0,1)<0.8 is established. If the judgment result is yes, the concatenated data is added to the training data set as training data. Figure 4 The method for constructing the training data set shown can make the training data set include both target texts with a data volume greater than 32K and target texts with a data volume less than or equal to 32K, and can also enrich the diversity of the training data in the training data set.

[0257] As an optional implementation, after obtaining the training data set by the training data set construction method described above, the training data set can be used to train the language model to improve the processing effect of the language model on the text. Specifically, the first prompt word of the training data is used to guide the language model to perform the task based on the target text to obtain an execution result. Based on the difference between the execution result and the first label, the loss of the language model is obtained, wherein the loss is positively correlated with the difference. Based on the loss of the language model, the parameters of the language model are updated until the loss of the language model converges. It should be understood that the device for training the language model using the training data set and the construction device may be different or the same.

[0258] Optionally, the number of parameters of the language model is 7 billion (B), and the first threshold is 32K tokens, that is, the long text in the target text is a text with a data volume greater than or equal to 32K tokens. Optionally, the training cycle (epoch) of the language model is 1 to 3. Optionally, the training of the language model is completed by supervised fine-tuning (SFT) training.

[0259] Optionally, after completing the training of the language model, the language model is tested. Specifically, short data and long data are mixed in different mixing methods to obtain different training data sets. Optionally, the long data is an open source data set: data in longbench. Then, the language model is trained using each training data set, and then the language model trained based on different training data sets is tested to evaluate the training effect of different training data sets. The test results are shown in Table 1 below.

[0260] As shown in Table 1, the test results include short data evaluation scores and long data evaluation scores, where the short data evaluation score is the result obtained by evaluating the trained language model using short data, and the long data evaluation score is the result obtained by evaluating the trained language model using long data. The long data evaluation score includes the average score, single text answer score, multi-text answer score, extraction summary score, and few-sample task score. The single text answer score is the evaluation result of the trained language model performing single text answer based on long data, the long text answer score is the evaluation result of the trained language model performing multi-text answer based on long data, the extraction summary score is the evaluation result of the trained language model performing extraction summary based on long data, and the few-sample task is the evaluation result of the trained language model performing few-sample tasks based on long data. The average score is the average of the single text question and answer score, the multi-text question and answer score, the extraction summary score, and the few-sample task score.

[0261] Single-text answer refers to answering questions based on one text in the test data. Multi-text answer refers to answering questions based on two or more texts in the test data. Extraction summary refers to extracting summaries of texts in the test data. Few-sample task refers to the number of examples in the prompt word of the test data is less than or equal to the seventh threshold, and optionally, the seventh threshold is 2.

[0262]

[0263]

[0264] Table 1

[0265] If the hybrid methods in Table 1 are called the first hybrid method, the second hybrid method, the third hybrid method, the fourth hybrid method, the fifth hybrid method, the sixth hybrid method, and the seventh hybrid method in order from top to bottom. It can be seen from Table 1 that when the language model is trained using the training data set obtained by the seventh hybrid method, the short data evaluation score, average score, and few-sample task score of the language model are the highest, and the single-text question-answering score, multi-text question-answering score, and extraction summary score are also high. When the language model is trained using the training data set obtained by the sixth hybrid method, the short data evaluation score of the language model is the second highest, and the average score is also high. Since the sixth and seventh hybrid methods are both obtained according to the method for constructing the training data set, the experimental data in Table 1 show that when the training data is obtained according to the method for constructing the training data set, the short data evaluation score of the language model can be improved by using the training data to train the language model.

[0266] Based on the test results in Table 1, the following conclusions can be drawn: (1) When the training data set is obtained by concatenating short data, the language model can be trained using the training data set to improve the long data evaluation score of the language model; (2) A small amount of long data labels can improve the score of the few-sample task; (3) When the training data set is constructed by concatenating short data with a data volume of 4K tokens to obtain training data and training 3 epcohs, the short data evaluation score, average score, and few-sample task score are the highest, and the single-text question-answering score, multi-text question-answering score, and extraction summary score are also high. This shows that the training data and training method can improve both the long data evaluation score and the short data evaluation score.

[0267] The embodiment of the present application also provides a dialogue method, the execution subject of the dialogue method is a dialogue device, wherein the dialogue device can be any electronic device that can execute the technical solution disclosed in the embodiment of the method of the present application. Optionally, the dialogue device can be one of the following: a computer, a server.

[0268] See also Figure 5 , Figure 5 A flowchart of a dialogue method provided in an embodiment of the present application.

[0269] 501. Obtain target questions and reference texts.

[0270] In the embodiment of the present application, the target question is a question expressed in natural language, and the target question can be any question. The reference text can be any text, and the data volume of the reference text can be less than the first threshold, or the data volume of the reference text can be greater than or equal to the first threshold, that is, the reference text can be a short text or a long text.

[0271] 502. Input the reference text into a target language model, so that the target language model can answer the target question based on the target text and obtain a target answer.

[0272] In one possible implementation, a target language model is obtained by training using a text set, wherein the target text in the text set is obtained according to the text processing method described above. Optionally, the training text can be obtained by annotating the target text. The language model is used to answer questions based on the training text to obtain a first answer. Based on the difference between the first answer and the label of the training text, the loss of the language model is obtained, wherein the loss is positively correlated with the difference. Based on the loss of the language model, the parameters of the language model are updated until the loss of the language model converges to obtain the target language model. Since the text set includes both short texts and long texts, and the data volume of the texts in the text set is more diverse, the target language model obtained by training the text set can improve the processing effect of the target language model on texts of different data volumes. Specifically, when the target language model is used to answer questions based on texts of different data volumes, a higher accuracy can be obtained.

[0273] In another possible implementation, the target language model is obtained by training using a training data set, wherein the training data in the training data set is obtained according to the method for constructing a training data set described above. Optionally, the first prompt word of the training data is used to guide the language model to answer questions based on the target text in the training data to obtain a second answer. Based on the difference between the second answer and the first label of the training data, the loss of the language model is obtained, wherein the loss is positively correlated with the difference. Based on the loss of the language model, the parameters of the language model are updated until the loss of the language model converges to obtain the target language model. Since the training data set includes both short data and long data, and the data volume of the text in the training data is more diverse, the target language model obtained by training using the training data set can improve the processing effect of the target language model on texts of different data volumes. Specifically, when using the target language model to answer questions based on texts of different data volumes, a higher accuracy can be obtained.

[0274] In the embodiment of the present application, since the target language model has a good processing effect on text of any data amount, after obtaining the target question and the reference text, the dialogue device inputs the reference text into the target language model, so that the target language model answers the target question based on the target text and obtains the target answer, which can improve the accuracy of the target answer.

[0275] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0276] The method of the embodiment of the present application is described in detail above, and the device of the embodiment of the present application is provided below.

[0277] See also Figure 6 , Figure 6 The present invention provides a structural diagram of a text processing device according to an embodiment of the present invention. The text processing device 1 is used to construct a text set, which is a set of target texts. The text processing device 1 includes: an acquisition unit 11, a processing unit 12, a determination unit 13, and a splicing unit 14. Specifically:

[0278] An acquisition unit 11 is used to acquire a set of texts to be processed, wherein the set of texts to be processed includes two or more texts to be processed, and the data volume of the texts to be processed is less than a first threshold;

[0279] A processing unit 12 is used to select one of the to-be-processed texts from the to-be-processed text set as an initial text, and the to-be-processed texts in the to-be-processed text set that are different from the initial text are candidate texts;

[0280] A determination unit 13, configured to determine whether to splice the initial text with the candidate text when splicing the initial text with the candidate text is regarded as a random event and the probability of occurrence of the random event is a target probability;

[0281] The processing unit 12 is used for taking the initial text as the target text when it is determined that the initial text and the candidate text are not to be spliced ​​together;

[0282] The processing unit 12 is used for splicing the initial text with the candidate text to obtain a spliced ​​text when it is determined that the initial text and the candidate text are to be spliced;

[0283] The concatenation unit 14 is configured to use the concatenated text as the target text when the data volume of the concatenated text is greater than or equal to the first threshold.

[0284] In combination with any embodiment of the present application, the processing unit 12 is further configured to:

[0285] When the data volume of the concatenated text is smaller than the first threshold, the concatenated text is used as the initial text.

[0286] In combination with any implementation manner of the present application, the text to be processed includes a first text and a second text, the data volume of the first text is greater than or equal to a second threshold, the data volume of the second text is less than the second threshold, and the second threshold is less than the first threshold;

[0287] A difference between the quantity of the second text and the quantity of the first text is greater than or equal to a third threshold.

[0288] In combination with any embodiment of the present application, the acquisition unit 11 is specifically used for:

[0289] Acquire an original text set, where the original text set is a set of the first text and the second text, and the difference between the number of the second text and the number of the first text is greater than or equal to a fourth threshold, and the fourth threshold is greater than the third threshold;

[0290] The to-be-processed text set is obtained by copying the first text in the original text set.

[0291] In combination with any embodiment of the present application, the acquisition unit 11 is specifically used for:

[0292] Copying the first text in the original text set to obtain a copy text;

[0293] Adding the copy text to the original text set to obtain an expanded text set;

[0294] Add serial numbers to the texts in the expanded text set to obtain the to-be-processed text set, wherein the serial numbers are the order in which the texts in the to-be-processed text set are selected, and the difference between the serial numbers of any two of the copy texts is greater than or equal to a fifth threshold.

[0295] In combination with any implementation manner of the present application, the target field includes the field involved in the content of the first text and the field involved in the content of the second text, and the target field includes two or more different fields.

[0296] In combination with any implementation manner of the present application, the determining unit 13 is specifically configured to:

[0297] Sampling a reference value from a target distribution, wherein the target distribution is a distribution within a preset interval, and the target distribution obeys a uniform distribution within the preset interval;

[0298] Determine the proportion of numbers within the preset interval that are less than or equal to the reference value within the preset interval as a sampling probability;

[0299] When the sampling probability is greater than or equal to the target probability, determining not to splice the initial text with the candidate text;

[0300] When the sampling probability is less than the target probability, it is determined to concatenate the initial text with the candidate text.

[0301] In combination with any implementation manner of the present application, the determining unit 13 is specifically configured to:

[0302] When the sampling probability is greater than or equal to the target probability, determining a target similarity between the content in the initial text and the content in the candidate text;

[0303] When the target similarity is less than or equal to a sixth threshold, it is determined to concatenate the initial text with the candidate text.

[0304] In the embodiment of the present application, the set of texts to be processed includes two or more texts to be processed, wherein the data volume of the texts to be processed is less than the first threshold value, that is, the set of texts to be processed includes two or more short texts. After obtaining the set of texts to be processed, the text processing device selects a text to be processed from the set of texts to be processed as the initial text, and the text to be processed that is different from the initial text in the set of texts to be processed is the candidate text. In the case where the splicing of the initial text and the candidate text is regarded as a random event, and the probability of the occurrence of the random event is the target probability, it is determined whether to splice the initial text with the candidate text, so that it is possible to splice the initial text with the candidate text, and it is also possible not to splice the initial text with the candidate text. In the case of determining not to splice the initial text with the candidate text, the initial text is used as the target text, thereby making the target text a short text, thereby making the text set include the short text. In the case of determining to splice the initial text with the candidate text, the initial text and the candidate text are spliced ​​to obtain the spliced ​​text. In the case where the data volume of the spliced ​​text is greater than or equal to the first threshold value, the spliced ​​text is used as the target text, thereby making the target text a long text, thereby making the text set include the long text. In general, by obtaining the target text through the embodiments of the present application, the target text can be either a short text or a long text, so that when a text set is a collection of target texts, the text set can include both long texts and short texts.

[0305] See also Figure 7 , Figure 7A training data set construction device provided in an embodiment of the present application, the training data set construction device 2 is used to construct a training data set, the training data set is a collection of training data, the training data includes a first prompt word and a first label, the first prompt word is used to guide the language model to perform a task based on a target text, the first label includes the result of performing the task based on the target text, the training data set construction device 2 includes: an acquisition unit 21, a processing unit 22, a determination unit 23, and a splicing unit 24, specifically:

[0306] An acquisition unit 21 is used to acquire a labeled data set, wherein the labeled data set includes two or more labeled data, the labeled data includes a second prompt word and a second label, the second prompt word is used to guide the language model to perform a task based on the text to be processed, the second label includes a result of performing the task based on the text to be processed, and the data volume of the text to be processed is less than a first threshold;

[0307] A processing unit 22 is used to select one of the labeled data from the labeled data set as initial data, and the labeled data in the labeled data set that is different from the initial data is candidate data;

[0308] A determination unit 23, configured to determine whether to splice the initial data with the candidate data when splicing the initial data with the candidate data is regarded as a random event and the probability of occurrence of the random event is a target probability;

[0309] The processing unit 22 is configured to use the initial data as the training data when it is determined that the initial data is not to be spliced ​​with the candidate data;

[0310] a splicing unit 24, configured to, when determining to splice the initial data with the candidate data, splice the initial data with the candidate data to obtain spliced ​​data, the spliced ​​data comprising a third prompt word and a third label, the third prompt word being used to guide the language model to perform a task based on the spliced ​​text, the third label comprising a result of performing the task based on the spliced ​​text, the spliced ​​text being obtained by splicing the to-be-processed text of the initial data and the to-be-processed text of the candidate data, the third label of the spliced ​​data comprising the second label of the initial data and the third label of the candidate data;

[0311] The processing unit 22 is configured to, when the data volume of the concatenated text is greater than or equal to the first threshold, use the third prompt word as the first prompt word and the third label as the first label to obtain the training data.

[0312] In combination with any embodiment of the present application, the processing unit 22 is further configured to:

[0313] When the data volume of the concatenated text is smaller than the first threshold, the concatenated data is used as the initial data.

[0314] In combination with any implementation manner of the present application, the text to be processed includes a first text and a second text, the data volume of the first text is greater than or equal to a second threshold, the data volume of the second text is less than the second threshold, and the second threshold is less than the first threshold;

[0315] A difference between the amount of the labeled data including the second text and the amount of the labeled data including the first text is greater than or equal to a third threshold.

[0316] In combination with any embodiment of the present application, the acquisition unit 21 is specifically used for:

[0317] Acquire an original data set, the original data set being a collection of long data and short data; the long data comprising a fourth prompt word and a fourth label, the fourth prompt word being used to guide the language model to perform a task based on the first text, and the fourth label comprising a result of performing the task based on the first text; the short data comprising a fifth prompt word and a fifth label, the fifth prompt word being used to guide the language model to perform a task based on the second text, and the fifth label comprising a result of performing the task based on the second text; the difference between the amount of the short data and the amount of the long data is greater than or equal to a fourth threshold, and the fourth threshold is greater than the third threshold;

[0318] The labeled data set is obtained by copying the long data in the original data set.

[0319] In combination with any embodiment of the present application, the acquisition unit 21 is specifically used for:

[0320] Copying the long data in the original data set to obtain copy data;

[0321] Adding the copy data to the original data set to obtain an expanded data set;

[0322] Add serial numbers to the data in the expanded data set to obtain the labeled data set, where the serial numbers are the order in which the data in the labeled data set are selected, and the difference between the serial numbers of any two of the copy data is greater than or equal to a fifth threshold.

[0323] In combination with any implementation manner of the present application, the target field includes the field involved in the content of the first text and the field involved in the content of the second text, and the target field includes two or more different fields.

[0324] In combination with any implementation manner of the present application, the determining unit 23 is specifically configured to:

[0325] Sampling a reference value from a target distribution, wherein the target distribution obeys a uniform distribution within a preset interval;

[0326] Determine the proportion of numbers within the preset interval that are less than or equal to the reference value within the preset interval as a sampling probability;

[0327] When the sampling probability is less than the target probability, determining not to splice the initial data with the candidate data;

[0328] When the sampling probability is greater than or equal to the target probability, it is determined to splice the initial data with the candidate data.

[0329] In combination with any implementation manner of the present application, the determining unit 23 is specifically configured to:

[0330] In a case where the sampling probability is greater than or equal to the target probability, determining a reference similarity between the content in the initial data and the content in the candidate data;

[0331] When the reference similarity is less than or equal to a sixth threshold, it is determined to splice the initial data with the candidate data.

[0332] In an embodiment of the present application, the labeled data set includes two or more labeled data, and the labeled data includes a second prompt word and a second label, wherein the second prompt word is used to guide the language model to perform a task based on the text to be processed, and the second label includes the result of performing the task based on the text to be processed, and the data volume of the text to be processed is less than the first threshold value, that is, the labeled data is short data, and the labeled data set includes two or more short data. After acquiring the labeled data set, the construction device selects a labeled data from the labeled data set as the initial data, and the labeled data different from the initial data in the labeled data set is the candidate data. In the case of splicing the initial data with the candidate data as a random event, and the probability of the random event occurring is the target probability, it is determined whether to splice the initial data with the candidate data, so that it is possible to splice the initial data with the candidate data, and it is also possible not to splice the initial data with the candidate data. In the case of determining not to splice the initial data with the candidate data, the initial data is used as training data, thereby making the training data short data, and then making the training data set include short data. In the case of determining to splice the initial data with the candidate data, the initial data and the candidate data are spliced ​​to obtain spliced ​​data. When it is determined that the initial data and the candidate data are to be spliced, the initial data and the candidate data are spliced ​​to obtain spliced ​​data, and when the amount of the spliced ​​text in the spliced ​​data is greater than or equal to the first threshold, the spliced ​​data is used as training data, thereby making the training data long data, and further making the training data set include long data. In general, by obtaining training data through the embodiments of the present application, the training data can be short data or long data, and thus when the training data set is a set of training data, the training data set can include both long data and short data.

[0333] See also Figure 8 , Figure 8 A dialogue device is provided in an embodiment of the present application. The dialogue device 3 includes: an acquisition unit 31 and an answering unit 32. Specifically:

[0334] An acquisition unit 31, used for acquiring a target question and a reference text;

[0335] The answering unit 32 is used to input the reference text into the target language model so that the target language model answers the target question based on the reference text and obtains the target answer; the target language model is trained using a text set, and the target text in the text set is obtained according to the text processing method described above; the target language model is also trained using a training data set, and the training data in the training data set is obtained according to the training data set construction method described above.

[0336] In the embodiment of the present application, since the target language model has a good processing effect on text of any data amount, after obtaining the target question and the reference text, the dialogue device inputs the reference text into the target language model, so that the target language model answers the target question based on the target text and obtains the target answer, which can improve the accuracy of the target answer.

[0337] In some embodiments, the functions or modules included in the device provided in the embodiments of the present application can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0338] Fig. 9 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. The electronic device 4 includes a processor 41 and a memory 42. Optionally, the electronic device 4 also includes an input device 43 and an output device 44. The processor 41, the memory 42, the input device 43 and the output device 44 are coupled via a connector, and the connector includes various interfaces, transmission lines or buses, etc., which are not limited in the embodiments of the present application. It should be understood that in each embodiment of the present application, coupling refers to mutual connection in a specific manner, including direct connection or indirect connection through other devices, for example, it can be connected through various interfaces, transmission lines, buses, etc.

[0339] The processor 41 may include one or more processors, for example, one or more central processing units (CPUs). When the processor is a CPU, the CPU may be a single-core CPU or a multi-core CPU. Optionally, the processor 41 may be a processor group consisting of multiple CPUs, and the multiple processors are coupled to each other through one or more buses. Optionally, the processor may also be other types of processors, etc., which are not limited in the embodiments of the present application.

[0340] The memory 42 can be used to store computer program instructions and various computer program codes including the program code for executing the program code of the present application. Optionally, the memory includes but is not limited to random access memory (RAM), read-only memory (ROM), erasable programmable read only memory (EPROM), or portable read only memory (CD-ROM), which is used for related instructions and data.

[0341] The input device 43 is used to input data and / or signals, and the output device 44 is used to output data and / or signals. The input device 43 and the output device 44 can be independent devices or an integrated device.

[0342] It can be understood that in the embodiment of the present application, the memory 42 can be used not only to store relevant instructions, but also to store relevant data. For example, the memory 42 can be used to store a set of texts to be processed obtained through the input device 43, or the memory 42 can also be used to store a target text obtained through the processor 41, etc. The embodiment of the present application does not limit the specific data stored in the memory.

[0343] Understandably, Fig. 9 Only a simplified design of an electronic device is shown. In practical applications, the electronic device may also include other necessary components, including but not limited to any number of input / output devices, processors, memories, etc., and all electronic devices that can implement the embodiments of the present application are within the protection scope of the present application.

[0344] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0345] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. Those skilled in the art can also clearly understand that the descriptions of the various embodiments of the present application have different focuses. For the convenience and brevity of description, the same or similar parts may not be repeated in different embodiments. Therefore, for parts not described or not described in detail in a certain embodiment, refer to the records of other embodiments.

[0346] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0347] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0348] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0349] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital versatile disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0350] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by a computer program to instruct the relevant hardware, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above-mentioned method embodiments. The aforementioned storage medium includes: a read-only memory (ROM) or a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

Claims

1. A text processing method, characterized in that: The method is used to construct a text set, where the text set is a set of target texts, and the method comprises: Acquire a text set to be processed, wherein the text set to be processed includes two or more texts to be processed, and the data volume of the texts to be processed is less than a first threshold; Selecting one of the to-be-processed texts from the to-be-processed text set as an initial text, and the to-be-processed texts in the to-be-processed text set that are different from the initial text are candidate texts; In the case where splicing the initial text with the candidate text is regarded as a random event and the probability of occurrence of the random event is a target probability, determining whether to splice the initial text with the candidate text; In the case where it is determined not to splice the initial text with the candidate text, taking the initial text as the target text; In the case where it is determined to splice the initial text with the candidate text, splicing the initial text with the candidate text to obtain a spliced ​​text; When the data volume of the concatenated text is greater than or equal to the first threshold, the concatenated text is used as the target text.

2. The method according to claim 1, characterized in that After obtaining the concatenated text, the method further includes: When the data volume of the concatenated text is smaller than the first threshold, the concatenated text is used as the initial text.

3. The method according to claim 1 or 2, characterized in that: The text to be processed includes a first text and a second text, the data volume of the first text is greater than or equal to a second threshold, the data volume of the second text is less than the second threshold, and the second threshold is less than the first threshold; A difference between the quantity of the second text and the quantity of the first text is greater than or equal to a third threshold.

4. The method according to claim 3, characterized in that The step of obtaining the text set to be processed includes: Acquire an original text set, where the original text set is a set of the first text and the second text, and the difference between the number of the second text and the number of the first text is greater than or equal to a fourth threshold, and the fourth threshold is greater than the third threshold; The to-be-processed text set is obtained by copying the first text in the original text set.

5. The method according to claim 4, characterized in that The step of obtaining the to-be-processed text set by copying the long text in the original text set includes: Copying the first text in the original text set to obtain a copy text; Adding the copy text to the original text set to obtain an expanded text set; Add serial numbers to the texts in the expanded text set to obtain the to-be-processed text set, wherein the serial numbers are the order in which the texts in the to-be-processed text set are selected, and the difference between the serial numbers of any two of the copy texts is greater than or equal to a fifth threshold.

6. The method according to claim 4, characterized in that The target field includes the field involved in the content of the first text and the field involved in the content of the second text, and the target field includes two or more different fields.

7. The method according to claim 1 or 2, characterized in that: The step of determining whether to splice the initial text with the candidate text, when splicing the initial text with the candidate text is regarded as a random event and the probability of occurrence of the random event is a target probability, comprises: Sampling a reference value from a target distribution, wherein the target distribution is a distribution within a preset interval, and the target distribution obeys a uniform distribution within the preset interval; Determine the proportion of numbers within the preset interval that are less than or equal to the reference value within the preset interval as a sampling probability; When the sampling probability is greater than or equal to the target probability, determining not to splice the initial text with the candidate text; When the sampling probability is less than the target probability, it is determined to concatenate the initial text with the candidate text.

8. The method according to claim 7, characterized in that The step of determining to splice the initial text with the candidate text when the sampling probability is greater than or equal to the target probability includes: When the sampling probability is greater than or equal to the target probability, determining a target similarity between the content in the initial text and the content in the candidate text; When the target similarity is less than or equal to a sixth threshold, it is determined to concatenate the initial text with the candidate text.

9. A method for constructing a training data set, characterized in that: The method is used to construct a training data set, wherein the training data set is a collection of training data, wherein the training data includes a first prompt word and a first label, wherein the first prompt word is used to guide a language model to perform a task based on a target text, and the first label includes a result of performing the task based on the target text, and the method includes: Acquire a labeled data set, the labeled data set including two or more labeled data, the labeled data including a second prompt word and a second label, the second prompt word is used to guide the language model to perform a task based on the text to be processed, the second label includes a result of performing the task based on the text to be processed, and the data volume of the text to be processed is less than a first threshold; Selecting one of the labeled data from the labeled data set as initial data, and the labeled data in the labeled data set that is different from the initial data is candidate data; In a case where splicing the initial data with the candidate data is regarded as a random event and the probability of occurrence of the random event is a target probability, determining whether to splice the initial data with the candidate data; In a case where it is determined not to splice the initial data with the candidate data, using the initial data as the training data; In the case where it is determined to splice the initial data with the candidate data, the initial data and the candidate data are spliced ​​to obtain spliced ​​data, the spliced ​​data includes a third prompt word and a third label, the third prompt word is used to guide the language model to perform a task based on the spliced ​​text, the third label includes a result of performing the task based on the spliced ​​text, the spliced ​​text is obtained by splicing the to-be-processed text of the initial data and the to-be-processed text of the candidate data, and the third label of the spliced ​​data includes the second label of the initial data and the third label of the candidate data; When the data volume of the concatenated text is greater than or equal to the first threshold, the third prompt word is used as the first prompt word, and the third label is used as the first label to obtain the training data.

10. The method according to claim 9, characterized in that After obtaining the splicing data, the method further includes: When the data volume of the concatenated text is smaller than the first threshold, the concatenated data is used as the initial data.

11. The method according to claim 9 or 10, characterized in that: The text to be processed includes a first text and a second text, the data volume of the first text is greater than or equal to a second threshold, the data volume of the second text is less than the second threshold, and the second threshold is less than the first threshold; A difference between the amount of the labeled data including the second text and the amount of the labeled data including the first text is greater than or equal to a third threshold.

12. The method according to claim 11, characterized in that The step of obtaining the labeled data set includes: Acquire an original data set, the original data set being a collection of long data and short data; the long data comprising a fourth prompt word and a fourth label, the fourth prompt word being used to guide the language model to perform a task based on the first text, and the fourth label comprising a result of performing the task based on the first text; the short data comprising a fifth prompt word and a fifth label, the fifth prompt word being used to guide the language model to perform a task based on the second text, and the fifth label comprising a result of performing the task based on the second text; the difference between the amount of the short data and the amount of the long data is greater than or equal to a fourth threshold, and the fourth threshold is greater than the third threshold; The labeled data set is obtained by copying the long data in the original data set.

13. The method according to claim 9 or 10, characterized in that: The step of determining whether to splice the initial data with the candidate data when splicing the initial data with the candidate data is regarded as a random event and the probability of occurrence of the random event is a target probability includes: Sampling a reference value from a target distribution, wherein the target distribution obeys a uniform distribution within a preset interval; Determine the proportion of numbers within the preset interval that are less than or equal to the reference value within the preset interval as a sampling probability; When the sampling probability is less than the target probability, determining not to splice the initial data with the candidate data; When the sampling probability is greater than or equal to the target probability, it is determined to splice the initial data with the candidate data.

14. A conversation method, characterized in that: The method comprises: Obtain target questions and reference texts; The reference text is input into a target language model so that the target language model answers the target question based on the reference text and obtains a target answer; the target language model is trained using a text set, and the target text in the text set is obtained according to the method described in any one of claims 1 to 8; or the target language model is trained using a training data set, and the training data in the training data set is obtained according to the method described in any one of claims 9 to 13.

15. A text processing device, characterized in that: The text processing device is used to construct a text set, wherein the text set is a collection of target texts, and the text processing device comprises: An acquisition unit, used for acquiring a text set to be processed, wherein the text set to be processed includes two or more texts to be processed, and the data volume of the texts to be processed is less than a first threshold; A processing unit, configured to select one of the to-be-processed texts from the to-be-processed text set as an initial text, and the to-be-processed texts in the to-be-processed text set that are different from the initial text are candidate texts; A determination unit, configured to determine whether to splice the initial text with the candidate text when splicing the initial text with the candidate text is regarded as a random event and the probability of occurrence of the random event is a target probability; The processing unit is configured to use the initial text as the target text when it is determined that the initial text and the candidate text are not to be spliced ​​together; The splicing unit is used for splicing the initial text with the candidate text to obtain a spliced ​​text when it is determined that the initial text and the candidate text are to be spliced; The processing unit is configured to use the concatenated text as the target text when the data volume of the concatenated text is greater than or equal to the first threshold.

16. A device for constructing a training data set, characterized in that: The training data set construction device is used to construct a training data set, the training data set is a collection of training data, the training data includes a first prompt word and a first label, the first prompt word is used to guide the language model to perform a task based on a target text, the first label includes a result of performing the task based on the target text, and the training data set construction device includes: an acquisition unit, configured to acquire a labeled data set, wherein the labeled data set includes two or more labeled data, the labeled data includes a second prompt word and a second label, the second prompt word is used to guide the language model to perform a task based on the text to be processed, the second label includes a result of performing the task based on the text to be processed, and the data volume of the text to be processed is less than a first threshold; A processing unit, configured to select one of the labeled data from the labeled data set as initial data, wherein the labeled data in the labeled data set that is different from the initial data is candidate data; a determination unit, configured to determine whether to splice the initial data with the candidate data when splicing the initial data with the candidate data is regarded as a random event and the probability of occurrence of the random event is a target probability; The processing unit is configured to use the initial data as the training data when it is determined that the initial data is not to be spliced ​​with the candidate data; a splicing unit, configured to, when determining to splice the initial data with the candidate data, splice the initial data with the candidate data to obtain spliced ​​data, the spliced ​​data comprising a third prompt word and a third label, the third prompt word being used to guide the language model to perform a task based on the spliced ​​text, the third label comprising a result of performing the task based on the spliced ​​text, the spliced ​​text being obtained by splicing the to-be-processed text of the initial data and the to-be-processed text of the candidate data, the third label of the spliced ​​data comprising the second label of the initial data and the third label of the candidate data; A processing unit is used to use the third prompt word as the first prompt word and the third label as the first label to obtain the training data when the data amount of the concatenated text is greater than or equal to the first threshold.

17. A conversation device, characterized in that: The dialogue device comprises: An acquisition unit, used to acquire target questions and reference texts; An answering unit is used to input the reference text into a target language model so that the target language model answers the target question based on the reference text and obtains a target answer; the target language model is trained using a text set, and the target text in the text set is obtained according to the method described in any one of claims 1 to 8; or the target language model is trained using a training data set, and the training data in the training data set is obtained according to the method described in any one of claims 9 to 13.

18. An electronic device, characterized in that: include: A processor and a memory, wherein the memory is used to store computer program codes, wherein the computer program codes include computer instructions. When the processor executes the computer instructions, the electronic device executes the method as claimed in any one of claims 1 to 8, or the electronic device executes the method as claimed in any one of claims 9 to 13.

19. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which includes program instructions. When the program instructions are executed by a processor, the processor is caused to execute the method described in any one of claims 1 to 8, or the processor is caused to execute the method described in any one of claims 9 to 13.

20. A computer program product, characterized in that The computer program product includes a computer program or instructions; when the computer program or instructions are run on a computer, the computer is enabled to execute the method described in any one of claims 1 to 8, or the computer is enabled to execute the method described in any one of claims 9 to 13.

Images