Text splicing method, text processing method and related products

Abstract

The invention discloses a text splicing method, a text processing method and a related product. The method comprises the steps that a to-be-processed text set is obtained, the to-be-processed text set comprises two or more to-be-processed texts, and the data size of the to-be-processed texts is smaller than a first threshold value; keywords of the two or more texts to be processed are extracted, two or more target keywords are obtained, and one text to be processed is provided with one or more target keywords; determining the to-be-processed texts with the same target keyword from the to-be-processed text set to obtain two or more to-be-spliced texts; and splicing the two or more texts to be spliced to obtain a synthesized text, the data volume of the synthesized text being greater than or equal to the first threshold value.

Description

Technical Field

[0001] The present application relates to the technical field of natural language processing, and in particular to a text splicing method, a text processing method and related products. Background Art

[0002] With the development of natural language processing technology, more and more scenarios involve using natural language processing technology to process text. Among them, the demand for using natural language processing technology to process large amounts of text is increasing. In order to better study the processing of large amounts of text, it is necessary to obtain large amounts of text. Therefore, how to obtain large amounts of text is of great significance. Summary of the invention

[0003] The present application provides a text splicing method, a text processing method and related products to obtain a synthetic text with a large amount of data, wherein the related products include a text splicing device, a text processing device, an electronic device, a computer-readable storage medium and a computer program product.

[0004] In a first aspect, a text splicing method is provided, the method comprising:

[0005] Acquire a text set to be processed, wherein the text set to be processed includes two or more texts to be processed, and the data volume of the texts to be processed is less than a first threshold;

[0006] Extracting keywords from the two or more texts to be processed to obtain two or more target keywords, wherein one of the texts to be processed has one or more of the target keywords;

[0007] Determine the texts to be processed having the same target keyword from the set of texts to be processed, and obtain two or more texts to be spliced;

[0008] The two or more texts to be spliced ​​are spliced ​​to obtain a composite text, and the data volume of the composite text is greater than or equal to the first threshold.

[0009] In combination with any embodiment of the present application, the two or more texts to be processed include a reference text, and the target keyword of the reference text includes a reference keyword of the reference text;

[0010] The step of extracting keywords from the two or more texts to be processed to obtain two or more target keywords includes:

[0011] Determine a reference query statement of the reference text, wherein the reference query statement is a basis for querying to obtain the reference text;

[0012] The reference keywords are obtained by extracting keywords from the reference query statement.

[0013] In combination with any implementation manner of the present application, the step of extracting keywords from the reference query statement to obtain the reference keywords includes:

[0014] Extracting keywords from the reference query statement to obtain one or more query keywords;

[0015] Selecting one keyword from the one or more query keywords as the reference keyword;

[0016] Alternatively, determining the correlation between the one or more query keywords and the reference query statement to obtain one or more first correlations;

[0017] The query keyword corresponding to the maximum value among the one or more first correlations is determined as the reference keyword.

[0018] In combination with any implementation manner of the present application, the step of extracting keywords of the reference query statement to obtain one or more query keywords includes:

[0019] Extracting keywords from the reference query to obtain one or more candidate keywords;

[0020] determining invalid keywords among the one or more candidate keywords;

[0021] The invalid keywords in the one or more candidate keywords are removed to obtain the one or more query keywords.

[0022] In combination with any implementation manner of the present application, the step of determining an invalid keyword among the one or more candidate keywords includes:

[0023] Determine that the words in the reference query sentence whose number of appearances is greater than or equal to a second threshold are high-frequency words;

[0024] The high-frequency words in the one or more candidate keywords are determined to be the invalid keywords.

[0025] In combination with any implementation manner of the present application, the step of determining a reference query statement of the reference text includes:

[0026] Obtaining a query statement generation model, wherein the query statement generation model is used to generate a text query statement based on the text, and the maximum data volume of the text that can be processed by the query statement generation model is a data volume threshold;

[0027] In the case where the data volume of the reference text exceeds the data volume threshold, dividing the reference text into two or more segmented texts, where the data volume of the segmented texts is less than or equal to the data volume threshold;

[0028] Inputting the two or more segmented texts into the query statement generation model respectively, so that the query statement generation model generates query statements for the reference texts, thereby obtaining two or more segmented query statements;

[0029] The reference query statement is obtained based on the two or more segmented query statements.

[0030] In combination with any embodiment of the present application, the step of determining the texts to be processed having the same target keyword from the set of texts to be processed to obtain two or more texts to be spliced ​​includes:

[0031] Determining the texts to be processed having the same target keyword from the set of texts to be processed as two or more candidate texts;

[0032] Selecting one of the two or more candidate texts as a seed text, and the text other than the seed text among the two or more candidate texts as the text to be confirmed;

[0033] Determining the similarity between the seed text and the text to be confirmed to obtain a first similarity;

[0034] Determine the first similarity that is less than or equal to a third threshold as a second similarity;

[0035] Based on the text to be confirmed and the seed text corresponding to the second similarity, the two or more texts to be spliced ​​are obtained.

[0036] In combination with any embodiment of the present application, the step of splicing the two or more texts to be spliced ​​to obtain a composite text includes:

[0037] Splicing the two or more texts to be spliced ​​to obtain a spliced ​​text;

[0038] Determining a synthesis probability of the concatenated text based on the content of the concatenated text, wherein the synthesis probability is a probability that the concatenated text is a text obtained by synthesis;

[0039] When the synthesis probability is less than or equal to a fourth threshold, the concatenated text is determined to be a synthesized text.

[0040] In a second aspect, a text processing method is provided, the method comprising:

[0041] Acquire a target task and a target text, wherein the target task is a natural language processing task, and the data volume of the target text is greater than or equal to a first threshold;

[0042] The target text is input into a target language model so that the target language model executes the target task based on the target text to obtain an execution result. The target language model is trained using training data, and the training data includes synthetic text. The synthetic text is obtained according to the first aspect and any one of its embodiments.

[0043] In combination with any implementation manner of the present application, the training data also includes non-synthetic text, and the data volume of the non-synthetic text is greater than or equal to a first threshold.

[0044] In a third aspect, a text splicing device is provided, wherein the text processing device comprises:

[0045] An acquisition unit, used for acquiring a text set to be processed, wherein the text set to be processed includes two or more texts to be processed, and the data volume of the texts to be processed is less than a first threshold;

[0046] An extraction unit, used for extracting keywords from the two or more texts to be processed to obtain two or more target keywords, wherein one of the texts to be processed has one or more target keywords;

[0047] A determination unit, used to determine the to-be-processed texts having the same target keyword from the to-be-processed text set, to obtain two or more to-be-joined texts;

[0048] The splicing unit is used to splice the two or more texts to be spliced ​​to obtain a composite text, and the data volume of the composite text is greater than or equal to the first threshold.

[0049] In combination with any embodiment of the present application, the two or more texts to be processed include a reference text, and the target keyword of the reference text includes a reference keyword of the reference text;

[0050] The extraction unit is specifically used for:

[0051] Determine a reference query statement of the reference text, wherein the reference query statement is a basis for querying to obtain the reference text;

[0052] The reference keywords are obtained by extracting keywords from the reference query statement.

[0053] In combination with any embodiment of the present application, the extraction unit is specifically used for:

[0054] Extracting keywords from the reference query statement to obtain one or more query keywords;

[0055] Selecting one keyword from the one or more query keywords as the reference keyword;

[0056] Alternatively, determining the correlation between the one or more query keywords and the reference query statement to obtain one or more first correlations;

[0057] The query keyword corresponding to the maximum value among the one or more first correlations is determined as the reference keyword.

[0058] In combination with any embodiment of the present application, the extraction unit is specifically used for:

[0059] Extracting keywords from the reference query to obtain one or more candidate keywords;

[0060] determining invalid keywords among the one or more candidate keywords;

[0061] The invalid keywords in the one or more candidate keywords are removed to obtain the one or more query keywords.

[0062] In combination with any embodiment of the present application, the extraction unit is specifically used for:

[0063] Determine that the words in the reference query sentence whose number of appearances is greater than or equal to a second threshold are high-frequency words;

[0064] The high-frequency words in the one or more candidate keywords are determined to be the invalid keywords.

[0065] In combination with any embodiment of the present application, the extraction unit is specifically used for:

[0066] Obtaining a query statement generation model, wherein the query statement generation model is used to generate a text query statement based on the text, and the maximum data volume of the text that can be processed by the query statement generation model is a data volume threshold;

[0067] In the case where the data volume of the reference text exceeds the data volume threshold, dividing the reference text into two or more segmented texts, where the data volume of the segmented texts is less than or equal to the data volume threshold;

[0068] Inputting the two or more segmented texts into the query statement generation model respectively, so that the query statement generation model generates query statements for the reference texts, thereby obtaining two or more segmented query statements;

[0069] The reference query statement is obtained based on the two or more segmented query statements.

[0070] In combination with any implementation manner of the present application, the determining unit is specifically configured to:

[0071] Determining the texts to be processed having the same target keyword from the set of texts to be processed as two or more candidate texts;

[0072] Selecting one of the two or more candidate texts as a seed text, and the text other than the seed text among the two or more candidate texts as the text to be confirmed;

[0073] Determining the similarity between the seed text and the text to be confirmed to obtain a first similarity;

[0074] Determine the first similarity that is less than or equal to a third threshold as a second similarity;

[0075] Based on the text to be confirmed and the seed text corresponding to the second similarity, the two or more texts to be spliced ​​are obtained.

[0076] In combination with any embodiment of the present application, the splicing unit is specifically used for:

[0077] Splicing the two or more texts to be spliced ​​to obtain a spliced ​​text;

[0078] Determining a synthesis probability of the concatenated text based on the content of the concatenated text, wherein the synthesis probability is a probability that the concatenated text is a text obtained by synthesis;

[0079] When the synthesis probability is less than or equal to a fourth threshold, the concatenated text is determined to be a synthesized text.

[0080] In a fourth aspect, a text processing device is provided, the text processing device comprising:

[0081] An acquisition unit, used to acquire a target task and a target text, wherein the target task is a natural language processing task, and the data volume of the target text is greater than or equal to a first threshold;

[0082] A processing unit is used to input the target text into a target language model so that the target language model executes the target task based on the target text to obtain an execution result, wherein the target language model is trained using a training text, wherein the training text includes a synthetic text, and the synthetic text is obtained according to the first aspect and any embodiment thereof.

[0083] In combination with any implementation manner of the present application, the training data also includes non-synthetic text, and the data volume of the non-synthetic text is greater than or equal to a first threshold.

[0084] In a fifth aspect, an electronic device is provided, comprising: a processor and a memory, the memory being used to store computer program code, the computer program code comprising computer instructions, and when the processor executes the computer instructions, the electronic device executes the first aspect and any embodiment thereof, or the electronic device executes the second aspect and any embodiment thereof.

[0085] In a sixth aspect, another electronic device is provided, comprising: a processor, a sending device, an input device, an output device and a memory, wherein the memory is used to store computer program code, and the computer program code includes computer instructions. When the processor executes the computer instructions, the electronic device executes the first aspect and any embodiment thereof, or the electronic device executes the second aspect and any embodiment thereof.

[0086] In the seventh aspect, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a processor, the processor is caused to execute the first aspect and any embodiment thereof, or the processor is caused to execute the second aspect and any embodiment thereof.

[0087] In an eighth aspect, a computer program product is provided, which includes a computer program or instructions, and when the computer program or instructions are run on a computer, the computer is caused to execute the above-mentioned first aspect and any embodiment thereof, or the computer is caused to execute the above-mentioned second aspect and any embodiment thereof.

[0088] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application.

[0089] In the present application, the set of texts to be processed includes two or more texts to be processed, and the data volume of the texts to be processed is less than a first threshold value. After acquiring the set of texts to be processed, the text splicing device extracts keywords of two or more texts to be processed to obtain two or more target keywords. The texts to be processed with the same target keywords are determined from the set of texts to be processed to obtain two or more texts to be spliced, thereby obtaining two or more texts to be spliced ​​with high semantic matching. The two or more texts to be spliced ​​are spliced ​​to obtain a composite text, wherein the data volume of the composite text is greater than or equal to the first threshold value, which can improve the authenticity of the composite text and make the content of the composite text more fluent. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the background technology, the drawings required for use in the embodiments of the present application or the background technology will be described below.

[0091] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0092] Figure 1 A flowchart of a text splicing method provided in an embodiment of the present application;

[0093] Figure 2 A flowchart of another text splicing method provided in an embodiment of the present application;

[0094] Figure 3 A flowchart of a text processing method provided in an embodiment of the present application;

[0095] Figure 4 A structural schematic diagram of a text splicing device provided in an embodiment of the present application;

[0096] Figure 5 A structural diagram of a text processing device provided in an embodiment of the present application;

[0097] Figure 6 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0098] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0099] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0100] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0101] The embodiment of the present application provides a text splicing method, the execution subject of the text splicing method is a text splicing device, wherein the text splicing device can be any electronic device that can execute the technical solution disclosed in the embodiment of the method of the present application. Optionally, the text splicing device can be one of the following: a computer, a server.

[0102] It should be understood that the method embodiment of the present application can also be implemented by a processor executing a computer program code. The following describes the embodiment of the present application in conjunction with the drawings in the embodiment of the present application. Figure 1 , Figure 1 A flowchart of a text splicing method provided in an embodiment of the present application.

[0103] 101. Obtain the text set to be processed.

[0104] In an embodiment of the present application, the set of texts to be processed includes two or more texts to be processed, wherein the data volume of the texts to be processed is less than a first threshold value. The first threshold value is a basis for measuring whether the data volume of a text is large or small. Specifically, if the data volume of a text is greater than or equal to the first threshold value, it means that the data volume of the text is large, and if the data volume of a text is less than the first threshold value, it means that the data volume of the text is small. For the convenience of expression, the text with a large data volume is referred to as a long text below, and the text with a small data volume is referred to as a short text. The texts to be processed in the set of texts to be processed are all short texts.

[0105] In an implementation of obtaining a set of text to be processed, the text splicing device receives the set of text to be processed input by a user through an input component. The input component includes at least one of the following: a keyboard, a mouse, a touch screen, a touch pad, and an audio input device.

[0106] In another implementation of obtaining the to-be-processed text set, the text splicing device receives the to-be-processed text set sent by a terminal. The terminal may be any of the following: a mobile phone, a computer, a tablet computer, or a server.

[0107] 102. Extract keywords from the two or more texts to be processed to obtain two or more target keywords.

[0108] In the embodiment of the present application, the target keyword is a keyword (key word) related to the semantics of the text to be processed. The target keyword can be used as a basis for querying to obtain the text to be processed. For example, the keywords of the text to be processed a include word b, then the text to be processed a can be queried based on word b. It should be understood that the keyword can be a word or a sentence. For example, the keyword is travel, and the keyword can also be a 3-day and 2-night tour.

[0109] In an embodiment of the present application, a text to be processed has one or more target keywords, so the text splicing device extracts keywords from two or more texts to be processed, and can extract one or more target keywords for each text to be processed, thereby obtaining two or more target keywords.

[0110] In one possible implementation, the text splicing device divides the text to be processed into one or more first sentences based on punctuation marks. For any one of the one or more first sentences, it is divided into one or more second sentences based on stop words, wherein the second sentences do not include stop words. Then, for each second sentence, one or more candidate keywords are obtained through division. The number of times each candidate keyword appears is determined. Based on the number, the probability that the candidate keyword is a target keyword is determined, wherein the probability is positively correlated with the number. Based on the probabilities of all candidate keywords in the text to be processed, the target keyword is determined, and optionally, the candidate keyword whose probability is greater than or equal to the probability threshold is determined as the target keyword.

[0111] In this implementation, the text to be processed is divided into one or more first sentences based on punctuation marks, which can improve the efficiency of division. Then, the short sentences are divided into one or more second sentences based on stop words, which can further divide the first sentence and remove the stop words in the first sentence, thereby reducing the interference of subsequent determination of target keywords. Then, one or more candidate keywords are obtained by dividing the second sentence, which is more conducive to the subsequent determination of target keywords based on candidate keywords. The probability of a candidate keyword being a target keyword is determined based on the number of times the candidate keyword appears, which can improve the accuracy of the probability. Finally, the target keyword is determined based on the probability of the candidate keyword, which can improve the accuracy of the target keyword.

[0112] In another possible implementation, the text splicing device extracts keywords from the text to be processed by using a rapid automatized keyword extraction (RAKE) function in a natural language processing toolkit (NLTK).

[0113] In another possible implementation, the text splicing device extracts a summary of the text to be processed. Extract keywords from the summary to obtain one or more summary keywords. Perform a query based on the summary keywords to obtain one or more query texts, and one or more query texts can be queried through a summary keyword. For one or more query texts obtained by querying with the same summary keyword, determine the similarity between each query text and the text to be processed to obtain one or more query similarities. Determine the average value of one or more query similarities to obtain a candidate similarity. When the candidate similarity is greater than or equal to the similarity threshold, determine the summary keyword corresponding to the candidate similarity as the target keyword.

[0114] For example, the summary keywords include summary keywords A and summary keywords B. The query text obtained by querying based on summary keyword A includes query text a and query text b. The query text obtained by querying based on summary keyword B includes query text c. The similarity between query text a and the text to be processed is determined to obtain query similarity d, the similarity between query text b and the text to be processed is determined to obtain query similarity e, and the similarity between query text c and the text to be processed is determined to obtain query similarity f. The average value of similarity d and similarity e is determined to obtain candidate similarity g. Query similarity f is used as candidate similarity h. When candidate similarity g is greater than or equal to the similarity threshold, summary keyword A is determined to be the target keyword. When candidate similarity h is greater than or equal to the similarity threshold, summary keyword B is determined to be the target keyword.

[0115] In this implementation, since the summary keyword is extracted from the summary of the text to be processed, the semantics of the summary keyword and the semantics of the text to be processed have a high degree of match, and it can be further determined whether the summary keyword is the target keyword of the text to be processed. In the case where the query text is obtained based on the summary keyword, the semantics of the query text and the semantics of the summary keyword have a high degree of match.

[0116] In the case of determining the candidate similarity based on the query similarity between the query text and the text to be processed, the matching degree between the semantics of the query text and the semantics of the text to be processed can be measured by the candidate similarity. Specifically, a large candidate similarity indicates that the semantics of the query text and the semantics of the text to be processed have a high matching degree. On the contrary, a small candidate similarity indicates that the semantics of the query text and the semantics of the text to be processed have a low matching degree. Based on the similarity threshold, determine whether the candidate similarity is large or small. Specifically, if the candidate similarity is greater than or equal to the similarity threshold, it indicates that the candidate similarity is large. On the contrary, if the candidate similarity is less than or equal to the similarity threshold, it indicates that the candidate similarity is small. In other words, if the candidate similarity is greater than or equal to the similarity threshold, it indicates that the semantics of the query text and the semantics of the text to be processed have a high matching degree. Because the semantics of the query text and the semantics of the abstract keyword have a high matching degree, the semantics of the text to be processed and the semantics of the abstract keyword have a high matching degree. At this time, taking the abstract keyword as the target keyword can improve the accuracy of the target keyword. Therefore, in the case where the candidate similarity is greater than or equal to the similarity threshold, the abstract keyword corresponding to the candidate similarity is determined as the target keyword.

[0117] 103. Determine the texts to be processed having the same target keyword from the set of texts to be processed, and obtain two or more texts to be spliced.

[0118] Since the semantics of the target keyword and the semantics of the text to be processed have a high matching degree, the semantics of the text to be processed with the same target keyword also have a high matching degree, that is, two or more texts to be spliced ​​are related texts. For example, the target keyword of the text to be processed a includes the target keyword A, and the target keyword of the text to be processed b also includes the target keyword A, then the text to be processed a and the text to be processed b both have the target keyword A, at this time, the semantics of the text to be processed a and the semantics of the text to be processed b have a high matching degree.

[0119] In the embodiment of the present application, the text to be spliced ​​is an object to be spliced ​​in subsequent processing. Since the number of texts to be processed with the same target keyword is two or more, the text splicing device can obtain two or more texts to be spliced ​​by determining the texts to be processed with the same target keyword from the set of texts to be processed, so that the semantic matching of the two or more texts to be spliced ​​is high.

[0120] In one possible implementation, the semantics of the text to be processed includes the subject described by the text to be processed. The subject described by the text to be processed with the same target keyword has a high matching degree. For example, the subject described by the text to be processed a is basketball shoes, and the subject described by the text to be processed b is basketball games. The target keywords of the text to be processed a include basketball, and the target keywords of the text to be processed b also include basketball. At this time, the text to be processed a and the text to be processed b both have the target keyword basketball. Obviously, the subject described by the text to be processed a and the subject described by the text to be processed b are both related to basketball, that is, the subject of the two has a high matching degree.

[0121] 104. Splice the two or more texts to be spliced ​​to obtain a composite text.

[0122] In the embodiment of the present application, the data volume of the synthesized text is greater than or equal to the first threshold, that is, the synthesized text is a long text. The text splicing device performs step 104 to splice two or more short texts to obtain a long text.

[0123] In one possible implementation, a text splicing device splices two or more texts to be spliced ​​to obtain a text to be optimized. A large language model (LLM) is used to generate a transition text for two adjacent texts to be spliced ​​in the text to be optimized, wherein the transition text is used to transition the content described by the text to be optimized from one text to be spliced ​​to another text to be spliced, that is, the transition text plays a connecting role. Optionally, the text splicing device generates a target prompt word (promt) based on two adjacent texts to be spliced ​​in the text to be optimized, wherein the target prompt word is used to guide the LLM to generate a transition text for two adjacent texts to be spliced. After generating the transition text, the text splicing device adds the transition text between two adjacent texts to be spliced ​​in the text to be optimized to obtain a composite text.

[0124] In this implementation, after the text splicing device splices two or more texts to be spliced ​​to obtain the text to be optimized, it uses LLM to generate transition text for two adjacent texts to be spliced ​​in the text to be optimized. Then, the transition text is added between the two adjacent texts to be spliced ​​in the text to be optimized to obtain a composite text, which can make the content of the composite text more fluent and more authentic.

[0125] In an embodiment of the present application, the set of texts to be processed includes two or more texts to be processed, and the data volume of the texts to be processed is less than a first threshold value. After acquiring the set of texts to be processed, the text splicing device extracts keywords of two or more texts to be processed to obtain two or more target keywords. The texts to be processed with the same target keywords are determined from the set of texts to be processed to obtain two or more texts to be spliced, thereby obtaining two or more texts to be spliced ​​with high semantic matching. The two or more texts to be spliced ​​are spliced ​​to obtain a composite text, wherein the data volume of the composite text is greater than or equal to the first threshold value, which can improve the authenticity of the composite text and make the content of the composite text more fluent.

[0126] As an optional implementation, the two or more to-be-processed texts include a reference text, and the target keyword of the reference text includes a reference keyword of the reference text. The reference text is any one of the two or more to-be-processed texts, and the reference keyword is any one of the target keywords of the reference text.

[0127] In this embodiment, the text splicing device performs the following steps during the execution of step 102:

[0128] 2001. Determine a reference query statement of the reference text.

[0129] In the embodiment of the present application, the reference query statement is a query statement (query) of the reference text. The reference query statement is the basis for querying to obtain the reference text, that is, the reference text can be queried by querying based on the reference query statement. It should be understood that both the reference query statement and the reference keyword can be used as the basis for querying to obtain the reference text, but the data volume of the reference query statement is larger than the data volume of the reference keyword. Optionally, the reference query statement includes a reference keyword.

[0130] In a possible implementation, a text splicing device obtains a query statement generation model, wherein the query statement generation model is used to generate a query statement of a text based on a text, and the maximum data volume of the text that the query statement generation model can process is a data volume threshold. Optionally, the query statement generation model is trained by the following training process: the text to be generated is input into the query statement generation model so that the query statement generation model outputs a candidate query statement of the text to be generated. Based on the difference between the candidate query statement and the real query statement of the text to be generated, the loss of the query statement generation model is determined, wherein the real query statement is the true value (ground truth, GT) of the query statement of the text to be generated, and the difference is positively correlated with the loss of the query statement generation model. Based on the loss of the query statement generation model, the parameters of the query statement generation model are updated until the loss of the query statement generation model converges, and the training of the query statement generation model is completed. Optionally, the query statement generation model is: stack exchange title body (stackexchange-title-body-t5-base-v1). The data volume threshold is the maximum data volume of the text that can be processed by the query statement generation model. For example, if the maximum data volume of the text that can be processed by the query statement generation model is 32K characters (tokens), then the data volume threshold is 32K tokens. When the data volume of the text input to the query statement generation model exceeds the data volume threshold, the low accuracy of the query statement means that the semantics of the query statement generated by the query statement generation model and the semantics of the text input to the query statement generation model have a low match.

[0131] Because the query statement generation model has the ability of the query statement of text generation based on text, therefore the query statement generation model can be utilized to generate the query statement of reference text.If the data amount of reference text exceeds the data amount threshold value, it means that the data amount of reference text exceeds the maximum data amount of the text that the query statement generation model can handle. At this moment, if the query statement generation model is utilized to generate the query statement of reference text, it is easy to cause the semantics of the generated query statement to be low in matching degree with the semantics of the reference text.Therefore, when the data amount of reference text exceeds the data amount threshold value, the reference text is divided into two or more segmentation texts, wherein, the data amount of the segmentation text is less than or equal to the data amount threshold value.For example, the maximum data amount that can be processed by stacking the exchange title body is 32K characters (token), wherein, K represents one thousand, and if the data amount of reference text is 50K tokens, the reference text can be divided into a segmentation text with a data amount of 32K tokens and a segmentation text with a data amount of 18K tokens. Then, the two or more segmented texts are respectively input into the query statement generation model, so that the query statement generation model generates query statements of each reference text, and two or more segmented query statements are obtained, which can improve the matching degree between the semantics of the segmented text and the semantics of the segmented query statements. Finally, based on the two or more segmented query statements, a reference query statement can be obtained. In one possible implementation method, one of the two or more segmented query statements is selected as the reference query statement. In another possible implementation method, the segmented query statement whose semantics best matches the semantics of the reference text is determined from the two or more segmented query statements as the reference query statement.

[0132] In this implementation, since the maximum data volume of the text that the query statement generation model can process is the data volume threshold, if the text with a data volume greater than the data volume threshold is input into the query statement generation model, it is easy to cause the semantics of the query statement generated by the query statement generation model to have a low matching degree with the semantics of the input text. Therefore, when the data volume of the reference text exceeds the data volume threshold, the text splicing device divides the reference text into two or more segmented texts. Then, the two or more segmented texts are respectively input into the query statement generation model so that the query statement generation model generates query statements of each reference text, and two or more segmented query statements are obtained, so that the matching degree of the semantics of the segmented text and the semantics of the segmented query statements can be improved. Finally, based on the two or more segmented query statements, a reference query statement of the reference text is obtained, which can improve the matching degree of the semantics of the reference text and the semantics of the reference query statement. Then, by extracting the keywords of the reference query statement to obtain the reference keywords of the reference text, the matching degree of the semantics of the reference text and the semantics of the reference keywords can be improved. Then, based on the reference keywords, two or more texts to be spliced, including the reference text, are determined from the text set to be processed, which can improve the semantic matching degree with the text to be spliced. In this way, by splicing two or more texts to be spliced ​​to obtain a synthetic text, the authenticity of the synthetic text can be improved, and the content of the synthetic text can be made more fluent.

[0133] In another possible implementation, the text splicing device obtains a reference query statement by encoding and decoding the reference text. In yet another possible implementation, the text splicing device generates an embedding of the reference text based on the reference text. The embedding of the reference text is then used to retrieve a query statement library to obtain a query statement having an embedding that matches the embedding of the reference text as the reference query statement, wherein the query statement library includes query statements and embeddings of the query statements.

[0134] 2002. Obtain the reference keywords by extracting keywords from the reference query sentence.

[0135] In this step, the method for extracting keywords is the same as the method for extracting keywords in step 102, and will not be repeated here.

[0136] In one possible implementation, the text splicing device extracts keywords of a reference query statement to obtain one or more query keywords. Determine the correlation between one or more query keywords and the reference query statement to obtain one or more first correlations. Determine the query keyword corresponding to the maximum value of the one or more first correlations as the reference keyword. For example, the one or more query keywords include query keyword a and query keyword b. Determine the correlation between query keyword a and the reference query statement to obtain the first correlation c, and determine the correlation between query keyword b and the reference query statement to obtain the first correlation d. If the first correlation c is greater than the first correlation d, then the first correlation c is the maximum value of the one or more first correlations. The query keyword corresponding to the first correlation c is query keyword a, so the reference keyword is query keyword a.

[0137] In this implementation, the first correlation can represent the degree of match between the semantics of the query keyword and the semantics of the query statement. After obtaining one or more first correlations, the text splicing device determines the query keyword corresponding to the maximum value of the one or more first correlations as the reference keyword, which can improve the degree of match between the semantics of the reference keyword and the semantics of the query statement. Because the semantics of the query statement and the semantics of the reference text have a high degree of match, the degree of match between the semantics of the reference keyword and the semantics of the reference text can be improved.

[0138] In another possible implementation, the text concatenation device extracts keywords from the reference query statement to obtain one or more query keywords, and selects any one of the one or more query keywords as a reference keyword.

[0139] Optionally, the text splicing device obtains one or more query keywords by performing the following steps: extracting keywords of the reference query statement to obtain one or more candidate keywords. Determining invalid keywords from one or more candidate keywords. Removing invalid keywords from one or more candidate keywords to obtain one or more query keywords.

[0140] In the embodiment of the present application, invalid keywords include stop words, wherein stop words are words with relatively low practical meanings. For example, "the following sentence" and "the best method" are words with relatively low practical meanings, and therefore, "the following sentence" and "the best method" are invalid keywords.

[0141] In one possible implementation, the text splicing device performs the following steps: determining that the words in the reference query statement whose number of appearances is greater than or equal to the second threshold are high-frequency words. Determining that the high-frequency words in one or more candidate keywords are invalid keywords. Invalid keywords usually appear at a high frequency in the text, that is, invalid keywords are usually high-frequency words. For example, modal particles are words with low actual meanings, that is, modal particles are invalid keywords. Obviously, modal particles are high-frequency words in the text. Therefore, the text splicing device determines whether the words in the reference query statement are high-frequency words based on the second threshold. Specifically, if the number of times a word appears is greater than or equal to the second threshold, it means that the word is a high-frequency word. Conversely, if the number of times a word appears is less than the second threshold, it means that the word is not a high-frequency word. Therefore, the text splicing device determines that the words in the reference query statement whose number of appearances is greater than or equal to the second threshold are high-frequency words. Then determine that the high-frequency words in one or more candidate keywords are invalid keywords.

[0142] Optionally, the number of times a word appears is the number of times the word appears in the reference text. For example, the reference query includes word A and word B, where word A appears 108 times in the reference text and word B appears 30 times in the reference text. If the second threshold is 50, then the number of times word A appears is greater than the second threshold, that is, word A is a high-frequency word. The number of times word B appears is less than the second threshold, that is, word A is a high-frequency word.

[0143] Optionally, the text splicing device determines that words that appear in the reference query sentence with a frequency greater than or equal to a fifth threshold are high-frequency words, wherein the frequency of word appearance is the ratio of the number of times the word appears in the reference text to the number of words in the reference text.

[0144] Since the semantic contribution of invalid keywords to the semantic contribution of reference query sentences is low, and the semantic contribution of invalid keywords to the semantic contribution of reference text is also low, the confidence that invalid keywords are reference keywords of reference text is low. Therefore, after determining invalid keywords in one or more candidate keywords, the text splicing device removes invalid keywords in one or more candidate keywords to obtain one or more query keywords, and can remove candidate keywords with low confidence, which is conducive to the subsequent determination of reference keywords based on one or more query keywords.

[0145] In this embodiment, the text splicing device first determines the reference query statement of the reference text, and then obtains the reference keywords of the reference text by extracting the keywords of the query statement. Since the data volume of the reference query statement is smaller than that of the reference text, by extracting the keywords of the reference query statement and obtaining the reference keywords of the reference text, the data processing volume can be reduced and the processing speed can be improved.

[0146] It should be understood that the technical solutions provided in step 2001 and step 2002 are applicable to extracting target keywords from any one of two or more texts to be processed.

[0147] As an optional implementation, the contents of two or more texts to be processed involve two or more different fields. In this implementation, the contents of two or more texts to be processed involve two or more different fields, which can increase the diversity of the fields involved in the contents of the two or more texts to be processed, and further increase the diversity of the fields involved in the contents of the synthesized text when a synthesized text is obtained based on the two or more texts to be processed.

[0148] For example, two or more texts to be processed include text a to be processed and text b to be processed, wherein the content of text a to be processed involves the legal field, the content of text a to be processed involves the legal field, and the content of text b to be processed involves the sports field.

[0149] As an optional implementation, the text splicing device performs the following steps during the execution of step 103:

[0150] 3001. Determine the texts to be processed having the same target keyword from the set of texts to be processed as two or more candidate texts.

[0151] The implementation principle of this step is the same as that of step 103, and will not be repeated here. However, two or more candidate texts are obtained by executing this step, while two or more texts to be spliced ​​are obtained by executing step 103.

[0152] 3002. Select any one of the two or more candidate texts as the seed text.

[0153] In the embodiment of the present application, the text other than the seed text among two or more candidate texts is the text to be confirmed, that is, the text to be confirmed is any text other than the seed text among the two or more candidate texts.

[0154] 3003. Determine the similarity between the seed text and the text to be confirmed to obtain a first similarity.

[0155] The first similarity is the similarity between the seed text and the text to be confirmed. It should be understood that the similarity between texts is different from the correlation between texts, wherein the correlation between texts is the semantic matching degree of the texts, and the similarity between texts is the similarity of the texts in the texts. For example, text a is a text describing basketball shoes, and text b is a text describing a basketball game. Since both text a and text b are texts related to basketball, text a and text b have a high correlation. However, since there are more texts related to basketball shoes in text a, and more texts related to basketball games in text b, the similarity between the texts of the two is not high, that is, the similarity between text a and text b is not high.

[0156] Since the number of texts to be confirmed is one or more, in step 3003, the text splicing device determines the similarity between the seed text and each text to be confirmed to obtain a first similarity, that is, one text to be confirmed corresponds to one first similarity, and the number of first similarities is one or more.

[0157] 3004. Determine the first similarity that is less than or equal to a third threshold as a second similarity.

[0158] In the embodiment of the present application, the third threshold is a basis for judging whether the first similarity is large or small. Specifically, if the first similarity is less than or equal to the third threshold, it means that the first similarity is small. On the contrary, if the first similarity is greater than the third threshold, it means that the first similarity is large. The text splicing device determines that the first similarity that is less than or equal to the third threshold is the second similarity, that is, the second similarity is smaller than the first similarity.

[0159] 3005. Based on the text to be confirmed corresponding to the second similarity and the seed text, the two or more texts to be spliced ​​are obtained.

[0160] Since the second similarity is smaller than the first similarity, the similarity between the text of the text to be confirmed corresponding to the second similarity and the text of the seed text is small, that is, the difference between the text of the text to be confirmed corresponding to the second similarity and the text of the seed text is large. At this time, the text splicing device obtains two or more texts to be spliced ​​based on the text to be confirmed and the seed text corresponding to the second similarity, which can improve the difference between the text of the two or more texts to be spliced. In this way, by subsequently splicing two or more texts to be spliced ​​to obtain a synthesized text, the text in the synthesized text can be differentiated, and the diversity of the content in the synthesized text can also be improved.

[0161] In one possible implementation, the text splicing device selects texts from the texts to be confirmed corresponding to the second similarity in order from low to high, as alternative texts, and when the sum of the data volume of the alternative texts and the data volume of the seed text is greater than or equal to a first threshold, determines that the alternative texts and the seed text are two or more texts to be spliced.

[0162] For example, the data volume of the seed text is 13K tokens, and the texts to be confirmed corresponding to the second similarity include text a to be confirmed, text b to be confirmed, and text c to be confirmed, wherein the second similarity of text a to be confirmed is greater than the second similarity of text b to be confirmed, and the second similarity of text b to be confirmed is greater than the second similarity of text c to be confirmed. The data volume of text a to be confirmed is 10K tokens, the data volume of text b to be confirmed is 14K tokens, and the data volume of text c to be confirmed is 17K tokens. The first threshold is 32K tokens. The text splicing device first selects text a to be confirmed from the texts to be confirmed corresponding to the second similarity as the alternative text. Since the data volume of text a to be confirmed and the data volume of the seed text are less than the first threshold, the text splicing device continues to select text b to be confirmed from the texts to be confirmed corresponding to the second similarity as the alternative text. At this time, the alternative texts include text a to be confirmed and text b to be confirmed. Since the sum of the data volume of text a to be confirmed, the data volume of text b to be confirmed, and the data volume of the seed text is greater than the first threshold, the text splicing device treats text a to be confirmed, text b to be confirmed, and seed text as two or more texts to be spliced.

[0163] Since the data volume of the synthetic text is required to be greater than or equal to the first threshold, and the synthetic text is obtained by splicing two or more texts to be spliced, the data volume of the two or more texts to be spliced ​​is required to be greater than or equal to the first threshold. Considering that the lower the second similarity of the text to be confirmed, the greater the difference between the text of the text to be confirmed and the text of the seed text, the text splicing device selects texts from the text to be confirmed corresponding to the second similarity in order from low to high, as candidate texts, which can increase the difference between the text of the candidate text and the text of the seed text. Finally, when the sum of the data volume of the candidate text and the data volume of the seed text is greater than or equal to the first threshold, it is determined that the candidate text and the seed text are two or more texts to be spliced, and the number of texts to be spliced ​​can be reduced and the difference between the texts of the two or more texts to be spliced ​​can be increased when the data volume of the two or more texts to be spliced ​​is greater than or equal to the first threshold.

[0164] In another possible implementation, the text splicing device uses the text to be confirmed corresponding to the second similarity and the seed text as two or more texts to be spliced. In yet another possible implementation, the text splicing device selects n texts from the text to be confirmed corresponding to the second similarity to obtain n backup texts, where n is a positive integer. The n backup texts and the seed text are used as two or more texts to be spliced.

[0165] In this embodiment, the text splicing device determines the texts to be processed with the same target keywords from the set of texts to be processed as two or more candidate texts. One of the two or more candidate texts is selected as the seed text, and the texts other than the seed text in the two or more candidate texts are the texts to be confirmed. The similarity between the seed text and the text to be confirmed is determined to obtain a first similarity. The first similarity that is less than or equal to the third threshold is determined as the second similarity. Based on the text to be confirmed and the seed text corresponding to the second similarity, two or more texts to be spliced ​​are obtained, which can improve the difference in the texts of the two or more texts to be spliced.

[0166] In a possible scenario, synthetic text is used to simulate a conversation in a real scene. Since a conversation in a real scene usually involves a change of topic, for example, in a conversation between Zhang San and Li Si, they first talked about basketball games and then talked about basketball shoes, that is, the topic between Zhang San and Li Si changed from basketball games to basketball shoes. After improving the difference between the texts of two or more texts to be spliced, the text splicing device splices two or more texts to be spliced ​​to obtain a synthetic text, which can make the content in the synthetic text more diverse, and thus make the synthetic text better simulate a conversation in a real scene.

[0167] Optionally, the synthetic text is used as training data to train the LLM, so as to improve the conversation ability of the LLM through training. In this case, if the synthetic text can better simulate the conversation in the real scene, the training effect of the LLM can be improved.

[0168] As an optional implementation, the text splicing device performs the following steps during the execution of step 104:

[0169] 4001. Splice the two or more texts to be spliced ​​to obtain a spliced ​​text.

[0170] 4002. Determine the synthesis probability of the concatenated text based on the content of the concatenated text.

[0171] In the embodiment of the present application, the synthesis probability is the probability that the spliced ​​text is obtained by synthesis, and the higher the synthesis probability, the higher the probability that the spliced ​​text is a synthesized text. In a possible implementation, the text splicing device determines the synthesis probability of the spliced ​​text based on one or more of the following in the spliced ​​text: the fluency of the content, the logic of the content, and the writing style of the content.

[0172] Specifically, compared with synthesized text, human-written text has more fluent and natural expressions, while synthetic text has more fluent and natural expressions, and the content is usually more fluent. Compared with synthesized text, human-written text usually has clearer logic and structure. Compared with synthesized text, human-written text has a more stable writing style.

[0173] In another possible implementation, the text splicing device inputs the spliced ​​text into the judgment model so that the judgment model determines the synthesis probability of the spliced ​​text based on the content of the spliced ​​text. The judgment model has the ability to determine the probability that the text is a text obtained by synthesis based on the content of the text.

[0174] 4003. When the synthesis probability is less than or equal to a fourth threshold, determine that the concatenated text is a synthesized text.

[0175] In the embodiment of the present application, the fourth threshold is a basis for judging whether the synthesis probability is large or small. Specifically, if the synthesis probability is less than or equal to the fourth threshold, it means that the synthesis probability is small. On the contrary, if the synthesis probability is greater than the fourth threshold, it means that the synthesis probability is large. Therefore, when the synthesis probability is less than or equal to the fourth threshold, the text splicing device determines that the spliced ​​text is a synthesized text, which can improve the authenticity of the synthesized text.

[0176] See also Figure 2 , Figure 2 A flowchart of another text splicing method provided in an embodiment of the present application. Figure 2As shown, firstly, a query statement is generated based on a query statement generation model, and a query statement of the text to be processed in the text set to be processed can be generated. Optionally, the query statement generation model is a document-to-query statement model (doc2query). Then, the keywords of the query statement are extracted based on rake, and the target keywords of the text to be processed can be obtained. Then, an inverted index library is obtained by establishing an inverted index. Based on the target keyword, two or more texts to be spliced ​​with the same target keyword can be found from the inverted index library, which can improve the efficiency of determining two or more texts to be spliced. Finally, a synthetic text can be generated by splicing two or more texts to be spliced. Optionally, two or more texts to be spliced ​​are encoded to obtain two or more encoding results, wherein the encoding results include tokens. Then, the two or more encoding results are spliced ​​to obtain a synthetic text.

[0177] As an optional implementation, after obtaining two or more synthetic texts by the text splicing method described above, the two or more synthetic texts are annotated to obtain two or more training texts. The language model is trained using two or more training texts to improve the processing effect of the language model on the text. Specifically, a training prompt word is generated based on the training text, wherein the training prompt word is used to guide the language model to perform a task based on the training text. Based on the training prompt word, the language model is guided to perform a task based on the target text to obtain an execution result. Based on the difference between the execution result and the label of the training text, the loss of the language model is obtained, wherein the loss is positively correlated with the difference. Based on the loss of the language model, the parameters of the language model are updated until the loss of the language model converges. It should be understood that the device for training the language model using two or more training texts and the text splicing device can be different or the same. Optionally, the amount of text data that can be processed by the language model is greater than 32K tokens. Optionally, the language model is a pythia model.

[0178] Optionally, after completing the training of the language model, the language model is tested. Specifically, the training text, the first text and the second text are mixed in different mixing methods to obtain different training data sets, wherein the first text is a short text, the second text is a long text, and neither the first text nor the second text is a text obtained by synthesis. Optionally, the second text is an open source data set: text in longbench. Then, the language model is trained separately using each training data set, and then the language model trained based on different training data sets is tested to evaluate the training effect of different training data sets. Table 1 below is one of the test results.

[0179]

[0180] Table 1

[0181] As shown in Table 1, the test results include average score, single text answer score, multi-text answer score, extractive summary score, few-shot task score, and synthetic task score. The single text answer score is the evaluation result of the trained language model performing single text answer based on the test data, the long text answer score is the evaluation result of the trained language model performing multi-text answer based on the test data, the extractive summary score is the evaluation result of the trained language model performing extractive summary based on the test data, the few-shot task is the evaluation result of the trained language model performing the few-shot task based on the test data, and the synthetic task is the evaluation result of the trained language model performing the synthetic task based on the test data. The average score is the average of the single text question answering score, multi-text question answering score, extractive summary score, few-shot task score, and synthetic task score.

[0182] Single-text answer refers to answering questions based on one text in the test data. Multi-text answer refers to answering questions based on two or more texts in the test data. Extracting summaries refers to extracting summaries of texts in the test data. Few-sample tasks refer to the number of examples in the prompt words of the test data being less than or equal to the sixth threshold, and optionally, the sixth threshold is 2. A synthetic task can be a combination of two or more of single-text answer, multi-text answer, and few-sample tasks.

[0183] In Table 1, there are four types of training data sets. The text in the first type of training data set is: training text determined by randomly selecting a target keyword, wherein the implementation process of randomly selecting a target keyword can refer to the implementation method of selecting any keyword from one or more query keywords as a reference keyword. The text in the second type of training data set is: training text determined by selecting a reference keyword with the maximum value of the first correlation as the target keyword, wherein the implementation process of selecting the reference keyword with the maximum value of the first correlation as the target keyword can refer to the implementation method of determining the query keyword corresponding to the maximum value in one or more first correlations as the reference keyword. The text in the third type of training data set is the second text. The text in the fourth type of training data set is the first text. The data volume of the text in each training data set is 4 billion (B) tokens.

[0184] It can be seen from Table 1 that when the language model is trained using the first training dataset, the average score of the language model is the highest. When the language model is trained using the second training dataset, the average score of the language model is the second highest. This shows that training the language model with the synthetic text obtained by the text concatenation method can improve the training effect of the language model. In addition, it can be seen from Table 1 that when the language model is trained using the first training dataset, the language model has the highest score for the few-sample task. When the language model is trained using the second training dataset, the language model has the second highest score for the few-sample task. This shows that training the language model with the synthetic text obtained by the text concatenation method can improve the language model's ability to perform few-sample tasks.

[0185] Table 2 below shows another test result.

[0186]

[0187] Table 2

[0188] In Table 2, there are four types of training data sets. The text in the first training data set is: the second text and the training text determined by randomly selecting the target keyword. Optionally, the text in the first training data set is obtained by splicing the second text and the training text. The text in the second training data set is: the second text and the training text determined by selecting the reference keyword with the maximum value of the first correlation as the target keyword. Optionally, the text in the second training data set is obtained by splicing the second text and the training text. The text in the third training data set is the first text and the second text. Optionally, the text in the third training data set is obtained by splicing the first text and the second text. The text in the fourth training data set is the second text. Optionally, the text in the third training data set is obtained by splicing the second text and the second text. The data amount of the text in each training data set is 8B tokens.

[0189] It can be seen from Table 2 that when the language model is trained using the first training data set, the average score of the language model is the highest. When the language model is trained using the second training data set, the average score of the language model is the second highest. This shows that using the second text and the synthetic text to train the language model can improve the training effect of the language model. In addition, it can be seen from Table 2 that when the language model is trained using the first training data set, the language model has the highest score for the few-sample task. When the language model is trained using the second training data set, the language model has the second highest score for the few-sample task. This shows that using the second text and the synthetic text to train the language model can improve the ability of the language model to perform the few-sample task. Moreover, by comparing Tables 1 and 2, it can be seen that the use of synthetic text and the second text can further improve the training effect of the language model and improve the ability of the language model to perform the few-sample task compared to the use of synthetic text.

[0190] The embodiment of the present application also provides a text processing method, the execution subject of the text processing method is a text processing device, wherein the text processing device can be any electronic device that can execute the technical solution disclosed in the embodiment of the method of the present application. Optionally, the text processing device can be one of the following: a computer, a server.

[0191] See also Figure 3 , Figure 3 A flowchart of a text processing method provided in an embodiment of the present application.

[0192] 301. Obtain target tasks and target texts.

[0193] In the embodiment of the present application, the target task is a natural language processing task, for example, the target task is to answer questions based on text. The data volume of the target text is greater than or equal to the first threshold.

[0194] 302. Input the target text into a target language model, so that the target language model executes the target task based on the target text to obtain an execution result.

[0195] In an embodiment of the present application, the target language model is obtained by training using training data, and the training data includes synthetic text, wherein the synthetic text is obtained according to the text splicing method described above.

[0196] In one possible implementation, the target task is to answer the target question based on the target text, and the execution result is the target answer to the target question determined by the target language model based on the target text. Optionally, training data can be obtained by annotating the target text. The language model is used to answer the question based on the training data to obtain a first answer. Based on the difference between the first answer and the label of the training data, the loss of the language model is obtained, wherein the loss is positively correlated with the difference. Based on the loss of the language model, the parameters of the language model are updated until the loss of the language model converges to obtain the target language model. Since the target text is a long text and the authenticity of the target text is relatively good, the target language model is obtained by training with the target text, which can improve the processing effect of the target language model on long texts. Specifically, when the target language model is used to answer questions based on long texts, a higher accuracy can be obtained.

[0197] In the embodiment of the present application, since the target language model has a better processing effect on long texts, and the target text is a long text, after obtaining the target question and the target text, the text processing device inputs the target text into the target language model, so that the target language model executes the target task based on the target text and obtains the execution result, which can improve the accuracy of the execution result.

[0198] Optionally, the training data also includes non-synthetic text, wherein the data volume of the non-synthetic text is greater than or equal to the first threshold. According to Tables 1 and 2, compared with using synthetic text, using synthetic text and the second text can further improve the training effect of the language model and improve the ability of the language model to perform small sample tasks, wherein the second text is non-synthetic text, that is, the second text is not a long text obtained by synthesis. Therefore, when the training data includes synthetic text and non-synthetic text, the ability of the target language model can be improved, and the accuracy of the execution result can be improved.

[0199] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0200] The method of the embodiment of the present application is described in detail above, and the device of the embodiment of the present application is provided below.

[0201] See also Figure 4 , Figure 4 The present invention provides a structural diagram of a text splicing device according to an embodiment of the present invention. The text splicing device 1 includes: an acquisition unit 11, an extraction unit 12, a determination unit 13, and a splicing unit 14. Specifically:

[0202] An acquisition unit 11 is used to acquire a set of texts to be processed, wherein the set of texts to be processed includes two or more texts to be processed, and the data volume of the texts to be processed is less than a first threshold;

[0203] An extraction unit 12 is used to extract keywords from the two or more texts to be processed to obtain two or more target keywords, wherein one of the texts to be processed has one or more target keywords;

[0204] A determination unit 13 is used to determine the to-be-processed texts having the same target keyword from the to-be-processed text set, to obtain two or more to-be-joined texts;

[0205] The splicing unit 14 is used to splice the two or more texts to be spliced ​​to obtain a composite text, where the data volume of the composite text is greater than or equal to the first threshold.

[0206] In combination with any embodiment of the present application, the two or more texts to be processed include a reference text, and the target keyword of the reference text includes a reference keyword of the reference text;

[0207] The extraction unit 12 is specifically used for:

[0208] Determine a reference query statement of the reference text, wherein the reference query statement is a basis for querying to obtain the reference text;

[0209] The reference keywords are obtained by extracting keywords from the reference query statement.

[0210] In combination with any embodiment of the present application, the extraction unit 12 is specifically used for:

[0211] Extracting keywords from the reference query statement to obtain one or more query keywords;

[0212] Selecting one keyword from the one or more query keywords as the reference keyword;

[0213] Alternatively, determining the correlation between the one or more query keywords and the reference query statement to obtain one or more first correlations;

[0214] The query keyword corresponding to the maximum value among the one or more first correlations is determined as the reference keyword.

[0215] In combination with any embodiment of the present application, the extraction unit 12 is specifically used for:

[0216] Extracting keywords from the reference query to obtain one or more candidate keywords;

[0217] determining invalid keywords among the one or more candidate keywords;

[0218] The invalid keywords in the one or more candidate keywords are removed to obtain the one or more query keywords.

[0219] In combination with any embodiment of the present application, the extraction unit 12 is specifically used for:

[0220] Determine that the words in the reference query sentence whose number of appearances is greater than or equal to a second threshold are high-frequency words;

[0221] The high-frequency words in the one or more candidate keywords are determined to be the invalid keywords.

[0222] In combination with any embodiment of the present application, the extraction unit 12 is specifically used for:

[0223] Obtaining a query statement generation model, wherein the query statement generation model is used to generate a text query statement based on the text, and the maximum data volume of the text that can be processed by the query statement generation model is a data volume threshold;

[0224] In the case where the data volume of the reference text exceeds the data volume threshold, dividing the reference text into two or more segmented texts, where the data volume of the segmented texts is less than or equal to the data volume threshold;

[0225] Inputting the two or more segmented texts into the query statement generation model respectively, so that the query statement generation model generates query statements for the reference texts, thereby obtaining two or more segmented query statements;

[0226] The reference query statement is obtained based on the two or more segmented query statements.

[0227] In combination with any implementation manner of the present application, the determining unit 13 is specifically configured to:

[0228] Determining the texts to be processed having the same target keyword from the set of texts to be processed as two or more candidate texts;

[0229] Selecting one of the two or more candidate texts as a seed text, and the text other than the seed text among the two or more candidate texts as the text to be confirmed;

[0230] Determining the similarity between the seed text and the text to be confirmed to obtain a first similarity;

[0231] Determine the first similarity that is less than or equal to a third threshold as a second similarity;

[0232] Based on the text to be confirmed and the seed text corresponding to the second similarity, the two or more texts to be spliced ​​are obtained.

[0233] In combination with any embodiment of the present application, the splicing unit 14 is specifically used for:

[0234] Splicing the two or more texts to be spliced ​​to obtain a spliced ​​text;

[0235] Determining a synthesis probability of the concatenated text based on the content of the concatenated text, wherein the synthesis probability is a probability that the concatenated text is a text obtained by synthesis;

[0236] When the synthesis probability is less than or equal to a fourth threshold, the concatenated text is determined to be a synthesized text.

[0237] In an embodiment of the present application, the set of texts to be processed includes two or more texts to be processed, and the data volume of the texts to be processed is less than a first threshold value. After acquiring the set of texts to be processed, the text splicing device extracts keywords of two or more texts to be processed to obtain two or more target keywords. The texts to be processed with the same target keywords are determined from the set of texts to be processed to obtain two or more texts to be spliced, thereby obtaining two or more texts to be spliced ​​with high semantic matching. The two or more texts to be spliced ​​are spliced ​​to obtain a composite text, wherein the data volume of the composite text is greater than or equal to the first threshold value, which can improve the authenticity of the composite text and make the content of the composite text more fluent.

[0238] See also Figure 5 , Figure 5 This is a structural diagram of a text processing device provided in an embodiment of the present application. The text processing device 2 includes: an acquisition unit 21 and a processing unit 22. Specifically:

[0239] An acquisition unit 21 is used to acquire a target task and a target text, wherein the target task is a natural language processing task, and the data volume of the target text is greater than or equal to a first threshold;

[0240] The processing unit 22 is used to input the target text into the target language model so that the target language model performs the target task based on the target text to obtain an execution result. The target language model is trained using a training text, and the training text includes a synthetic text. The synthetic text is obtained according to the first aspect and any embodiment thereof.

[0241] In combination with any implementation manner of the present application, the training data also includes non-synthetic text, and the data volume of the non-synthetic text is greater than or equal to a first threshold.

[0242] In the embodiment of the present application, since the target language model has a better processing effect on long texts, and the target text is a long text, after obtaining the target question and the target text, the text processing device inputs the target text into the target language model, so that the target language model executes the target task based on the target text and obtains the execution result, which can improve the accuracy of the execution result.

[0243] In some embodiments, the functions or modules included in the device provided in the embodiments of the present application can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0244] Figure 6 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. The electronic device 3 includes a processor 31 and a memory 32. Optionally, the electronic device 3 also includes an input device 33 and an output device 34. The processor 31, the memory 32, the input device 33 and the output device 34 are coupled via a connector, and the connector includes various interfaces, transmission lines or buses, etc., which are not limited in the embodiments of the present application. It should be understood that in each embodiment of the present application, coupling refers to mutual connection in a specific manner, including direct connection or indirect connection through other devices, for example, it can be connected through various interfaces, transmission lines, buses, etc.

[0245] The processor 31 may include one or more processors, for example, one or more central processing units (CPUs). When the processor is a CPU, the CPU may be a single-core CPU or a multi-core CPU. Optionally, the processor 31 may be a processor group consisting of multiple CPUs, and the multiple processors are coupled to each other via one or more buses. Optionally, the processor may also be other types of processors, etc., which are not limited in the embodiments of the present application.

[0246] The memory 32 can be used to store computer program instructions and various computer program codes including the program code for executing the program code of the present application. Optionally, the memory includes but is not limited to random access memory (RAM), read-only memory (ROM), erasable programmable read only memory (EPROM), or portable read only memory (CD-ROM), which is used for related instructions and data.

[0247] The input device 33 is used to input data and / or signals, and the output device 34 is used to output data and / or signals. The input device 33 and the output device 34 can be independent devices or an integrated device.

[0248] It can be understood that in the embodiment of the present application, the memory 32 can be used not only to store relevant instructions, but also to store relevant data. For example, the memory 32 can be used to store a set of texts to be processed obtained through the input device 33, or the memory 32 can also be used to store a target text obtained through the processor 31, etc. The embodiment of the present application does not limit the specific data stored in the memory.

[0249] Understandably, Figure 6 Only a simplified design of an electronic device is shown. In practical applications, the electronic device may also include other necessary components, including but not limited to any number of input / output devices, processors, memories, etc., and all electronic devices that can implement the embodiments of the present application are within the protection scope of the present application.

[0250] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0251] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. Those skilled in the art can also clearly understand that the descriptions of the various embodiments of the present application have different focuses. For the convenience and brevity of description, the same or similar parts may not be repeated in different embodiments. Therefore, for parts not described or not described in detail in a certain embodiment, refer to the records of other embodiments.

[0252] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0253] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0254] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0255] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital versatile disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0256] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by a computer program to instruct the relevant hardware, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above-mentioned method embodiments. The aforementioned storage medium includes: a read-only memory (ROM) or a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

Claims

1. A text splicing method, characterized in that: The method comprises: Acquire a text set to be processed, wherein the text set to be processed includes two or more texts to be processed, and the data volume of the texts to be processed is less than a first threshold; Extracting keywords from the two or more texts to be processed to obtain two or more target keywords, wherein one of the texts to be processed has one or more of the target keywords; Determine the texts to be processed having the same target keyword from the set of texts to be processed, and obtain two or more texts to be spliced; The two or more texts to be spliced ​​are spliced ​​to obtain a composite text, and the data volume of the composite text is greater than or equal to the first threshold.

2. The method according to claim 1, characterized in that The two or more texts to be processed include a reference text, and the target keywords of the reference text include reference keywords of the reference text; The step of extracting keywords from the two or more texts to be processed to obtain two or more target keywords includes: Determine a reference query statement of the reference text, wherein the reference query statement is a basis for querying and obtaining the reference text; The reference keywords are obtained by extracting keywords from the reference query statement.

3. The method according to claim 2, characterized in that The step of extracting the keywords of the reference query statement to obtain the reference keywords includes: Extracting keywords from the reference query statement to obtain one or more query keywords; Selecting one keyword from the one or more query keywords as the reference keyword; Alternatively, determining the correlation between the one or more query keywords and the reference query statement to obtain one or more first correlations; The query keyword corresponding to the maximum value among the one or more first correlations is determined as the reference keyword.

4. The method according to claim 3, characterized in that The extracting keywords of the reference query statement to obtain one or more query keywords includes: Extracting keywords from the reference query to obtain one or more candidate keywords; determining invalid keywords among the one or more candidate keywords; The invalid keywords in the one or more candidate keywords are removed to obtain the one or more query keywords.

5. The method according to claim 4, characterized in that The step of determining an invalid keyword among the one or more candidate keywords includes: Determine that the words in the reference query sentence whose number of appearances is greater than or equal to a second threshold are high-frequency words; The high-frequency words in the one or more candidate keywords are determined to be the invalid keywords.

6. The method according to any one of claims 2 to 5, characterized in that The step of determining a reference query statement of the reference text includes: Obtaining a query statement generation model, wherein the query statement generation model is used to generate a text query statement based on the text, and the maximum data volume of the text that can be processed by the query statement generation model is a data volume threshold; In the case where the data volume of the reference text exceeds the data volume threshold, dividing the reference text into two or more segmented texts, where the data volume of the segmented texts is less than or equal to the data volume threshold; Inputting the two or more segmented texts into the query statement generation model respectively, so that the query statement generation model generates query statements for the reference texts, and obtaining two or more segmented query statements; The reference query statement is obtained based on the two or more segmented query statements.

7. The method according to any one of claims 1 to 5, characterized in that The step of determining the to-be-processed texts having the same target keyword from the to-be-processed text set to obtain two or more to-be-joined texts includes: Determining the texts to be processed having the same target keyword from the set of texts to be processed as two or more candidate texts; Selecting one of the two or more candidate texts as a seed text, and the text other than the seed text among the two or more candidate texts as the text to be confirmed; Determining the similarity between the seed text and the text to be confirmed to obtain a first similarity; Determine the first similarity that is less than or equal to a third threshold as a second similarity; Based on the text to be confirmed and the seed text corresponding to the second similarity, the two or more texts to be spliced ​​are obtained.

8. The method according to any one of claims 1 to 5, characterized in that The step of splicing the two or more texts to be spliced ​​to obtain a composite text includes: Splicing the two or more texts to be spliced ​​to obtain a spliced ​​text; Determining a synthesis probability of the concatenated text based on the content of the concatenated text, wherein the synthesis probability is a probability that the concatenated text is a text obtained by synthesis; When the synthesis probability is less than or equal to a fourth threshold, the concatenated text is determined to be a synthesized text.

9. A text processing method, characterized in that: The method comprises: Acquire a target task and a target text, wherein the target task is a natural language processing task, and the data volume of the target text is greater than or equal to a first threshold; The target text is input into a target language model so that the target language model executes the target task based on the target text to obtain an execution result, wherein the target language model is trained using training data, wherein the training data includes synthetic text, and the synthetic text is obtained according to the method described in any one of claims 1 to 8.

10. The method according to claim 9, characterized in that The training data also includes non-synthetic text, and the data volume of the non-synthetic text is greater than or equal to a first threshold.

11. A text splicing device, characterized in that: The text processing device comprises: An acquisition unit, used for acquiring a text set to be processed, wherein the text set to be processed includes two or more texts to be processed, and the data volume of the texts to be processed is less than a first threshold; An extraction unit, used for extracting keywords from the two or more texts to be processed to obtain two or more target keywords, wherein one of the texts to be processed has one or more target keywords; A determination unit, used to determine the to-be-processed texts having the same target keyword from the to-be-processed text set, to obtain two or more to-be-joined texts; The splicing unit is used to splice the two or more texts to be spliced ​​to obtain a composite text, and the data volume of the composite text is greater than or equal to the first threshold.

12. A text processing device, characterized in that: The text processing device comprises: An acquisition unit, used to acquire a target task and a target text, wherein the target task is a natural language processing task, and the data volume of the target text is greater than or equal to a first threshold; A processing unit is used to input the target text into a target language model so that the target language model performs the target task based on the target text to obtain an execution result, wherein the target language model is trained using a training text, wherein the training text includes a synthetic text, and the synthetic text is obtained according to the method described in any one of claims 1 to 8.

13. An electronic device, characterized in that: include: A processor and a memory, wherein the memory is used to store computer program code, wherein the computer program code includes computer instructions, and when the processor executes the computer instructions, the electronic device executes the method as claimed in any one of claims 1 to 8, or the electronic device executes the method as claimed in claim 9 or 10.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which includes program instructions. When the program instructions are executed by a processor, the processor is caused to execute the method described in any one of claims 1 to 8, or the processor is caused to execute the method described in claim 9 or 10.

15. A computer program product, characterized in that The computer program product includes a computer program or instructions; when the computer program or instructions are run on a computer, the computer is enabled to execute the method according to any one of claims 1 to 8, or the computer is enabled to execute the method according to claim 9 or 10.

Images