Text method and device, electronic equipment and storage medium
By performing sentence splitting, segmenting and splicing processing on text processed by pre-trained language models, the problems of loss of sentence components and semantic incoherence in long text processing are solved, and the quality and efficiency of model training are improved.
Patent Information
- Application Number
- CN202311057190.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-08-21
- Publication Date
- 2025-02-21
AI Technical Summary
Existing pre-trained language models can easily lead to loss of sentence components and semantic incoherence when processing long texts, affecting the model training effect.
By splitting each training sample in the to be processed document, obtaining at least one sentence and segmenting or splicing according to the length of the sentence, ensuring that the length of each clause or splicing sentence is less than or equal to the preset cutoff length while maintaining semantic coherence .
It effectively avoids the problems of sentence component loss and semantic incoherence, and improves the quality and efficiency of model training.
Smart Images
Figure CN119494344A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computers, and more specifically, to a method, device, electronic device and storage medium for text processing in the field of computers. Background Art
[0002] Currently, most pre-trained language models choose Transformer as the basic model. Unlike recurrent neural networks, Transformer can calculate all representations of input samples at the same time. In theory, it can remember contexts of any length. However, large models such as Transformer, which consume a lot of memory, are easily limited by video memory. The longer the maximum cutoff length we set, the longer the context length, the more information Transformer can pay attention to, and the more video memory it occupies. The maximum length that a graphics card can support is limited, so the input length must be truncated according to the maximum cutoff length. Therefore, the processing of pre-training corpus becomes a difficult problem.
[0003] The current method of processing pre-training corpus is: directly truncate the training sample according to the maximum cutoff length. If the training sample length is less than the maximum cutoff length, directly splice the next training sample until the length reaches the maximum cutoff length; if the training sample length is greater than the maximum cutoff length, directly truncate it and splice the remaining text to the next training sample. The above processing method is likely to lead to problems such as loss of sentence components and semantic incoherence, which in turn has an adverse effect on model training. Summary of the invention
[0004] The present application provides a method, device, electronic device and storage medium for text processing, which can avoid problems such as loss of sentence components and semantic incoherence when processing text, thereby avoiding adverse effects on model training.
[0005] In a first aspect, a method for text processing is provided, the method comprising: obtaining a document to be processed; wherein the document to be processed contains a plurality of training samples; splitting each of the training samples in the document to be processed to obtain at least one sentence corresponding to each of the training samples; if the length of the sentence is greater than a preset cutoff length, splitting the sentence according to a separator in the sentence to obtain a plurality of clauses corresponding to the sentence; wherein the length of each of the clauses is less than or equal to the preset cutoff length; if the length of the sentence is less than the preset cutoff length, concatenating the sentence with other sentences in the training sample except the sentence to obtain a concatenated sentence having a length less than or equal to the preset cutoff length and semantically coherent.
[0006] In the above technical solution, each training sample in the document to be processed is split to obtain at least one sentence corresponding to each training sample, that is, the sentence is used as the smallest unit during the splitting to avoid splitting the complete sentence. If the length of the sentence is greater than the preset cutoff length, the sentence is split according to the delimiter in the sentence to obtain multiple clauses corresponding to the sentence, and the length of each clause is less than or equal to the preset cutoff length. By splitting the sentence according to the delimiter, the semantics that are easily cut due to the truncation of a single sentence according to the preset cutoff length can be avoided to a certain extent, and at the same time, the length of the cut clause can be ensured not to exceed the preset cutoff length. If the length of the sentence is less than the preset cutoff length, the sentence is spliced to obtain a spliced sentence with a length less than or equal to the preset cutoff length and semantic coherence, which can also avoid semantic incoherence caused by random splicing.
[0007] In combination with the first aspect, in some possible implementations, after obtaining the document to be processed, it also includes: splicing the training samples in the document to be processed whose length is less than a preset cutoff length to obtain a target document; wherein the length of each target training sample in the target document is greater than or equal to the preset cutoff length; splitting each of the training samples in the document to be processed to obtain at least one sentence corresponding to each of the training samples includes: splitting each of the target training samples in the target document to obtain at least one sentence corresponding to each of the target training samples.
[0008] In combination with the first aspect and the above-mentioned implementation methods, in some possible implementation methods, the splicing of training samples in the document to be processed whose length is less than a preset cutoff length to obtain a target document includes: obtaining documents of the same category as the document to be processed; extracting samples to be spliced from the training samples of the documents of the same category, and splicing the samples to be spliced with the training samples in the document to be processed whose length is less than the preset cutoff length to obtain the target document.
[0009] In combination with the first aspect and the above-mentioned implementation methods, in some possible implementation methods, splitting each of the training samples in the document to be processed to obtain at least one sentence corresponding to each of the training samples includes: splitting each of the training samples in the document to be processed according to sentence terminators to obtain at least one sentence corresponding to each of the training samples, and retaining the separators in the at least one sentence.
[0010] In combination with the first aspect and the above-mentioned implementation manner, in some possible implementation manners, the segmenting the sentence according to the delimiters in the sentence to obtain multiple clauses corresponding to the sentence includes: segmenting the sentence according to the preset cutoff length to obtain multiple segmentation results; for a target segmentation result with a delimiter in the multiple segmentation results, segmenting the target segmentation result according to the last delimiter in the target segmentation result; taking the first part of the subwords in the target segmentation result before the last delimiter as the clause corresponding to the sentence, and merging the second part of the subwords in the target segmentation result after the last delimiter with the next segmentation result after the target segmentation result to obtain a merged segmentation result; if the length of the merged segmentation result is less than or equal to the preset cutoff length, taking the merged segmentation result as the clause corresponding to the sentence; if the length of the merged segmentation result is greater than the preset cutoff length, taking the merged segmentation result as the sentence and continuing to perform the step of segmenting the sentence according to the preset cutoff length to obtain multiple segmentation results until multiple clauses corresponding to the sentence are obtained.
[0011] In combination with the first aspect and the above-mentioned implementation methods, in some possible implementation methods, the sentence is concatenated with other sentences in the training sample except the sentence to obtain a concatenated sentence whose length is less than or equal to the preset cutoff length and is semantically coherent, including: determining a target sentence for concatenation with the sentence from the other sentences according to the length of the sentence and the length of the other sentences; wherein the target sentence and the sentence belong to multiple consecutive sentences; and the sentence and the target sentence are concatenated to obtain a concatenated sentence whose length is less than or equal to the preset cutoff length and is semantically coherent.
[0012] In combination with the first aspect and the above-mentioned implementation manner, in some possible implementation manners, determining a target sentence for splicing with the sentence from among the other sentences based on the length of the sentence and the lengths of the other sentences includes: when the sentence is the first sentence in the training sample, determining a target sentence for splicing with the sentence from among the other sentences after the sentence based on the length of the sentence and the lengths of other sentences after the sentence; when the sentence is not the first sentence in the training sample, determining a target sentence for splicing with the sentence from among the other sentences before the sentence based on the length of the sentence and the lengths of other sentences before the sentence.
[0013] In a second aspect, a device for text processing is provided, which includes: an acquisition module, used to acquire a document to be processed; wherein the document to be processed contains multiple training samples; a splitting module, used to split each of the training samples in the document to be processed to obtain at least one sentence corresponding to each of the training samples; a segmentation module, used to split the sentence according to the separators in the sentence if the length of the sentence is greater than a preset cutoff length, to obtain multiple clauses corresponding to the sentence; wherein the length of each of the clauses is less than or equal to the preset cutoff length; a splicing module, used to splice the sentence with other sentences in the training sample except the sentence if the length of the sentence is less than the preset cutoff length, to obtain a spliced sentence with a length less than or equal to the preset cutoff length and semantic coherence.
[0014] In combination with the second aspect, in some possible implementations, the device also includes: a target document acquisition module, used to splice training samples in the document to be processed whose length is less than a preset cutoff length to obtain a target document; wherein the length of each target training sample in the target document is greater than or equal to the preset cutoff length; a splitting module, specifically used to split each of the target training samples in the target document to obtain at least one sentence corresponding to each of the target training samples.
[0015] In combination with the second aspect and the above-mentioned implementation methods, in some possible implementation methods, the target document acquisition module is specifically used to obtain documents of the same category as the document to be processed; extract samples to be spliced from training samples of the documents of the same category, and splice the samples to be spliced with training samples in the document to be processed whose length is less than a preset cutoff length to obtain the target document.
[0016] In combination with the second aspect and the above-mentioned implementation methods, in some possible implementation methods, the splitting module is specifically used to: split each of the training samples in the document to be processed according to the sentence end symbol, obtain at least one sentence corresponding to each of the training samples, and retain the separators in the at least one sentence.
[0017] In combination with the second aspect and the above-mentioned implementation methods, in some possible implementation methods, the segmentation module is specifically used to: segment the sentence according to the preset cutoff length to obtain multiple segmentation results; for a target segmentation result with a separator in the multiple segmentation results, segment the target segmentation result according to the last separator in the target segmentation result; use the first part of sub-words in the target segmentation result before the last separator as the clause corresponding to the sentence, and merge the second part of sub-words in the target segmentation result after the last separator with the next segmentation result after the target segmentation result to obtain a merged segmentation result; if the length of the merged segmentation result is less than or equal to the preset cutoff length, use the merged segmentation result as the clause corresponding to the sentence; if the length of the merged segmentation result is greater than the preset cutoff length, use the merged segmentation result as the sentence to continue to perform the step of segmenting the sentence according to the preset cutoff length to obtain multiple segmentation results until multiple clauses corresponding to the sentence are obtained.
[0018] In combination with the second aspect and the above-mentioned implementation methods, in some possible implementation methods, the splicing module is specifically used to: determine a target sentence for splicing with the sentence in the other sentences according to the length of the sentence and the length of the other sentences; wherein the target sentence and the sentence belong to multiple consecutive sentences; and splice the sentence and the target sentence to obtain a spliced sentence having a length less than or equal to the preset cutoff length and semantically coherent.
[0019] In combination with the second aspect and the above-mentioned implementation methods, in some possible implementation methods, the splicing module is specifically used to: when the sentence is the first sentence in the training sample, determine the target sentence for splicing with the sentence from other sentences after the sentence based on the length of the sentence and the lengths of other sentences after the sentence; when the sentence is not the first sentence in the training sample, determine the target sentence for splicing with the sentence from other sentences before the sentence based on the length of the sentence and the lengths of other sentences before the sentence.
[0020] In a third aspect, an electronic device is provided, comprising a memory and a processor. The memory is used to store executable program code, and the processor is used to call and run the executable program code from the memory, so that the electronic device executes the method in the first aspect or any possible implementation of the first aspect.
[0021] In a fourth aspect, a computer program product is provided, comprising: a computer program code, which, when executed on a computer, enables the computer to execute the method in the first aspect or any possible implementation of the first aspect.
[0022] In a fifth aspect, a computer-readable storage medium is provided, which stores a computer program code. When the computer program code runs on a computer, the computer executes the method in the above-mentioned first aspect or any possible implementation manner of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] Figure 1 is a schematic flow chart of a text processing method provided in an embodiment of the present application;
[0024] Figure 2 is a schematic flow chart of another text processing method provided in an embodiment of the present application;
[0025] Figure 3 is a structural schematic diagram of a text processing device provided in an embodiment of the present application;
[0026] Figure 4 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0027] The technical solution in the present application will be described clearly and in detail below in conjunction with the accompanying drawings. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B: "and / or" in the text is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.
[0028] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features.
[0029] The development of pre-trained language models has brought research in the field of NLP (Neuro-Linguistic Programming) to a new stage. Without manual labeling, common language representations can be learned from massive corpora and significantly improve downstream tasks.
[0030] At present, most pre-trained language models choose Transformer as the basic model. Unlike recurrent neural networks, Transformer can calculate all representations of input samples at the same time. In theory, it can remember contexts of any length. However, large models such as Transformer, which consume a lot of memory, are easily limited by video memory. The longer the maximum cutoff length we set, the longer the context length, the more information Transformer can pay attention to, and the more video memory it occupies. The maximum length that the graphics card can support is limited, so the input length must be truncated according to the maximum cutoff length. Therefore, the processing of pre-training corpus becomes a difficult problem. For example, under the premise that each training sample is as complete as possible, how to cut long texts so that the number of text segments is the least and the length of each training sample is as long as possible. In order to make training more efficient, how to merge short text sentences to reach the maximum cutoff length.
[0031] The current method of processing pre-training corpus is: directly truncate the training sample according to the maximum cutoff length. If the length of the training sample is less than the maximum cutoff length, directly splice the next training sample until the length reaches the maximum cutoff length; if the length of the training sample is greater than the maximum cutoff length, directly truncate it and splice the remaining text to the next training sample.
[0032] However, if the training sample length is greater than the maximum cutoff length, direct truncation will result in the cutting of long texts, resulting in incomplete sentences, which will cause the loss of sentence components in the training sample and even semantic cutting. If the training sample length is less than the maximum cutoff length, the cut samples will be introduced or other samples will be spliced, resulting in incoherent sentence semantic information in the training sample.
[0033] Based on this, in order to solve the problems that the above-mentioned processing method easily leads to loss of sentence components and semantic incoherence, the embodiment of the present application provides a text processing method so that when long texts are cut, each training sample after cutting is as complete a sentence as possible, and when short texts are spliced, appropriate sentences are spliced to avoid problems such as loss of sentence components and semantic incoherence.
[0034] Figure 1 It is a schematic flow chart of a text processing method provided in an embodiment of the present application.
[0035] For example, Figure 1 As shown, the method includes:
[0036] Step 101: Obtain a document to be processed; wherein the document to be processed includes multiple training samples.
[0037] Step 102: Split each training sample in the document to be processed to obtain at least one sentence corresponding to each training sample.
[0038] Step 103: If the length of the sentence is greater than the preset cutoff length, the sentence is segmented according to the separators in the sentence to obtain a plurality of clauses corresponding to the sentence; wherein the length of each clause is less than or equal to the preset cutoff length.
[0039] Step 104: If the length of the sentence is less than the preset cutoff length, concatenate the sentence with other sentences in the training sample except the sentence to obtain a concatenated sentence whose length is less than or equal to the preset cutoff length and has semantic coherence.
[0040] exist Figure 1 In the illustrated embodiment, each training sample in the document to be processed is split to obtain at least one sentence corresponding to each training sample, that is, the sentence is used as the smallest unit during the splitting to avoid splitting the complete sentence. If the length of the sentence is greater than the preset cutoff length, the sentence is split according to the delimiter in the sentence to obtain multiple clauses corresponding to the sentence, and the length of each clause is less than or equal to the preset cutoff length. By splitting the sentence according to the delimiter, the semantics that are easily cut due to the truncation of a single sentence according to the preset cutoff length can be avoided to a certain extent, and at the same time, the length of the cut clause can be ensured not to exceed the preset cutoff length. If the length of the sentence is less than the preset cutoff length, the sentence is spliced to obtain a spliced sentence with a length less than or equal to the preset cutoff length and semantic coherence, which can also avoid semantic incoherence caused by random splicing.
[0041] Below Figure 1 The specific implementation of each step in the embodiment shown is described as follows:
[0042] In step 101, a document to be processed may be obtained; wherein the document to be processed may include multiple documents containing multiple training samples, the multiple documents may be n documents, and the n documents may be represented as: {1, ..., n}.
[0043] For example, after obtaining n documents, the n documents may be classified. For example, the n documents may be classified into categories such as mathematics, English, and Chinese according to subject. In a specific implementation, the n documents may also be classified according to other classification standards, but this embodiment does not specifically limit this.
[0044] In step 102, each training sample in the document to be processed may be split into sentences to obtain at least one sentence corresponding to each training sample. This step aims to split each training sample in the document to be processed into a single sentence.
[0045] In an exemplary embodiment, the implementation of step 102 may include: splitting each training sample in the to-be-processed document according to sentence terminators to obtain at least one sentence corresponding to each training sample, and retaining the separators in the sentences.
[0046] Specifically, each training sample can be segmented into sentences by the sentence ending symbol, and the separators should be retained to ensure the integrity of the sentence. The sentence ending symbol refers to the punctuation mark indicating the end of a sentence, such as a period, an exclamation mark, a question mark, a combination of a quotation mark and a period, a combination of a quotation mark and an exclamation mark, a combination of a quotation mark and a question mark, etc. The separator can be other symbols in a sentence except the sentence ending symbol, such as a comma, a colon, a semicolon, etc.
[0047] In an exemplary embodiment, after the above step 101, the method further includes: splicing the training samples whose length is less than a preset cutoff length in the to-be-processed document to obtain a target document; wherein the length of each target training sample in the target document is greater than or equal to the preset cutoff length. The corresponding implementation of the above step 102 may include: splitting each target training sample in the target document to obtain at least one sentence corresponding to each target training sample.
[0048] Specifically, for each document to be processed, the shorter training samples in the document to be processed can be merged to form a new document, i.e., the target document. The shorter training samples are training samples whose length is less than a preset cutoff length, and the preset cutoff length can be the maximum cutoff length supported by the graphics card. The length here can be understood as the number of subwords. For example, the length of the training sample can be the number of all subwords obtained after the training sample is segmented.
[0049] In this embodiment, all training samples in the document to be processed can be traversed, and training samples whose length is less than the maximum cutoff length can be spliced, so that the length of the spliced training samples is greater than or equal to the maximum cutoff length. The training samples in the target document, i.e., the target training samples, are all greater than or equal to the preset cutoff length.
[0050] In an exemplary embodiment, the above-mentioned splicing of training samples in the document to be processed whose length is less than a preset cutoff length to obtain a target document includes: obtaining documents of the same category as the document to be processed; extracting samples to be spliced from training samples of documents of the same category, and splicing the samples to be spliced with training samples in the document to be processed whose length is less than the preset cutoff length to obtain the target document.
[0051] Specifically, all training samples in the document to be processed can be traversed, and for the training samples whose length is less than the preset cutoff length, samples to be spliced are extracted from the same category of the document to be processed, and the extracted samples to be spliced are spliced with the training samples whose length is less than the preset cutoff length in the document to be processed, and finally a new document is formed, that is, the target document n1 = {d1, d2, ..., dn}. Among them, d1, d2, ..., dn are n target training samples in the target document n1, and the length of each target training sample is greater than or equal to the preset cutoff length.
[0052] In this embodiment, since documents of the same category are more similar in semantics, when splicing training samples, the samples to be spliced are found from documents of the same category as the document to be processed, so that the possibility of semantic coherence of the spliced training samples is also greater.
[0053] After concatenating the training samples and obtaining the target document, each target training sample in the target document can be split by sentence to obtain all the sentences corresponding to each target training sample, and finally form a di = {s1,...,sn}. Among them, di is the i-th target training sample, s1 represents the first sentence in the i-th target training sample, and sn represents the n-th sentence in the i-th target training sample, that is, the last sentence in the i-th target training sample.
[0054] After splitting each target training sample into multiple individual sentences, that is, obtaining di = {s1,...,sn}, the length of each sentence can be calculated. For example, the sentence can be segmented and the token length of the sentence can be counted. Optionally, the vocabulary of the language model to be trained can be used, and the vocabulary can be loaded using the sentencespiece word open source toolkit to segment the sentence into subwords (tokens), and the number of tokens can be counted and recorded as ti. Record the token length ti of each sentence in each document. For each target training sample, maintain a sentence length list T = [t1,...,tn], where t1 is the length of the first sentence in each target training sample, and tn is the length of the nth sentence in each target training sample, that is, the length of the last sentence in each target training sample.
[0055] After obtaining the length of each split sentence, it can be determined for each sentence whether the length of the sentence is greater than or less than the preset cutoff length. If the length of the sentence is greater than the preset cutoff length, the process proceeds to step 103; if the length of the sentence is less than the preset cutoff length, the process proceeds to step 104; if the length of the sentence is equal to the preset cutoff length, it means that the length of the sentence meets the standard and does not need to be truncated or spliced.
[0056] In step 103, if the length of the sentence is greater than the preset cutoff length, the sentence is segmented according to the separator in the sentence to obtain a plurality of clauses corresponding to the sentence; wherein the length of each clause is less than or equal to the preset cutoff length. The separator may be other symbols in a sentence except the sentence end symbol, such as a comma, a semicolon, or a place where a semantic pause may occur in a single sentence. The sentence is segmented according to the separator in the sentence, that is, the separator in the sentence is used as a segmentation point to segment the sentence, so as to segment a single sentence into a plurality of clauses whose length is less than the preset cutoff length.
[0057] For example, a sentence with a length of 11 is represented as [abcde, fghjk. ], and the preset cutoff length is 8. Then the delimiter "," is used as the segmentation point to segment the sentence into [abcde,] and [fghjk. ]
[0058] In an exemplary embodiment, the sentence is segmented according to the separators in the sentence to obtain multiple clauses corresponding to the sentence, including the following S1 to S4:
[0059] S1: Segment the sentence according to the preset cutoff length to obtain multiple segmentation results.
[0060] Specifically, the sentence can be segmented using the preset cutoff length as the segmentation length to obtain multiple segmentation results after the sentence is segmented. If the multiple segmentation results are m, then at least m-1 segmentation results have a length equal to the preset cutoff length, and one segmentation result has a length less than or equal to the preset cutoff length. The separator is retained in the segmentation results.
[0061] For example, as in the above example, a sentence with a length of 11 is represented as [abcde, fghjk. ], and the preset cutoff length is 8. Then the sentence is segmented according to the preset cutoff length, and two segmentation results are obtained: [abcde, fg], [hjk. ].
[0062] S2: For a target segmentation result with a separator among the multiple segmentation results, segment the target segmentation result according to the last separator in the target segmentation result.
[0063] Specifically, in this embodiment, the segmentation result with a separator in the multiple segmentation results is used as the target segmentation result. Then, the target segmentation result is segmented according to the last separator in the target segmentation result as the segmentation point to obtain two segmentation sub-results with the last separator as the segmentation boundary.
[0064] For example, in the above example, [abcde, fg] is the target segmentation result, then the target segmentation result is segmented according to the last separator in the target segmentation result to obtain [abcde,] and [fg].
[0065] Exemplarily, if the target segmentation result does not exist in the multiple segmentation results, the multiple segmentation results can be directly used as multiple clauses corresponding to the sentence.
[0066] S3: The first part of the subwords before the last separator in the target segmentation result is taken as the clause corresponding to the sentence, and the second part of the subwords after the last separator in the target segmentation result is merged with the next segmentation result after the target segmentation result to obtain a merged segmentation result.
[0067] For example, the target segmentation result is the above-mentioned [abcde, fg], the first part of the subwords [abcde,] before the last separator in the target segmentation result is taken as the clause corresponding to the sentence, and the second part of the subwords [fg] after the last separator in the target segmentation result is merged with the next segmentation result [hjk. ] after the target segmentation result to obtain the merged segmentation result [fghjk. ].
[0068] S4: If the length of the merged segmentation result is less than or equal to the preset cutoff length, the merged segmentation result is used as the clause corresponding to the sentence.
[0069] For example, if the length of the merged segmentation result [fghjk. ] is less than or equal to the preset cutoff length 8, [fghjk. ] can be directly used as the clause corresponding to the sentence. In other words, the two clauses obtained by segmenting the sentence [abcde, fghjk. ] are [abcde, ] and [fghjk. ] respectively.
[0070] S5: If the length of the merged segmentation result is greater than the preset cutoff length, the merged segmentation result is used as a sentence to continue segmenting the sentence according to the preset cutoff length to obtain multiple segmentation results until multiple clauses corresponding to the sentence are obtained.
[0071] For example, assuming that the sentence is [abcde, fghjklmnopq. ], the sentence is segmented according to the preset cutoff length 8, and the multiple segmentation results obtained are: [abcde, fg], [hjklmnop], [q. ] respectively.
[0072]
abcde, fg
abcde,
fg
abcde,
fg
hjklmnop
fghjklmnop
fghjklmnop
fghjklmnop
fghjklmn
op
fghjklmn
op
q.
opq.
opq.
abcde, fghjklmnopq.
abcde,
fghjklmn
opq.
[0073] In step 104, if the length of the sentence is less than the preset cut-off length, then the sentence is concatenated with other sentences in the training sample except the sentence to obtain a concatenated sentence with a length less than or equal to the preset cut-off length and semantic coherence. Among them, a semantically coherent concatenated sentence means that when performing sentence concatenation, it is not randomly concatenated, but continuous sentences with this sentence are searched before and after the sentence for concatenation to ensure that the obtained concatenated sentence is semantically coherent.
[0074] Specifically, if the length of the sentence is less than the preset cut-off length, it means that the sentence is a short sentence. Therefore, the short sentences can be concatenated to obtain a concatenated sentence with a length equal to the preset cut-off length and semantic coherence, or a concatenated sentence with a length as close as possible to the preset cut-off length and semantic coherence. In specific implementation, the next sentence in the same document can be concatenated through a greedy algorithm. The goal is to ensure that each training sample is a complete sentence and, if possible, ensure that each sample contains as many sentences as possible.
[0075] Exemplarily, the greedy algorithm can be as follows:
[0076] max_length # maximum cut-off length
[0077] i = j = 0 # record the starting and ending points of the number of sentences contained in each training sample, corresponding to the first and second values of ans below
[0078] R = [] # used to record which sentences are included in the final sample
[0079] while i < len(T): # T is the above T =
t1,..., tn
[0080] ans = [0, 0] # Record the start and end points of the number of sentences contained in each training sample. For example, [0, 2] means the sample contains the first and second sentences, and [3, 5] means the 4th and 5th sentences.
[0081] sums = T[i] # Record the current length of the training sample
[0082] j = i + 1 #
[0083] while j < len(T) and sums + T[j] <= max_length: # Exit when the total length of the sentences accumulated in the sample exceeds the maximum cut-off length
[0084] sums += T[j] # The total length of the sentences accumulated in the current sample
[0085] j += 1
[0086] k = j # After exiting when the total length of the sentences accumulated in the sample exceeds the maximum cut-off length, record the end point of the sample at this time (excluding this point)
[0087] ans[1] = j # Update the recorded end point to the end position of ans
[0088] j = i # Go back from the start point
[0089] while j - 1 >= 0 and sums + T[j - 1] <= max_length: # Enter the loop when the number of segments of the training sample is greater than 1. Go back from the start point until the total length of the sentences accumulated in the sample exceeds the maximum cut-off length and then exit
[0090] sums += T[j - 1] # The total length of the sentences accumulated in the current sample
[0091] j -= 1 # Go back from the start point
[0092] ans[0] = j
[0093] R.append(ans)
[0094] i = k #
[0095] return result # Left-closed right-open
[0096] In an exemplary embodiment, the above-mentioned sentence concatenation of the sentence with other sentences in the training sample except the sentence to obtain a concatenated sentence having a length less than or equal to the preset cutoff length and semantically coherent includes: determining a target sentence for concatenation with the sentence from other sentences based on the length of the sentence and the lengths of the other sentences; wherein the target sentence and the sentence belong to multiple consecutive sentences; and concatenating the sentence with the target sentence to obtain a concatenated sentence having a length less than or equal to the preset cutoff length and semantically coherent.
[0097] Specifically, candidate sentences that can still be semantically coherent after being spliced with the sentence can be determined from other sentences, and then the target sentence can be screened out from the candidate sentences according to the length of the candidate sentences and the length of the sentence. The target sentence and the sentence belong to multiple consecutive sentences, and the target sentence may be one or more, and the target sentence is located before or after the sentence. The length of the spliced sentence obtained after the target sentence and the sentence are spliced is less than or equal to the preset cutoff length and is semantically coherent.
[0098] In an exemplary embodiment, the above-mentioned determining a target sentence for splicing with the sentence from among the other sentences based on the length of the sentence and the length of the other sentences includes: if the sentence is the first sentence in the training sample, determining a target sentence for splicing with the sentence from among the other sentences after the sentence based on the length of the sentence and the length of the other sentences after the sentence; if the sentence is not the first sentence in the training sample, determining a target sentence for splicing with the sentence from among the other sentences before the sentence based on the length of the sentence and the length of the other sentences before the sentence. The clauses obtained after the above-mentioned sentence segmentation can also participate in sentence splicing as target sentences if they meet the conditions for becoming target sentences.
[0099] Specifically, when a sentence is the first sentence in a training sample, a target sentence for splicing with the sentence is determined from other sentences after the sentence according to the length of the sentence and the lengths of other sentences after the sentence. The target sentence may be one sentence after the sentence or multiple sentences. That is, when a sentence is the first sentence in a training sample, the target sentence is searched for from other sentences after the sentence. The sum of the lengths of the target sentence and the first sentence in the training sample is less than or equal to a preset cutoff length, and the sum of the lengths of the target sentence and the next sentence of the target sentence and the first sentence in the training sample is greater than the preset cutoff length. That is, the sum of the lengths of the target sentence and the first sentence in the training sample has reached a length that is closest to or equal to the preset cutoff length. And since the target sentence and the first sentence in the training sample are continuous sentences, their semantics after splicing are also relatively coherent.
[0100] Specifically, when the sentence is not the first sentence in the training sample, a target sentence for splicing with the sentence is determined from other sentences before the sentence according to the length of the sentence and the length of other sentences before the sentence. That is, when the sentence is not the first sentence in the training sample, the target sentence is searched from other sentences before the sentence, so as to maximize the number of spliced sentences and to ensure that the length of the spliced sentences is long enough under the condition of being less than or equal to the preset cutoff length, and the spliced sentences can also be semantically coherent.
[0101] To make it easier to understand, let's take an example as follows:
[0102] Suppose that a training sample of a long text is segmented. When the first segmented sample (i.e. the first sentence in the training sample) exceeds the maximum cutoff length, the first segmented sample is saved. If the second segmented sample (i.e. the second sentence in the training sample) is not enough for the maximum cutoff length, the sentences contained in the first segmented sample are added back, so as to make each sample contain as many sentences as possible and the sample length as long as possible.
[0103] For example, when T = [20, 20, 40, 50] (indicates that the first sentence has 20 tokens, that is, the length of the first sentence is 20, the second sentence has 30 tokens, that is, the length of the second sentence is 30, the length of the third sentence is 40, and the length of the fourth sentence is 50), and the maximum cutoff length is 60, the segmented samples, that is, the concatenated sentences, are as follows:
[0104] R1=[0, 1] means that the first segmented sample, i.e., the first concatenated sentence, consists of the first and second sentences, with a length of 20+20;
[0105] R2=[1, 2] means that the second segmented sample, i.e., the second concatenated sentence, consists of the second and third sentences, with a length of 20+40;
[0106] R3=[3] indicates that the third segmented sample, i.e., the third concatenated sentence, is composed of the fourth sentence and has a length of 50.
[0107] In an exemplary embodiment, a flowchart of the text processing method can be found in Figure 2 ,include:
[0108] Step 201: Input multiple documents containing multiple training samples.
[0109] Step 202: Classify multiple documents containing multiple training samples, such as subject classification, mathematics, English, Chinese and other categories.
[0110] Step 203: Merge the shorter training samples to obtain the target document.
[0111] That is to say, for training samples whose length is less than the preset cutoff length, training samples are extracted from other documents of the same category for splicing, and finally a new document is formed, namely the target document n1 = {d1, d2, ..., dn}.
[0112] Step 204: segment each target training sample di in the target document into sentences.
[0113] Specifically, the sample can be segmented by the sentence ending symbol, while the separator should be retained to ensure the integrity of the sentence. The punctuation marks that indicate the end of a sentence are: period; exclamation mark; question mark; combination of quotation mark and period; combination of quotation mark and exclamation mark; combination of quotation mark and question mark, finally forming a sentence di = {s1,...,sn}
[0114] Step 205: Segment the sentence and count the token length of the sentence.
[0115] Specifically, you can use the vocabulary of the language model to be trained, load the vocabulary using the sentencespiece open source word toolkit, segment the sentence into subwords (tokens), and count the number of tokens as ti. Record the token length of each sentence in each document. Maintain a list T = [t1,...,tn].
[0116] Step 206: Determine whether the length of a single sentence is greater than a preset cutoff length, if so, proceed to step 207, otherwise proceed to step 208.
[0117] Step 207: Segment the sentence according to the maximum cutoff length, and segment the sentence at the last separator of the segmentation result.
[0118] For example, the sentence is [abcde, fghjk. ], assuming the maximum cutoff length is 8, it is segmented into: [abcde, fg], [hjk. ] The above segmentation results are recursively split according to the last separator of the first segmentation result: [abcde,] The remaining subwords are merged with the next segment [fghjk. ], and this is repeated recursively.
[0119] Step 208: Concatenate the single sentence with the target sentence of the same document through a greedy strategy.
[0120] The goal of splicing is to ensure that each training sample is a complete sentence and to ensure that each sample contains as many sentences as possible.
[0121] Step 209: Finally, multiple training texts are obtained based on the single long text.
[0122] The above-mentioned single long text is the target training sample. After sentence segmentation and sentence concatenation of the target training sample, the clauses or concatenated sentences having a length less than or equal to a preset cutoff length are obtained to form the above-mentioned multiple training texts.
[0123] Through steps 201 to 209, the training samples in the document can be merged into short texts and segmented into long texts, so that each training sample is a complete sentence and it is possible to ensure that each sample contains as many sentences as possible.
[0124] In summary, the pre-training data processing method based on the long text segmentation strategy proposed in this application mainly solves two problems: First, under the premise that each training sample is a complete sentence as much as possible, how to cut the long text so that the number of text segments is the least and the length of each training sample is as long as possible. Specifically, by adopting the above-mentioned long text segmentation strategy to divide the training samples, the integrity of the sentences can be guaranteed as much as possible, and the length of each training sample can be made as long as possible through a greedy splicing strategy. Second, in order to make the training more efficient, how to merge short text sentences to reach a preset cutoff length. Specifically, by reasonably splicing the short text sentences without affecting the semantic coherence and reaching the preset cutoff length.
[0125] Figure 3 It is a structural diagram of a text processing device provided in an embodiment of the present application.
[0126] For example, Figure 3 As shown, the device includes: an acquisition module 301, used for acquiring a document to be processed; wherein the document to be processed contains multiple training samples; a splitting module 302, used for splitting each of the training samples in the document to be processed to obtain at least one sentence corresponding to each of the training samples; a segmentation module 303, used for, if the length of the sentence is greater than a preset cutoff length, segmenting the sentence according to the separator in the sentence to obtain multiple clauses corresponding to the sentence; wherein the length of each of the clauses is less than or equal to the preset cutoff length; a splicing module 304, used for, if the length of the sentence is less than the preset cutoff length, splicing the sentence with other sentences in the training sample except the sentence to obtain a spliced sentence with a length less than or equal to the preset cutoff length and semantic coherence.
[0127] In a possible implementation, the device also includes: a target document acquisition module, used to splice training samples in the document to be processed whose length is less than a preset cutoff length to obtain a target document; wherein the length of each target training sample in the target document is greater than or equal to the preset cutoff length; a splitting module 302, specifically used to split each of the target training samples in the target document to obtain at least one sentence corresponding to each of the target training samples.
[0128] In one possible implementation, the target document acquisition module is specifically used to obtain documents of the same category as the document to be processed; extract samples to be spliced from training samples of the documents of the same category, and splice the samples to be spliced with training samples in the document to be processed whose length is less than a preset cutoff length to obtain the target document.
[0129] In a possible implementation, the splitting module 302 is specifically configured to split each of the training samples in the to-be-processed document according to sentence terminators to obtain at least one sentence corresponding to each of the training samples, and retain the separators in the at least one sentence.
[0130] In a possible implementation, the segmentation module 303 is specifically used to: segment the sentence according to the preset cutoff length to obtain multiple segmentation results; for a target segmentation result with a separator in the multiple segmentation results, segment the target segmentation result according to the last separator in the target segmentation result; use the first part of subwords in the target segmentation result before the last separator as the clause corresponding to the sentence, and merge the second part of subwords in the target segmentation result after the last separator with the next segmentation result after the target segmentation result to obtain a merged segmentation result; if the length of the merged segmentation result is less than or equal to the preset cutoff length, use the merged segmentation result as the clause corresponding to the sentence; if the length of the merged segmentation result is greater than the preset cutoff length, use the merged segmentation result as the sentence to continue to perform the step of segmenting the sentence according to the preset cutoff length to obtain multiple segmentation results until multiple clauses corresponding to the sentence are obtained.
[0131] In one possible implementation, the splicing module 304 is specifically used to: determine a target sentence for splicing with the sentence in the other sentences according to the length of the sentence and the length of the other sentences; wherein the target sentence and the sentence belong to a plurality of consecutive sentences; and splice the sentence and the target sentence to obtain a spliced sentence having a length less than or equal to the preset cutoff length and semantically coherent.
[0132] In a possible implementation, the splicing module 304 is specifically used to: when the sentence is the first sentence in the training sample, determine a target sentence for splicing with the sentence from other sentences after the sentence based on the length of the sentence and the lengths of other sentences after the sentence; when the sentence is not the first sentence in the training sample, determine a target sentence for splicing with the sentence from other sentences before the sentence based on the length of the sentence and the lengths of other sentences before the sentence.
[0133] Figure 4 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application.
[0134] For example, Figure 4 As shown, the electronic device includes: a memory 401 and a processor 402, wherein the memory 401 stores executable program codes, and the processor 402 is used to call and execute the executable program codes to perform a text processing method.
[0135] In this embodiment, the electronic device can be divided into functional modules according to the above method example. For example, each functional module can be corresponded to, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware. It should be noted that the division of modules in this embodiment is schematic and is only a logical function division. There may be other division methods in actual implementation.
[0136] In the case of dividing each functional module according to each function, the electronic device may include: an acquisition module, a splitting module, a segmentation module, a splicing module, etc. It should be noted that all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module, which will not be repeated here.
[0137] The electronic device provided in this embodiment is used to execute the above-mentioned text processing method, and thus can achieve the same effect as the above-mentioned implementation method.
[0138] In the case of an integrated unit, the electronic device may include a processing module and a storage module. The processing module may be used to control and manage the actions of the electronic device. The storage module may be used to support the electronic device in executing mutual program codes and data.
[0139] The processing module may be a processor or a controller, which may implement or execute various exemplary logic blocks, modules and circuits represented in combination with the contents disclosed in this application. The processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, etc. The storage module may be a memory.
[0140] This embodiment also provides a computer-readable storage medium, in which a computer program code is stored. When the computer program code is executed on a computer, the computer executes the above-mentioned related method steps to implement a text processing method in the above-mentioned embodiment.
[0141] This embodiment also provides a computer program product. When the computer program product runs on a computer, it enables the computer to execute the above-mentioned related steps to implement a text processing method in the above-mentioned embodiment.
[0142] In addition, the electronic device provided in the embodiments of the present application may specifically be a chip, a component or a module, and the electronic device may include a connected processor and a memory; wherein the memory is used to store instructions, and when the electronic device is running, the processor may call and execute the instructions so that the chip executes a text processing method in the above-mentioned embodiments.
[0143] Among them, the electronic device, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above and will not be repeated here.
[0144] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.
[0145] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0146] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A method for text processing, characterized in that: The method comprises: Obtaining a document to be processed; wherein the document to be processed includes multiple training samples; Splitting each of the training samples in the document to be processed to obtain at least one sentence corresponding to each of the training samples; If the length of the sentence is greater than the preset cutoff length, the sentence is segmented according to the separator in the sentence to obtain a plurality of clauses corresponding to the sentence; wherein the length of each of the clauses is less than or equal to the preset cutoff length; If the length of the sentence is less than the preset cutoff length, the sentence is concatenated with other sentences in the training sample except the sentence to obtain a concatenated sentence whose length is less than or equal to the preset cutoff length and has semantic coherence.
2. The method according to claim 1, characterized in that After obtaining the document to be processed, the method further includes: splicing the training samples whose lengths in the to-be-processed document are less than a preset cutoff length to obtain a target document; wherein the length of each target training sample in the target document is greater than or equal to the preset cutoff length; The step of splitting each of the training samples in the document to be processed to obtain at least one sentence corresponding to each of the training samples includes: Each target training sample in the target document is split to obtain at least one sentence corresponding to each target training sample.
3. The method according to claim 2, characterized in that The step of splicing the training samples whose lengths in the to-be-processed documents are less than a preset cutoff length to obtain a target document comprises: Acquire documents of the same category as the document to be processed; A sample to be spliced is extracted from the training samples of the same category of documents, and the sample to be spliced is spliced with a training sample whose length is less than a preset cutoff length in the document to be processed to obtain a target document.
4. The method according to claim 1, characterized in that: The step of splitting each of the training samples in the document to be processed to obtain at least one sentence corresponding to each of the training samples includes: Each training sample in the to-be-processed document is split according to the sentence terminator to obtain at least one sentence corresponding to each training sample, and the separator in the at least one sentence is retained.
5. The method according to claim 1, characterized in that The sentence is segmented according to the separator in the sentence to obtain a plurality of clauses corresponding to the sentence, including: Segmenting the sentence according to the preset cutoff length to obtain multiple segmentation results; For a target segmentation result having a separator among the multiple segmentation results, segmenting the target segmentation result according to the last separator in the target segmentation result; The first part of the subwords in the target segmentation result before the last separator is used as the clause corresponding to the sentence, and the second part of the subwords in the target segmentation result after the last separator is merged with the next segmentation result after the target segmentation result to obtain a merged segmentation result; If the length of the merged segmentation result is less than or equal to the preset cutoff length, taking the merged segmentation result as the clause corresponding to the sentence; If the length of the merged segmentation result is greater than the preset cutoff length, the merged segmentation result is used as the sentence to continue the step of segmenting the sentence according to the preset cutoff length to obtain multiple segmentation results until multiple clauses corresponding to the sentence are obtained.
6. The method according to claim 1, characterized in that The sentence concatenation is performed on the sentence and other sentences in the training sample except the sentence to obtain a concatenated sentence whose length is less than or equal to the preset cutoff length and has semantic coherence, including: Determining a target sentence for concatenation with the sentence from among the other sentences according to the length of the sentence and the length of the other sentences; wherein the target sentence and the sentence belong to a plurality of consecutive sentences; The sentence and the target sentence are concatenated to obtain a concatenated sentence whose length is less than or equal to the preset cutoff length and whose semantics are coherent.
7. The method according to claim 6, characterized in that The step of determining, in the other sentences according to the length of the sentence and the length of the other sentences, a target sentence for concatenation with the sentence comprises: In the case where the sentence is the first sentence in the training sample, determining a target sentence for concatenation with the sentence from among the other sentences after the sentence according to the length of the sentence and the lengths of other sentences after the sentence; In the case that the sentence is not the first sentence in the training sample, a target sentence for concatenation with the sentence is determined from other sentences before the sentence according to the length of the sentence and the lengths of other sentences before the sentence.
8. A text processing device, characterized in that: The device comprises: An acquisition module, used to acquire a document to be processed; wherein the document to be processed includes a plurality of training samples; A splitting module, used for splitting each of the training samples in the document to be processed to obtain at least one sentence corresponding to each of the training samples; A segmentation module, configured to segment the sentence according to the separator in the sentence if the length of the sentence is greater than a preset cutoff length, to obtain a plurality of clauses corresponding to the sentence; wherein the length of each of the clauses is less than or equal to the preset cutoff length; A concatenation module is used to concatenate the sentence with other sentences in the training sample except the sentence if the length of the sentence is less than the preset cutoff length, so as to obtain a concatenated sentence whose length is less than or equal to the preset cutoff length and has semantic coherence.
9. An electronic device, characterized in that: The electronic device comprises: A memory for storing executable program codes; A processor, configured to call and run the executable program code from the memory, so that the electronic device executes the method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed, the method according to any one of claims 1 to 7 is implemented.