A text processing method, device, equipment and storage medium

Abstract

The present disclosure provides a text processing method, apparatus, device and storage medium, which encode and process a plurality of target words and the number of target words divided from sentences separated by paragraph delimiters selected from a text to be processed to obtain a sentence vector, and judge whether two sentences can be merged into the same sentence according to the sentence merging recognition result corresponding to the sentence vector, so as to learn the relationship between the two sentences, and thus learn the connection between the two paragraphs separated by paragraph delimiters, so as to judge whether the two paragraphs can be merged, and improve the accuracy of the sentence merging recognition result, thereby improving the efficiency and accuracy of paragraph merging, and can reduce the influence of too many paragraphs generated by a large number of paragraph delimiters on text analysis, which is conducive to improving the running speed of text analysis.

Description

Technical Field

[0001] The present disclosure relates to the field of natural language processing, and in particular, to a text processing method, apparatus, device and storage medium. Background Art

[0002] In the field of natural language processing, when acquiring text for analysis, we often encounter the problem of too many paragraphs in the text due to system differences, format differences after article transcoding, and the writer adding a large number of paragraph separators as the basis for paragraph division in order to make the article content more hierarchical and beautiful.

[0003] Text analysis models often analyze text data based on paragraph delimiters in the text as the basis for paragraph segmentation. Therefore, too many paragraph delimiters will result in too many paragraphs, which will also increase the computational complexity of the text analysis model and reduce the running speed of the text analysis model and the efficiency of text processing. Summary of the invention

[0004] The embodiments of the present disclosure at least provide a text processing method and apparatus, device, and storage medium, which can solve at least one of the above problems.

[0005] The present disclosure provides a text processing method, the method comprising:

[0006] Determine at least one group of sentences to be merged in the text to be processed, wherein the sentences to be merged include a first sentence and a second sentence that are adjacent to each other, and the first sentence and the second sentence are located in different paragraphs of the text to be processed;

[0007] Determine a first sentence vector corresponding to the first sentence and a second sentence vector corresponding to the second sentence;

[0008] Inputting the first sentence vector and the second sentence vector into a trained paragraph merging recognition model to obtain a sentence merging recognition result;

[0009] If the sentence merging recognition result indicates that the first sentence and the second sentence meet a merging condition, the paragraph where the first sentence is located is merged with the paragraph where the second sentence is located.

[0010] In an optional implementation manner, determining at least one group of sentences to be merged in the text to be processed includes:

[0011] Get text data of the text to be processed;

[0012] According to the paragraph separator identified from the text data, determining a first paragraph adjacent to the paragraph separator and located before the paragraph separator and a second paragraph located after the paragraph separator in the text to be processed;

[0013] According to a preset sentence identification rule, determining a first sentence located at the last sentence position in the first paragraph and a second sentence located at the first sentence position in the second paragraph, wherein the first sentence and the second sentence each include at least one natural sentence;

[0014] The first statement and the second statement are determined as a group of statements to be merged in at least one group of statements to be merged.

[0015] In an optional implementation manner, determining a first sentence vector corresponding to the first sentence and a second sentence vector corresponding to the second sentence includes:

[0016] Obtaining a word segmentation method for the text to be processed;

[0017] According to the word segmentation method, the first sentence and the second sentence are respectively segmented to obtain a plurality of first target words corresponding to the first sentence and the number of the plurality of first target words, and a plurality of second target words corresponding to the second sentence and the number of the plurality of second target words;

[0018] encoding the first sentence according to the plurality of first target words and the number of the plurality of first target words to obtain a first sentence vector corresponding to the first sentence;

[0019] The second sentence is encoded according to the multiple second target words and the number of the multiple second target words to obtain a second sentence vector corresponding to the second sentence.

[0020] In an optional implementation manner, the inputting the first sentence vector and the second sentence vector into a trained paragraph merging recognition model to obtain a sentence merging recognition result includes:

[0021] Inputting the first sentence vector and the second sentence vector into the semantic recognition layer in the trained paragraph merging recognition model to obtain a semantic distance vector between the first sentence vector and the second sentence vector;

[0022] The semantic distance vector is input into a merge recognition layer in the paragraph merge recognition model to obtain a sentence merge recognition result for the first sentence and the second sentence, wherein the sentence merge recognition result includes a sentence merge probability.

[0023] In an optional implementation, if the sentence merging recognition result indicates that the first sentence and the second sentence meet a merging condition, merging the paragraph where the first sentence is located with the paragraph where the second sentence is located includes:

[0024] If the statement merging probability is greater than a preset probability threshold, determining that the first statement and the second statement meet a merging condition;

[0025] When the first sentence and the second sentence meet the merging condition, the paragraph where the first sentence is located is merged with the paragraph where the second sentence is located.

[0026] In an optional implementation, the trained paragraph merging recognition model is obtained by the following steps:

[0027] Acquire multiple sample sentence groups identified from the sample text and a true sample merge label of each sample sentence group, wherein the sample sentence group includes a first sample sentence and a second sample sentence;

[0028] Determine a first sample vector corresponding to the first sample sentence and a second sample vector corresponding to the second sample sentence;

[0029] The constructed neural network is trained using the first sample vector, the second sample vector and the real sample merge label until the neural network meets the preset training conditions, thereby obtaining a trained paragraph merge recognition model.

[0030] In an optional implementation, determining whether the neural network satisfies the preset training condition is performed by the following steps:

[0031] In the process of training the neural network, based on the estimated merged result of the neural network for the test sample and the actual merged label of the test sample, a model training index of the neural network is determined, wherein the model training index includes at least one of model accuracy, model precision and model recall;

[0032] When the model training index is greater than a preset index threshold, it is determined that the neural network meets the preset training condition, and the trained neural network is used as the trained paragraph merging recognition model.

[0033] In an optional implementation, before obtaining a plurality of sample sentence groups identified from the sample text and a real sample merged label for each sample sentence group, the method includes:

[0034] Determining the contents of multiple paragraphs of text in the sample text;

[0035] Identifying a plurality of positive sample sentence groups from the plurality of textual contents, wherein the positive sample sentence groups include two adjacent sentences in the same paragraph;

[0036] Identifying a plurality of negative sample sentence groups from the plurality of textual contents, wherein the negative sample sentence groups include two sentences located in different paragraphs;

[0037] The identified multiple groups of positive sample sentence groups and the real sample merged labels of each positive sample sentence group, as well as the identified multiple groups of negative sample sentence groups and the real sample merged labels of each negative sample sentence group, are used as the multiple sample sentence groups and the real sample merged labels of each sample sentence group for training the paragraph merged recognition model.

[0038] The present disclosure also provides a text processing device, the device comprising:

[0039] A sentence acquisition module, used to determine at least one group of sentences to be merged in the text to be processed, wherein the sentences to be merged include a first sentence and a second sentence that are adjacent to each other, and the first sentence and the second sentence are located in different paragraphs of the text to be processed;

[0040] A vector determination module, used to determine a first sentence vector corresponding to the first sentence and a second sentence vector corresponding to the second sentence;

[0041] A result recognition module, used for inputting the first sentence vector and the second sentence vector into a trained paragraph merging recognition model to obtain a sentence merging recognition result;

[0042] The paragraph merging module is configured to merge the paragraph where the first sentence is located with the paragraph where the second sentence is located if the sentence merging recognition result indicates that the first sentence and the second sentence meet a merging condition.

[0043] In an optional implementation manner, the statement acquisition module is specifically used to:

[0044] Get text data of the text to be processed;

[0045] According to the paragraph separator identified from the text data, determining a first paragraph adjacent to the paragraph separator and located before the paragraph separator and a second paragraph located after the paragraph separator in the text to be processed;

[0046] According to a preset sentence identification rule, determining a first sentence located at the last sentence position in the first paragraph and a second sentence located at the first sentence position in the second paragraph, wherein the first sentence and the second sentence each include at least one natural sentence;

[0047] The first statement and the second statement are determined as a group of statements to be merged in at least one group of statements to be merged.

[0048] In an optional implementation manner, the vector determination module is specifically used to:

[0049] Obtaining a word segmentation method for the text to be processed;

[0050] According to the word segmentation method, the first sentence and the second sentence are respectively segmented to obtain a plurality of first target words corresponding to the first sentence and the number of the plurality of first target words, and a plurality of second target words corresponding to the second sentence and the number of the plurality of second target words;

[0051] encoding the first sentence according to the plurality of first target words and the number of the plurality of first target words to obtain a first sentence vector corresponding to the first sentence;

[0052] The second sentence is encoded according to the multiple second target words and the number of the multiple second target words to obtain a second sentence vector corresponding to the second sentence.

[0053] In an optional implementation manner, the result identification module is specifically used to:

[0054] Inputting the first sentence vector and the second sentence vector into the semantic recognition layer in the trained paragraph merging recognition model to obtain a semantic distance vector between the first sentence vector and the second sentence vector;

[0055] The semantic distance vector is input into a merge recognition layer in the paragraph merge recognition model to obtain a sentence merge recognition result for the first sentence and the second sentence, wherein the sentence merge recognition result includes a sentence merge probability.

[0056] In an optional implementation manner, the paragraph merging module is specifically used for:

[0057] If the statement merging probability is greater than a preset probability threshold, determining that the first statement and the second statement meet a merging condition;

[0058] When the first sentence and the second sentence meet the merging condition, the paragraph where the first sentence is located is merged with the paragraph where the second sentence is located.

[0059] In an optional implementation, the device further includes a model training module, and the model training module is used to obtain a trained paragraph merging recognition model through the following steps:

[0060] Acquire multiple sample sentence groups identified from the sample text and a true sample merge label of each sample sentence group, wherein the sample sentence group includes a first sample sentence and a second sample sentence;

[0061] Determine a first sample vector corresponding to the first sample sentence and a second sample vector corresponding to the second sample sentence;

[0062] The constructed neural network is trained using the first sample vector, the second sample vector and the real sample merge label until the neural network meets the preset training conditions, thereby obtaining a trained paragraph merge recognition model.

[0063] In an optional implementation, the model training module determines whether the neural network meets the preset training condition by the following steps:

[0064] In the process of training the neural network, based on the estimated merged result of the neural network for the test sample and the actual merged label of the test sample, a model training index of the neural network is determined, wherein the model training index includes at least one of model accuracy, model precision and model recall;

[0065] When the model training index is greater than a preset index threshold, it is determined that the neural network meets the preset training condition, and the trained neural network is used as the trained paragraph merging recognition model.

[0066] In an optional implementation, the device further includes a sample acquisition module, and the sample acquisition module is specifically configured to:

[0067] Determining the contents of multiple paragraphs of text in the sample text;

[0068] Identifying a plurality of positive sample sentence groups from the plurality of textual contents, wherein the positive sample sentence groups include two adjacent sentences in the same paragraph;

[0069] Identifying a plurality of negative sample sentence groups from the plurality of textual contents, wherein the negative sample sentence groups include two sentences located in different paragraphs;

[0070] The identified multiple groups of positive sample sentence groups and the real sample merged labels of each positive sample sentence group, as well as the identified multiple groups of negative sample sentence groups and the real sample merged labels of each negative sample sentence group, are used as the multiple sample sentence groups and the real sample merged labels of each sample sentence group for training the paragraph merged recognition model.

[0071] The embodiment of the present disclosure also provides a computer device, including: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps in the above-mentioned text processing method are performed.

[0072] The embodiment of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned text processing method are executed.

[0073] The text processing method, apparatus, device and storage medium provided by the embodiments of the present disclosure determine at least one group of sentences to be merged in the text to be processed, wherein the sentences to be merged include adjacent first and second sentences, and the first and second sentences are separated by a paragraph separator in the text to be processed; according to a plurality of first target words divided from the first sentence and the number of the plurality of first target words, and a plurality of second target words divided from the second sentence and the number of the plurality of second target words, respectively determine a first sentence vector corresponding to the first sentence and a second sentence vector corresponding to the second sentence; input the first sentence vector and the second sentence vector into a trained paragraph merging recognition model to obtain a sentence merging recognition result; if the sentence merging recognition result indicates that the first sentence and the second sentence are contents of the same sentence, determine that the paragraph where the first sentence is located and the paragraph where the second sentence is located can be merged into one paragraph.

[0074] In this way, a sentence vector is obtained by encoding multiple target words and the number of target words divided from the sentences separated by paragraph separators selected from the text to be processed, and judging whether two sentences can be merged into the same sentence based on the sentence merging recognition result corresponding to the sentence vector. The relationship between the two sentences can be learned, and thus the connection between the two paragraphs separated by the paragraph separator can be learned, so as to judge whether the two paragraphs can be merged. The accuracy of the sentence merging recognition result can be improved, thereby improving the efficiency and accuracy of paragraph merging, and reducing the influence of too many paragraphs generated by a large number of paragraph separators on text analysis, which is conducive to improving the running speed of text analysis.

[0075] Furthermore, in the embodiments of the present disclosure, by selecting positive sample sentence groups and negative sample sentence groups from the sample text for training the paragraph merging recognition model, the relationship between sentences and the connection between paragraphs in the sample text can be accurately learned, which helps to improve the accuracy of the model in recognizing sentences in the processed text and improve the effectiveness of the paragraph merging recognition model.

[0076] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following is a brief introduction to the drawings required for use in the embodiments. The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used together with the specification to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can also be obtained based on these drawings without creative work.

[0078] Figure 1 A flowchart of a text processing method provided by an embodiment of the present disclosure;

[0079] Figure 2 A flowchart of another text processing method provided by an embodiment of the present disclosure;

[0080] Figure 3 A data processing schematic diagram of another text processing method provided by an embodiment of the present disclosure;

[0081] Figure 4 A schematic diagram of the text content to be processed obtained according to an embodiment of the present disclosure;

[0082] Figure 5 One of the structural diagrams of a text processing device provided by an embodiment of the present disclosure;

[0083] Figure 6 A second structural diagram of a text processing device provided by an embodiment of the present disclosure;

[0084] Figure 7 A schematic diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0085] In order to make the purpose, technical scheme and advantages of the embodiments of the present disclosure clearer, the technical scheme in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The components of the embodiments of the present disclosure generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the present disclosure for protection, but merely represents the selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present disclosure.

[0086] Research has found that when acquiring text for analysis, there are often too many paragraphs in the text due to system differences, differences in the format of the article after transcoding, and the fact that the writer adds a large number of paragraph separators as the basis for paragraph division in order to make the article content more hierarchical and beautiful. When analyzing text, the text analysis model often uses paragraph separators as the basis for paragraph segmentation. Therefore, too many paragraph separators will increase the computational complexity of the text analysis model and reduce the running speed of the text analysis model and the efficiency of processing text.

[0087] Based on the above research, an embodiment of the present disclosure provides a text processing method, which obtains a sentence vector by encoding multiple target words and the number of target words divided from sentences separated by paragraph separators selected from the text to be processed, and judges whether two sentences can be merged into the same sentence based on the sentence merging recognition result corresponding to the sentence vector. The relationship between the two sentences can be learned, and thus the connection between the two paragraphs separated by paragraph separators can be learned to determine whether the two paragraphs can be merged. The accuracy of the sentence merging recognition results can be improved, thereby improving the efficiency and accuracy of paragraph merging, and the impact of too many paragraphs generated by a large number of paragraph separators on text analysis can be reduced, which is conducive to improving the running speed of text analysis.

[0088] The defects existing in the above solutions are the results obtained by the inventor after practice and careful research. Therefore, the discovery process of the above problems and the solutions proposed by the present disclosure for the above problems below should be the contributions made by the inventor to the present disclosure during the disclosure process.

[0089] The technical solutions in the present disclosure will be clearly and completely described below in conjunction with the drawings in the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The components of the present disclosure generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the present disclosure claimed for protection, but merely represents the selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present disclosure.

[0090] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0091] To facilitate understanding of this embodiment, a text processing method disclosed in the embodiment of the present disclosure is first introduced in detail. The execution subject of the text processing method provided in the embodiment of the present disclosure is generally a computer device with certain computing capabilities, and the computer device includes, for example: a terminal device or a server or other processing device, and the terminal device can be a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. In some possible implementations, the web page consistency detection method can be implemented by a processor calling a computer-readable instruction stored in a memory.

[0092] The text processing method provided by the embodiment of the present disclosure is described below by taking the execution subject as a terminal device as an example.

[0093] See also Figure 1 , Figure 1 Flow chart of a text processing method provided by an embodiment of the present disclosure. Figure 1 As shown in , the text processing method provided by the embodiment of the present disclosure includes:

[0094] S101: Determine at least one group of sentences to be merged in a text to be processed, wherein the sentences to be merged include a first sentence and a second sentence that are adjacent to each other, and the first sentence and the second sentence are located in different paragraphs of the text to be processed.

[0095] In this step, when determining whether two different paragraphs of the text to be processed can be merged, at least one group of two adjacent sentences located in different paragraphs of the text to be processed can be selected from the text to be processed, and the two sentences can be used as sentences to be merged to determine whether the two paragraphs to which the sentences to be merged belong can be merged.

[0096] The text to be processed may be text data with multiple paragraphs in any format. For example, the any format may be the following formats: Portable Document Format (PDF) format or Text File (TXT) format.

[0097] When different paragraphs are identified in the text to be processed, at least one group of two adjacent sentences belonging to different paragraphs may be selected from the text to be processed as sentences to be merged.

[0098] S102: Determine a first sentence vector corresponding to the first sentence and a second sentence vector corresponding to the second sentence.

[0099] In this step, after the first sentence and the second sentence are obtained, in order to satisfy the input condition of the paragraph merging recognition model, a first sentence vector corresponding to the first sentence and a second sentence vector corresponding to the second sentence are determined.

[0100] S103: Input the first sentence vector and the second sentence vector into a trained paragraph merging recognition model to obtain a sentence merging recognition result.

[0101] In this step, after obtaining the first vector and the second vector, the first vector and the second vector are input into the trained paragraph merging recognition model as the sentence vectors to be merged, and accordingly a sentence merging recognition result corresponding to the sentence vectors to be merged will be obtained.

[0102] The paragraph merging recognition model is a model having a semantic recognition layer and a merging recognition layer.

[0103] S104: If the sentence merging recognition result indicates that the first sentence and the second sentence meet a merging condition, merge the paragraph where the first sentence is located with the paragraph where the second sentence is located.

[0104] In this step, after obtaining the sentence merging recognition result, it can be determined whether the first sentence and the second sentence meet the merging condition based on the sentence merging recognition result. If the sentence merging recognition result indicates that the first sentence and the second sentence meet the merging condition, it can be determined that the paragraph where the first sentence is located is merged with the paragraph where the second sentence is located.

[0105] The text processing method provided by the embodiment of the present disclosure obtains a sentence vector by encoding multiple target words and the number of target words divided from sentences separated by paragraph separators selected from the text to be processed, and determines whether two sentences can be merged into the same sentence based on the sentence merging recognition result corresponding to the sentence vector. The relationship between the two sentences can be learned, and thus the connection between the two paragraphs separated by paragraph separators can be learned to determine whether the two paragraphs can be merged. The accuracy of the sentence merging recognition result can be improved, thereby improving the efficiency and accuracy of paragraph merging, and the influence of too many paragraphs generated by a large number of paragraph separators on text analysis can be reduced, which is conducive to improving the running speed of text analysis.

[0106] See also Figure 2 , Figure 2 Flow chart of another text processing method provided by the embodiment of the present disclosure. Figure 2 As shown in , another text processing method provided by the embodiment of the present disclosure includes:

[0107] S201: Determine at least one group of sentences to be merged in a text to be processed, wherein the sentences to be merged include a first sentence and a second sentence that are adjacent to each other, and the first sentence and the second sentence are located in different paragraphs of the text to be processed.

[0108] S202: Determine a first sentence vector corresponding to the first sentence and a second sentence vector corresponding to the second sentence.

[0109] S203: Acquire multiple sample sentence groups identified from the sample text and a true sample merge label of each sample sentence group, wherein the sample sentence group includes a first sample sentence and a second sample sentence.

[0110] In this step, before using the trained paragraph merging recognition model, the paragraph merging recognition model needs to be trained, and accordingly, a sample sentence group and a real sample merging label for each sample sentence group need to be obtained to complete the model training, wherein the sample sentence group includes a first sample sentence and a second sample sentence.

[0111] S204: Determine a first sample vector corresponding to the first sample sentence and a second sample vector corresponding to the second sample sentence.

[0112] In this step, after obtaining the first sample sentence and the second sample sentence, a first sample vector corresponding to the first sample sentence and a second sample vector corresponding to the second sample sentence are further determined.

[0113] Among them, the method for obtaining the first sample vector corresponding to the first sample statement and the second sample vector corresponding to the second sample statement can refer to the method for obtaining the first statement vector and the second statement vector of the merged statement to be processed, and can achieve the same technical effect and solve the same technical problems, which will not be repeated here.

[0114] S205: Using the first sample vector, the second sample vector and the real sample merge label, the constructed neural network is trained until the neural network meets the preset training conditions, thereby obtaining a trained paragraph merge recognition model.

[0115] In this step, after obtaining the first sample vector, the second sample vector and the real sample merge label, they are input into the constructed neural network for training. For each group of the first sample vectors and each group of the second sample vectors, the corresponding first sample output results and second sample output results can be obtained. According to the first sample output results and the second sample output results, as well as the real sample merge label, the weight of the constructed neural network is adjusted until the neural network meets the preset training conditions, thereby obtaining a trained paragraph merging recognition model.

[0116] The preset training condition may be set according to the recognition accuracy of the neural network or the error value between the output result and the combined label of the real sample.

[0117] S206: Input the first sentence vector and the second sentence vector into the trained paragraph merging recognition model to obtain a sentence merging recognition result.

[0118] S207: If the sentence merging recognition result indicates that the first sentence and the second sentence meet a merging condition, merge the paragraph where the first sentence is located with the paragraph where the second sentence is located.

[0119] Among them, steps S201, S202, S206, and S207 can refer to the description of steps S101 to S104, and can achieve the same technical effects and solve the same technical problems, and will not be described in detail here.

[0120] Next, this embodiment is further described in conjunction with specific implementation methods.

[0121] In an optional implementation manner, step S201 includes:

[0122] Get the text data of the text to be processed.

[0123] In this step, the text to be processed may be obtained from one or more articles in different formats.

[0124] According to the paragraph separator identified from the text data, a first paragraph adjacent to the paragraph separator and located before the paragraph separator and a second paragraph located after the paragraph separator in the text to be processed are determined.

[0125] In this step, a first paragraph adjacent to the paragraph separator and located before the paragraph separator and a second paragraph adjacent to the paragraph separator and located after the paragraph separator can be determined according to the paragraph separator identified from the text data of the text to be processed.

[0126] According to preset sentence recognition rules, a first sentence located at the last sentence position in the first paragraph and a second sentence located at the first sentence position in the second paragraph are determined, wherein the first sentence and the second sentence each include at least one natural sentence.

[0127] In this step, according to the preset sentence recognition rules, the last sentence adjacent to the paragraph separator in the first paragraph is confirmed as the first sentence, and the first sentence adjacent to the paragraph separator in the second paragraph is confirmed as the second sentence, wherein the first sentence and the second sentence each include at least one natural sentence.

[0128] The sentence identification rules may include the following aspects: the first sentence and the second sentence may be divided according to the period, exclamation mark, etc. in the paragraph, and secondly, in the absence of the period, exclamation mark, etc., the first paragraph and the second paragraph may be regarded as a sentence respectively. In addition, in the actual writing process, due to the author's typos or writing style, a modal particle may be a separate sentence, or a single sentence may have very little information, and such sentences cannot be considered as independent natural sentences. Therefore, such sentences may be divided into one sentence with sentences with greater correlation before and after according to the actual situation, and then such sentences may be organized into independent natural sentences.

[0129] Exemplarily, according to the sentence recognition rule, when the "\n" character is identified from the text data of the text to be processed, the first paragraph of text and the second paragraph of text adjacent to the "\n" are determined according to the position of the "\n". If the first paragraph of text ends with a period, the two periods closest to the "\n" are determined, and the sentence between the two periods is the first sentence. If the second paragraph of text does not have a period, the second paragraph of text is used as the second sentence.

[0130] The first statement and the second statement are determined as a group of statements to be merged in at least one group of statements to be merged.

[0131] In this step, after the first statement and the second statement are determined, the first statement and the second statement are combined to generate a group of statements to be merged in at least one group of statements to be merged.

[0132] In an optional implementation manner, step S202 includes:

[0133] Obtain a word segmentation method for the text to be processed.

[0134] In this step, after obtaining at least one group of sentences to be merged in the text to be processed, different word segmentation methods of the sentences to be merged can be determined according to different attributes of the text to be processed.

[0135] The word segmentation method may be to perform word segmentation according to the attributes of the text to be processed.

[0136] Exemplarily, word segmentation may be performed based on properties such as the content of the selected text to be processed, the size of the text, and the field to which the text belongs.

[0137] Among them, the word segmentation method can be the following methods: a dictionary-based word segmentation algorithm, a statistics-based machine learning algorithm, wherein the dictionary-based word segmentation algorithm matches the character string to be matched with a word in a sufficiently large dictionary according to a certain strategy, and the statistics-based machine learning algorithm is to perform annotation training on Chinese characters.

[0138] According to the word segmentation method, the first sentence and the second sentence are respectively segmented to obtain multiple first target words corresponding to the first sentence and the number of the multiple first target words, as well as multiple second target words corresponding to the second sentence and the number of the second target words.

[0139] In this step, the first sentence and the second sentence are segmented according to the word segmentation method, so that multiple first target words corresponding to the first sentence and the number of the multiple first target words, as well as multiple second target words corresponding to the second sentence and the number of the second target words can be obtained.

[0140] Exemplarily, the sentence "the content of the article involves environmental improvement" is segmented to obtain: ['article', 'content', 'involving', 'to', 'environment', 'improvement'].

[0141] encoding the first sentence according to the plurality of first target words and the number of the plurality of first target words to obtain a first sentence vector corresponding to the first sentence;

[0142] The second sentence is encoded according to the multiple second target words and the number of the multiple second target words to obtain a second sentence vector corresponding to the second sentence.

[0143] In this step, based on the multiple first target words divided from the first sentence and the number of the multiple first target words, and based on the multiple second target words divided from the second sentence and the number of the multiple second target words, the first sentence and the second sentence are encoded to obtain a first sentence vector corresponding to the first sentence.

[0144] The encoding process can be implemented through a bag-of-words model or a word vector model.

[0145] In an optional implementation manner, step S206 includes:

[0146] The first sentence vector and the second sentence vector are input into the semantic recognition layer in the trained paragraph merging recognition model to obtain a semantic distance vector between the first sentence vector and the second sentence vector.

[0147] In this step, after obtaining the first sentence vector and the second sentence vector, they are input into the semantic recognition layer in the trained paragraph merging recognition model to obtain the semantic distance vector between the first sentence vector and the second sentence vector.

[0148] Wherein, the semantic recognition layer can be a fully connected neural network.

[0149] The semantic distance vector is used to calculate the similarity between two sentences.

[0150] Exemplarily, if the first sentence vector is represented as h1, the second sentence vector is represented as h2, and the semantic distance vector is represented as H, then the relationship between the three is: H=||h1-h2||1.

[0151] The semantic distance vector is input into a merge recognition layer in the paragraph merge recognition model to obtain a sentence merge recognition result for the first sentence and the second sentence, wherein the sentence merge recognition result includes a sentence merge probability.

[0152] In this step, after obtaining the semantic distance vector, the semantic distance vector is input into the merge recognition layer in the paragraph merge recognition model, and the semantic distance vector is processed to obtain a sentence merge recognition result for the first sentence and the second sentence, wherein the sentence merge recognition result includes a sentence merge probability.

[0153] The merged recognition layer is used to complete the normalization process of the semantic distance vector.

[0154] Exemplarily, the normalization processing function of the merged recognition layer can be an exponential function. For example, the semantic distance vector is represented as H, and the sentence merging probability is represented as Y. Then the normalization processing process of the semantic distance vector can be expressed as the following formula: Y=exp(-H).

[0155] See also Figure 3 , Figure 3 This is a data processing diagram of another text processing method provided by an embodiment of the present disclosure. Figure 3 As shown in , the first sentence and the second sentence obtained are respectively segmented, and then encoded according to the first target words and the number of the first target words of the first sentence, as well as the number of the second target words and the second target words of the second sentence obtained after the word segmentation, and the encoded first sentence vector and the second sentence vector are input into the semantic recognition layer, and finally the semantic distance vector is input into the merged recognition layer to obtain the sentence merged recognition result for the first sentence and the second sentence.

[0156] In an optional implementation manner, step S207 includes:

[0157] If the statement merging probability is greater than a preset probability threshold, determining that the first statement and the second statement meet a merging condition;

[0158] When the first sentence and the second sentence meet the merging condition, the paragraph where the first sentence is located is merged with the paragraph where the second sentence is located.

[0159] The preset probability threshold may be set as required, for example, different thresholds may be set according to the accuracy of the statement merging result.

[0160] Exemplarily, if the statement merging probability is greater than 0.5, it can be determined that the first statement and the second statement are the content of the same statement; if the statement merging probability is less than 0.5, it can be determined that the first statement and the second statement are not the content of the same statement.

[0161] If the first sentence and the second sentence are contents of the same sentence, it is determined that the paragraph where the first sentence is located and the paragraph where the second sentence is located can be merged into one paragraph.

[0162] In this step, when it is determined that the first sentence and the second sentence are contents of the same sentence, that is, the first sentence and the second sentence can be merged, it can be determined that the paragraph where the first sentence is located and the paragraph where the second sentence is located can be merged into one paragraph.

[0163] In an optional implementation, determining whether the neural network satisfies the preset training condition is performed by the following steps:

[0164] In the process of training the neural network, a model training index of the neural network is determined based on an estimated merging result of the neural network for a test sample and an actual merging label of the test sample, wherein the model training index includes at least one of a model accuracy rate, a model precision rate, and a model recall rate.

[0165] In this step, during the training of the neural network, in order to determine whether the neural network meets the preset training conditions, the training effect of the model can be evaluated.

[0166] First, obtain the estimated merged result of the test sample and the comparison result of the actual merged label of the test sample, wherein the method for obtaining the estimated merged result of the test sample can refer to the description of the method for obtaining the corresponding output results of the first sample statement and the second sample statement, and can achieve the same technical effect and solve the same technical problem, which is not repeated here.

[0167] Secondly, after obtaining the estimated merging result of the neural network for the test sample, a threshold can be set, and the estimated merging label can be determined based on the comparison result of the estimated merging result and the threshold. After determining the comparison result of the estimated merging label and the real merging label of the test sample, the result is statistically analyzed to determine the model training index of the neural network, wherein the model training index includes at least one of the model accuracy, model precision and model recall rate, wherein the threshold can be set manually according to the actual situation. For example, the threshold is set to 0.85. If the preset estimated merging result is greater than 0.85, the estimated merging label is determined to be a real mergible label among the real merging labels. If the preset estimated merging result is less than 0.85, the estimated merging label is determined to be a real non-mergible label among the real merging labels.

[0168] Among them, for each test sample, its label attribute is one of mergeable and non-mergeable. When determining the model training index of the neural network based on the test sample, there will be inconsistency between the estimated merged label and the actual merged label for the test sample. For example, if the estimated merged label of a test sample is selected as a mergeable label, and the actual merged label of this test sample is an non-mergeable label, this will affect the accuracy of the model, so the accuracy of the model is further ensured through the model training index.

[0169] Among them, the model accuracy is the ratio of the number of samples whose estimated merged labels are consistent with the true merged labels of the test samples to the total number of samples; the model precision is the ratio of the number of positive samples whose estimated merged labels are consistent with the true merged labels to the sum of the number of positive samples whose estimated merged labels are consistent with the true merged labels and the number of negative samples whose estimated merged labels are inconsistent with the true merged labels; the model recall is the ratio of the number of positive samples whose estimated merged labels are consistent with the true merged labels to the number of all positive samples.

[0170] When the model training index is greater than a preset index threshold, it is determined that the neural network meets the preset training condition, and the trained neural network is used as the trained paragraph merging recognition model.

[0171] In this step, when the model training index is greater than a preset index threshold, it can be determined that the neural network meets the preset training conditions, and the trained neural network can be used as the trained paragraph merging recognition model.

[0172] Exemplarily, if the model accuracy is greater than 80%, that is, it is determined that the neural network meets the preset training conditions, then the trained neural network can be used as the trained paragraph merging recognition model; if the model accuracy is less than 80%, that is, it is determined that the neural network does not meet the preset training conditions, then the trained neural network cannot be used as the trained paragraph merging recognition model.

[0173] In an optional implementation, before obtaining a plurality of sample sentence groups identified from the sample text and a real sample merged label for each sample sentence group, the method includes:

[0174] Determine the content of multiple paragraphs of text in the sample text.

[0175] In this step, when training the paragraph merging recognition model, it is necessary to determine the training text required for training the paragraph merging recognition model. The sample text can be determined as a training sample for training the model, and multiple paragraphs of text content can be determined from the sample text.

[0176] A plurality of positive sample sentence groups are identified from the plurality of text content segments, wherein the positive sample sentence groups include two adjacent sentences in the same paragraph.

[0177] In this step, based on the multiple paragraphs of text content determined from the sample text, multiple groups of two adjacent sentences in the same paragraph are identified in the multiple paragraphs of text content, and each group of two adjacent sentences in the same paragraph is determined as one of the multiple groups of positive sample sentence groups.

[0178] For example, see Figure 4 , Figure 4 A schematic diagram of the sample text content.

[0179] like Figure 4 As shown in , the positive sample sentence groups in the sample text can be the following two groups of sample sentence groups: "The so-called scenery is essentially a scene composed of mountains, rivers, and certain natural and cultural phenomena under certain conditions that is sufficient to arouse people's aesthetic appreciation and appreciation." and "Scenery, landscape perception and conditions are the three basic elements that constitute scenery."; "Scenery is an active factor and subjective reflection of the composition of scenery. It is a person's ability to perceive, identify and feel the scenery, such as vision, hearing, smell, taste, touch, association, psychology, etc." and "Conditions are the restrictive factors and causal means of the composition of scenery. It is a special relationship between the subject of scenery appreciation and the object of scenery.".

[0180] Identifying a plurality of negative sample sentence groups from the plurality of textual contents, wherein the negative sample sentence groups include two sentences located in different paragraphs;

[0181] In this step, based on the multiple paragraphs of text content determined from the sample text, multiple groups of two sentences in different paragraphs are identified in the multiple paragraphs of text content, and each group of two sentences in different paragraphs is determined as one of the multiple groups of negative sample sentence groups.

[0182] For example, Figure 4 As shown in , the negative sample sentence groups in the sample text can be the following two groups of sample sentence groups: "In ancient Chinese books, especially in the poetry and prose of purely literary works, it has been used for a long time, and even describes scenery more than romance, and is almost integrated with tourism." and "Scenery is an active factor and subjective reaction in the composition of scenery. It is a person's ability to perceive, identify and feel the scenery."

[0183] The identified multiple groups of positive sample sentence groups and the real sample merged labels of each positive sample sentence group, as well as the identified multiple groups of negative sample sentence groups and the real sample merged labels of each negative sample sentence group, are used as the multiple sample sentence groups and the real sample merged labels of each sample sentence group for training the paragraph merged recognition model.

[0184] In this step, in order to improve the accuracy of the model recognition results, after identifying multiple groups of positive sample sentence groups and the real sample merged labels of each positive sample sentence group, as well as identifying multiple groups of negative sample sentence groups and the real sample merged labels of each negative sample sentence group, they are used as multiple sample sentence groups and the real sample merged labels of each sample sentence group for training the paragraph merged recognition model, and then input into the paragraph merged recognition model for training.

[0185] In a specific embodiment, in order to improve the effect of model training, when extracting samples, the sample text selected may be a text of the same type as the text to be processed.

[0186] Furthermore, in order to better enable the model to learn the association between texts, some positive samples for training the model may be directly extracted from the text to be processed.

[0187] Specifically, multiple paragraphs of text content may be first determined from the text to be processed, and then multiple groups of positive sample sentences may be identified from the multiple paragraphs of text content, wherein the positive sample sentence group includes two adjacent sentences located in the same paragraph. Then, the multiple groups of positive sample sentence groups extracted from the text to be processed may be used as part of the multiple groups of positive sample sentence groups used for model training, thereby further improving the recognition accuracy of the paragraph merging recognition model.

[0188] The text processing method provided by the embodiment of the present disclosure obtains a sentence vector by encoding a plurality of target words and the number of target words divided from the sentences separated by paragraph delimiters selected from the text to be processed, and judges whether two sentences can be merged into the same sentence according to the sentence merging recognition result corresponding to the sentence vector, so as to learn the relationship between the two sentences, thereby learning the connection between the two paragraphs separated by the paragraph delimiters, and judging whether the two paragraphs can be merged, which can improve the accuracy of the sentence merging recognition result, thereby improving the efficiency and accuracy of paragraph merging, and can reduce the influence of too many paragraphs generated by a large number of paragraph delimiters on text analysis, which is conducive to improving the running speed of text analysis. And by selecting sample data for training the paragraph merging recognition model from the text to be processed, the relationship between sentences and the connection between paragraphs in the text to be processed can be accurately learned, which helps to improve the accuracy of the model in recognizing sentences in the text to be processed and improve the effectiveness of paragraph merging recognition in the text to be processed.

[0189] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0190] Based on the same inventive concept, a text processing device corresponding to the text processing method is also provided in the embodiment of the present disclosure. Since the principle of solving the problem by the device in the embodiment of the present disclosure is similar to the above-mentioned text processing method in the embodiment of the present disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0191] See also Figures 5 and 6 , Figure 5This is one of the structural diagrams of a text processing device provided in an embodiment of the present disclosure, Figure 6 This is a second structural diagram of a text processing device provided in an embodiment of the present disclosure.

[0192] like Figure 5 As shown in , the text processing device 500 provided by the embodiment of the present disclosure includes:

[0193] A sentence acquisition module 510 is used to determine at least one group of sentences to be merged in the text to be processed;

[0194] A vector determination module 520, configured to determine a first sentence vector corresponding to the first sentence and a second sentence vector corresponding to the second sentence according to a plurality of first target words divided from the first sentence and the number of the first target words, and a plurality of second target words divided from the second sentence and the number of the second target words;

[0195] A result recognition module 530 is used to input the first sentence vector and the second sentence vector into a trained paragraph merging recognition model to obtain a sentence merging recognition result;

[0196] The paragraph merging module 540 is configured to determine whether the paragraph where the first sentence is located and the paragraph where the second sentence is located can be merged into one paragraph if the sentence merging recognition result indicates that the first sentence and the second sentence are contents of the same sentence.

[0197] In an optional implementation manner, the statement acquisition module 510 is specifically used to:

[0198] Get text data of the text to be processed;

[0199] According to the paragraph separator identified from the text data, determining a first paragraph adjacent to the paragraph separator and located before the paragraph separator and a second paragraph located after the paragraph separator in the text to be processed;

[0200] According to a preset sentence identification rule, determining a first sentence located at the last sentence position in the first paragraph and a second sentence located at the first sentence position in the second paragraph, wherein the first sentence and the second sentence each include at least one natural sentence;

[0201] The first statement and the second statement are determined as a group of statements to be merged in at least one group of statements to be merged.

[0202] In an optional implementation manner, the vector determination module 520 is specifically configured to:

[0203] Obtaining a word segmentation method for the text to be processed;

[0204] According to the word segmentation method, the first sentence and the second sentence are respectively segmented to obtain a plurality of first target words corresponding to the first sentence and the number of the plurality of first target words, and a plurality of second target words corresponding to the second sentence and the number of the plurality of second target words;

[0205] encoding the first sentence according to the plurality of first target words and the number of the plurality of first target words to obtain a first sentence vector corresponding to the first sentence;

[0206] The second sentence is encoded according to the multiple second target words and the number of the multiple second target words to obtain a second sentence vector corresponding to the second sentence.

[0207] In an optional implementation manner, the result identification module 530 is specifically used to:

[0208] Inputting the first sentence vector and the second sentence vector into the semantic recognition layer in the trained paragraph merging recognition model to obtain a semantic distance vector between the first sentence vector and the second sentence vector;

[0209] The semantic distance vector is input into a merge recognition layer in the paragraph merge recognition model to obtain a sentence merge recognition result for the first sentence and the second sentence, wherein the sentence merge recognition result includes a sentence merge probability.

[0210] In an optional implementation manner, the paragraph merging module 540 is specifically used for:

[0211] If the statement merging probability is greater than a preset probability threshold, determining that the first statement and the second statement meet a merging condition;

[0212] When the first sentence and the second sentence meet the merging condition, the paragraph where the first sentence is located is merged with the paragraph where the second sentence is located.

[0213] In an optional implementation, if Figure 6 As shown in , the text processing device 500 further includes a model training module 550, and the model training module 550 is specifically used for:

[0214] Acquire multiple sample sentence groups identified from the sample text and a true sample merge label of each sample sentence group, wherein the sample sentence group includes a first sample sentence and a second sample sentence;

[0215] Determine a first sample vector corresponding to the first sample sentence and a second sample vector corresponding to the second sample sentence;

[0216] The constructed neural network is trained using the first sample vector, the second sample vector and the real sample merge label until the neural network meets the preset training conditions, thereby obtaining a trained paragraph merge recognition model.

[0217] In an optional implementation manner, when the model training module 550 determines that the neural network meets the preset training condition through the following steps, it is specifically configured to:

[0218] In the process of training the neural network, based on the estimated merged result of the neural network for the test sample and the actual merged label of the test sample, a model training index of the neural network is determined, wherein the model training index includes at least one of model accuracy, model precision and model recall;

[0219] When the model training index is greater than a preset index threshold, it is determined that the neural network meets the preset training condition, and the trained neural network is used as the trained paragraph merging recognition model.

[0220] In an optional implementation, if Figure 6 As shown in , the text processing device 500 further includes a sample acquisition module 560, and the sample acquisition module 560 is specifically used for:

[0221] Determining the contents of multiple paragraphs of text in the sample text;

[0222] Identifying a plurality of positive sample sentence groups from the plurality of textual contents, wherein the positive sample sentence groups include two adjacent sentences in the same paragraph;

[0223] Identifying a plurality of negative sample sentence groups from the plurality of textual contents, wherein the negative sample sentence groups include two sentences located in different paragraphs;

[0224] The identified multiple groups of positive sample sentence groups and the real sample merged labels of each positive sample sentence group, as well as the identified multiple groups of negative sample sentence groups and the real sample merged labels of each negative sample sentence group, are used as the multiple sample sentence groups and the real sample merged labels of each sample sentence group for training the paragraph merged recognition model.

[0225] The text processing device provided by the embodiment of the present disclosure can obtain a sentence vector by encoding multiple target words and the number of target words divided from the sentences separated by paragraph delimiters selected from the text to be processed, and judge whether two sentences can be merged into the same sentence according to the sentence merging recognition result corresponding to the sentence vector, and can learn the relationship between the two sentences, thereby learning the connection between the two paragraphs separated by the paragraph delimiter, so as to judge whether the two paragraphs can be merged, and can improve the accuracy of the sentence merging recognition result, thereby improving the efficiency and accuracy of paragraph merging, and can reduce the impact of too many paragraphs generated by a large number of paragraph delimiters on text analysis, which is conducive to improving the running speed of text analysis. And by selecting sample data for training the paragraph merging recognition model from the text to be processed, the relationship between sentences and the connection between paragraphs in the text to be processed can be accurately learned, which helps to improve the accuracy of the model in recognizing sentences in the text to be processed and improve the effectiveness of paragraph merging recognition in the text to be processed.

[0226] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference may be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.

[0227] Based on the same technical concept, the embodiment of the present disclosure also provides an electronic device. Figure 7 As shown, it is a schematic diagram of the structure of the electronic device 700 provided in the embodiment of the present application, including a processor 710, a memory 720, and a bus 730. Among them, the memory 720 is used to store execution instructions, including a memory 721 and an external memory 722; the memory 721 here is also called an internal memory, which is used to temporarily store the operation data in the processor 710, and the data exchanged with the external memory 722 such as a hard disk. The processor 710 exchanges data with the external memory 722 through the memory 721. When the electronic device 700 is running, the processor 710 communicates with the memory 720 through the bus 730, so that the processor 710 can execute the steps of the text processing method described in the above method embodiment.

[0228] The present disclosure also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the text processing method described in the above method embodiment are executed. The storage medium may be a volatile or non-volatile computer-readable storage medium.

[0229] A computer program product of a text processing method provided in an embodiment of the present disclosure includes a computer-readable storage medium storing program code, and the instructions included in the program code can be used to execute the steps of the text processing method described in the above method embodiment. For details, please refer to the above method embodiment, which will not be repeated here.

[0230] The present disclosure also provides a computer program, which implements any one of the methods of the aforementioned embodiments when executed by a processor. The computer program product can be implemented in hardware, software, or a combination thereof. In an optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK), etc.

[0231] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the device or unit can be electrical, mechanical or other forms.

[0232] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0233] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0234] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0235] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The protection scope of the present disclosure is not limited thereto. Although the present disclosure is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed in the present disclosure, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A text processing method, characterized in that: The method comprises: Determine at least one group of sentences to be merged in the text to be processed, wherein the sentences to be merged include a first sentence and a second sentence that are adjacent to each other, the first sentence and the second sentence are respectively located in a first paragraph and a second paragraph of the text to be processed, the first paragraph includes other sentences except the first sentence, and the second paragraph includes other sentences except the second sentence; Determine a first sentence vector corresponding to the first sentence and a second sentence vector corresponding to the second sentence; Inputting the first sentence vector and the second sentence vector into a trained paragraph merging recognition model to obtain a sentence merging recognition result; If the sentence merging recognition result indicates that the first sentence and the second sentence meet the merging condition, merging the first paragraph with the second paragraph; The step of determining at least one group of sentences to be merged in the text to be processed includes: Obtain text data of the text to be processed; According to the paragraph separator identified from the text data, determining the first paragraph adjacent to the paragraph separator and located before the paragraph separator and the second paragraph located after the paragraph separator in the text to be processed; According to a preset sentence identification rule, determining a first sentence located at the last sentence position in the first paragraph and a second sentence located at the first sentence position in the second paragraph, wherein the first sentence and the second sentence each include at least one natural sentence; The first statement and the second statement are determined as a group of statements to be merged in at least one group of statements to be merged.

2. The method according to claim 1, characterized in that The determining a first sentence vector corresponding to the first sentence and a second sentence vector corresponding to the second sentence includes: Obtaining a word segmentation method for the text to be processed; According to the word segmentation method, the first sentence and the second sentence are respectively segmented to obtain a plurality of first target words corresponding to the first sentence and the number of the plurality of first target words, and a plurality of second target words corresponding to the second sentence and the number of the plurality of second target words; encoding the first sentence according to the plurality of first target words and the number of the plurality of first target words to obtain a first sentence vector corresponding to the first sentence; The second sentence is encoded according to the multiple second target words and the number of the multiple second target words to obtain a second sentence vector corresponding to the second sentence.

3. The method according to claim 1, characterized in that The step of inputting the first sentence vector and the second sentence vector into a trained paragraph merging recognition model to obtain a sentence merging recognition result includes: Inputting the first sentence vector and the second sentence vector into the semantic recognition layer in the trained paragraph merging recognition model to obtain a semantic distance vector between the first sentence vector and the second sentence vector; The semantic distance vector is input into a merge recognition layer in the paragraph merge recognition model to obtain a sentence merge recognition result for the first sentence and the second sentence, wherein the sentence merge recognition result includes a sentence merge probability.

4. The method according to claim 3, characterized in that: If the sentence merging recognition result indicates that the first sentence and the second sentence meet a merging condition, merging the first paragraph with the second paragraph includes: If the statement merging probability is greater than a preset probability threshold, determining that the first statement and the second statement meet a merging condition; When the first sentence and the second sentence meet a merging condition, the first paragraph and the second paragraph are merged.

5. The method according to claim 1, characterized in that Obtain the trained paragraph merging recognition model by following the steps below: Acquire multiple sample sentence groups identified from the sample text and a true sample merge label of each sample sentence group, wherein the sample sentence group includes a first sample sentence and a second sample sentence; Determine a first sample vector corresponding to the first sample sentence and a second sample vector corresponding to the second sample sentence; The constructed neural network is trained using the first sample vector, the second sample vector and the real sample merge label until the neural network meets the preset training conditions, thereby obtaining a trained paragraph merge recognition model.

6. The method according to claim 5, characterized in that Determine that the neural network meets the preset training condition by the following steps: In the process of training the neural network, based on the estimated merged result of the neural network for the test sample and the actual merged label of the test sample, a model training index of the neural network is determined, wherein the model training index includes at least one of model accuracy, model precision and model recall; When the model training index is greater than a preset index threshold, it is determined that the neural network meets the preset training condition, and the trained neural network is used as the trained paragraph merging recognition model.

7. The method according to claim 5, characterized in that Before obtaining a plurality of sample sentence groups identified from the sample text and a real sample merged label for each sample sentence group, the method includes: Determining the contents of multiple paragraphs of text in the sample text; Identifying a plurality of positive sample sentence groups from the plurality of textual contents, wherein the positive sample sentence groups include two adjacent sentences in the same paragraph; Identifying a plurality of negative sample sentence groups from the plurality of textual contents, wherein the negative sample sentence groups include two sentences located in different paragraphs; The identified multiple groups of positive sample sentence groups and the real sample merged labels of each positive sample sentence group, as well as the identified multiple groups of negative sample sentence groups and the real sample merged labels of each negative sample sentence group, are used as the multiple sample sentence groups and the real sample merged labels of each sample sentence group for training the paragraph merged recognition model.

8. A text processing device, characterized in that: The device comprises: A sentence acquisition module, used to determine at least one group of sentences to be merged in the text to be processed, wherein the sentences to be merged include a first sentence and a second sentence that are adjacent to each other, the first sentence and the second sentence are respectively located in a first paragraph and a second paragraph of the text to be processed, the first paragraph includes other sentences except the first sentence, and the second paragraph includes other sentences except the second sentence; A vector determination module, used to determine a first sentence vector corresponding to the first sentence and a second sentence vector corresponding to the second sentence; A result recognition module, used for inputting the first sentence vector and the second sentence vector into a trained paragraph merging recognition model to obtain a sentence merging recognition result; a paragraph merging module, configured to merge the first paragraph with the second paragraph if the sentence merging recognition result indicates that the first sentence and the second sentence meet a merging condition; Wherein, the statement acquisition module is specifically used for: Obtain text data of the text to be processed; According to the paragraph separator identified from the text data, determining the first paragraph adjacent to the paragraph separator and located before the paragraph separator and the second paragraph located after the paragraph separator in the text to be processed; According to a preset sentence identification rule, determining a first sentence located at the last sentence position in the first paragraph and a second sentence located at the first sentence position in the second paragraph, wherein the first sentence and the second sentence each include at least one natural sentence; The first statement and the second statement are determined as a group of statements to be merged in at least one group of statements to be merged.

9. A computer device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the text processing method as described in any one of claims 1 to 7 are performed.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the text processing method according to any one of claims 1 to 7 are executed.

Images