Text paragraph identification method and device, equipment and storage medium

Abstract

The application provides a text paragraph identification method, device and equipment and a storage medium. First, a first sample set and a second sample set are obtained, the first sample set and the second sample set including text paragraphs of a target text type and a non-target text type, then a digital training set is obtained according to the first sample set and the second sample set, a preset identification model is trained according to the digital training set to obtain a target identification model, and finally, the target identification model is used to perform paragraph identification on a text to be identified, the text to be identified being a target text not including a prefix feature, and the prefix feature being used to represent a text type of the target text. Thus, a text paragraph identification method is provided for a text without a prefix feature, the risk of misidentification is avoided, the identification accuracy is high, and the text structured processing requirement is met.

Description

Technical Field

[0001] This application relates to the field of text structuring technology, and in particular to a method, apparatus, device and storage medium for text paragraph recognition. Background Technology

[0002] In the field of text structuring technology, it is often necessary to extract paragraphs of text and extract metadata based on the text content within each paragraph. This metadata can then be processed according to the specified format of the Journal Publishing Tag Set (JATS). Therefore, paragraph recognition is the foundation for a series of subsequent tasks.

[0003] Conventional text paragraph recognition methods often rely on prefix features that characterize text types. For example, in grant-related texts, phrases such as "fund projects:", "project source:", "research source:", "fund programs:", "foundings:", and "foundation item:" can be considered prefix features characterizing the text type. In other words, when these prefix features appear in grant-related texts, they can be used to segment the text into paragraphs, thereby identifying the paragraphs.

[0004] However, when there are no prefix features in the fund text, paragraph recognition may need to be performed based on some preset keywords contained in the fund text. Although this recognition method is intuitive and simple, the recognition results are less accurate, there is a significant risk of false recognition, and for large-scale recognition, the recognition task is arduous, thus failing to meet the needs of batch processing of structured text. Summary of the Invention

[0005] This application provides a text paragraph recognition method, apparatus, device, and storage medium, which provides a paragraph recognition method for grant papers without prefix features, avoiding the risk of misidentification and achieving high accuracy in the recognition results.

[0006] Firstly, this application provides a text paragraph recognition method, including:

[0007] Obtain a sample set, which includes a first sample set and a second sample set, wherein the first sample set and the second sample set respectively include text paragraphs of target text type and non-target text type;

[0008] A digital training set is obtained based on the first sample set and the second sample set, and a target recognition model is obtained by training a preset recognition model based on the digital training set.

[0009] The target recognition model is used to identify paragraphs of the text to be identified. The text to be identified is target text that does not include prefix features. The prefix features are used to characterize the text type of the target text.

[0010] In one possible design, obtaining the digitized training set based on the first sample set and the second sample set includes:

[0011] Each text segment of the first sample set and the second sample set is preprocessed, and the resulting preprocessed data is then segmented into words.

[0012] According to the preset segmentation rules, each word in the result data obtained from the word segmentation process is divided into word groups to form a paragraph recognition feature set;

[0013] The first feature sequence corresponding to the preprocessed result data is obtained based on the paragraph recognition feature set, so as to form the digital training set based on each first feature sequence.

[0014] In one possible design, the preprocessing of each text segment in the first sample set and the second sample set includes:

[0015] Content unrelated to preset metadata is filtered out from each text segment, and preset hyphens are deleted, so that each text segment after filtering and deletion operations is determined as the preprocessed result data.

[0016] In one possible design, the word segmentation process performed on the obtained preprocessed result data includes:

[0017] Each first result data and each second result data is segmented using a preset word segmentation model to obtain the corresponding third result data and each fourth result data.

[0018] The preprocessed result data includes the first result data and the second result data, and the result data obtained by the word segmentation process includes the third result data and the fourth result data.

[0019] In one possible design, obtaining the first feature sequence corresponding to the preprocessed result data based on the paragraph recognition feature set, and forming the digital training set based on each first feature sequence, includes:

[0020] The number of words in the vocabulary groups included in each of the first result data and each of the second result data is obtained according to the paragraph recognition feature set, so as to obtain the first feature sequence corresponding to each of the first result data and each of the second result data;

[0021] Each of the first feature sequences is determined as a subset of the digital training set to obtain the digital training set.

[0022] In one possible design, after obtaining the corresponding third and fourth result data, the following is also included:

[0023] The frequency of each word in each third result data and each fourth result data is statistically analyzed to obtain the mapping relationship between each third result data and each fourth result data and each frequency.

[0024] The preset partitioning rules are generated based on the mapping relationship.

[0025] One possible design also includes:

[0026] The target recognition model is used to identify paragraphs in each verification sample in the verification sample set, and the accuracy of the recognition results is determined.

[0027] If not, adjust the word groups in the paragraph recognition feature set according to the recognition results;

[0028] The second feature sequence corresponding to the preprocessed result data is obtained based on the adjusted paragraph recognition feature set, and an optimized training set is formed based on each second feature sequence.

[0029] The target recognition model is optimized based on the optimized training set until the prediction result meets the preset threshold range. The prediction result is used to characterize the probability that the optimized target recognition model's recognition result is a correct result or an incorrect result.

[0030] Secondly, this application provides a text paragraph recognition device, comprising:

[0031] A sampling module is used to acquire a sample set, which includes a first sample set and a second sample set, wherein the first sample set and the second sample set respectively include text paragraphs of target text type and non-target text type;

[0032] The processing module is used to obtain a digital training set based on the first sample set and the second sample set, and to train a preset recognition model based on the digital training set to obtain a target recognition model.

[0033] The recognition module is used to perform paragraph recognition on the text to be recognized using the target recognition model. The text to be recognized is target text that does not include prefix features. The prefix features are used to characterize the text type of the target text as the target text type.

[0034] In one possible design, the processing module includes:

[0035] The first processing submodule is used to preprocess each text segment of the first sample set and the second sample set respectively, and to perform word segmentation on the obtained preprocessed result data.

[0036] The second processing submodule is used to divide each word in the result data obtained from the word segmentation process into word groups according to the preset segmentation rules, forming a paragraph recognition feature set;

[0037] The third processing submodule is used to obtain the first feature sequence corresponding to the preprocessed result data based on the paragraph recognition feature set, so as to form the digital training set based on each first feature sequence.

[0038] In one possible design, the first processing submodule is specifically used for:

[0039] Content unrelated to preset metadata is filtered out from each text segment, and preset hyphens are deleted, so that each text segment after filtering and deletion operations is determined as the preprocessed result data.

[0040] In one possible design, the first processing submodule is further specifically used for:

[0041] Each first result data and each second result data is segmented using a preset word segmentation model to obtain the corresponding third result data and each fourth result data.

[0042] The preprocessed result data includes the first result data and the second result data, and the result data obtained by the word segmentation process includes the third result data and the fourth result data.

[0043] In one possible design, the third processing submodule is specifically used for:

[0044] The number of words in the vocabulary groups included in each of the first result data and each of the second result data is obtained according to the paragraph recognition feature set, so as to obtain the first feature sequence corresponding to each of the first result data and each of the second result data;

[0045] Each of the first feature sequences is determined as a subset of the digital training set to obtain the digital training set.

[0046] In one possible design, the text paragraph recognition device further includes: a statistics and generation module; the statistics and generation module is used for:

[0047] The frequency of each word in each third result data and each fourth result data is statistically analyzed to obtain the mapping relationship between each third result data and each fourth result data and each frequency.

[0048] The preset partitioning rules are generated based on the mapping relationship.

[0049] In one possible design, the text paragraph recognition device further includes: a verification and optimization module; the verification and optimization module is specifically used for:

[0050] The target recognition model is used to identify paragraphs in each verification sample in the verification sample set, and the accuracy of the recognition results is determined.

[0051] If not, adjust the word groups in the paragraph recognition feature set according to the recognition results;

[0052] The second feature sequence corresponding to the preprocessed result data is obtained based on the adjusted paragraph recognition feature set, and an optimized training set is formed based on each second feature sequence.

[0053] The target recognition model is optimized based on the optimized training set until the prediction result meets the preset threshold range. The prediction result is used to characterize the probability that the optimized target recognition model's recognition result is a correct result or an incorrect result.

[0054] Thirdly, this application provides an electronic device, comprising:

[0055] Processor; and,

[0056] Memory for storing the computer program of the processor;

[0057] The processor is configured to execute any of the possible text paragraph recognition methods provided in the first aspect by executing the computer program.

[0058] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements any of the possible text paragraph recognition methods provided in the first aspect.

[0059] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements any of the possible text paragraph recognition methods provided in the first aspect.

[0060] This application provides a method, apparatus, device, and storage medium for text paragraph recognition. First, a sample set is acquired, comprising a first sample set and a second sample set, each containing text paragraphs of two different text types: a target text type and a non-target text type. Then, a digitized training set is obtained based on the first and second sample sets, and a preset recognition model is trained using this training set to obtain a target recognition model. Finally, the target recognition model is used to perform paragraph recognition on the text to be recognized, which is the target text excluding prefix features. Prefix features are used to characterize the text type of the target text. This provides a method for text paragraph recognition without prefix features, avoiding the risk of misidentification and achieving high recognition accuracy, thus meeting the needs of text structure processing. Attached Figure Description

[0061] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0062] Figure 1 This is a schematic diagram of an application scenario provided by an embodiment of this application;

[0063] Figure 2 A flowchart illustrating a text paragraph recognition method provided in an embodiment of this application;

[0064] Figure 3 A flowchart illustrating another text paragraph recognition method provided in this application embodiment;

[0065] Figure 4 A flowchart illustrating another text paragraph recognition method provided in this application embodiment;

[0066] Figure 5 A flowchart illustrating another text paragraph recognition method provided in this application embodiment;

[0067] Figure 6 This is a schematic diagram of the structure of a text paragraph recognition device provided in an embodiment of this application;

[0068] Figure 7 This is a schematic diagram of the structure of a processing module provided in an embodiment of this application;

[0069] Figure 8 This is a schematic diagram of the structure of an electronic device provided in this application. Detailed Implementation

[0070] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of methods and apparatus consistent with some aspects of this application as detailed in the appended claims.

[0071] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0072] Conventional text paragraph recognition methods often rely on prefix features that characterize text types. For example, in grant-related texts, phrases like "fund projects:", "project sources:", "research sources:", "fund programs:", "foundings:", and "foundation item:" can be considered prefix features representing text types. In other words, when these prefix features appear in a grant-related text, they can be used to segment the text into paragraphs, thus identifying the paragraphs. However, when no prefix features are present in the grant-related text, paragraph recognition may need to be based on pre-defined keywords. While this method is intuitive and simple, it has low accuracy, a significant risk of false positives, and is too cumbersome for large-scale recognition, failing to meet the needs of structured batch text processing.

[0073] To address the aforementioned problems in existing technologies, this application provides a text paragraph recognition method, apparatus, device, and storage medium. The inventive concept of the text paragraph recognition method provided in this application is as follows: For unprefixed text, a sample set is first obtained, which includes text paragraphs of different text types, such as paragraphs of the target text type and paragraphs of non-target text types. Then, a digital training set is obtained based on these text paragraphs, and a preset recognition model is trained using the digital training set to obtain a target recognition model. Finally, the target recognition model is used to perform paragraph recognition on the text to be recognized. This provides a paragraph recognition method for unprefixed text of the target text type, which not only avoids the misidentification of paragraphs in existing technologies for this type of unprefixed text, but also effectively improves the recognition accuracy by using a model for paragraph recognition. Furthermore, the target recognition model allows for batch paragraph recognition, meeting the needs of structured batch text processing.

[0074] The following describes exemplary application scenarios of the embodiments of this application.

[0075] Figure 1 This is a schematic diagram of an application scenario provided in an embodiment of this application, such as... Figure 1 As shown, a corresponding database for storing text to be recognized is running on the terminal device 11. The processor in the electronic device 12 is configured to execute the text paragraph recognition method provided in this application embodiment. By acquiring a sample set and obtaining a digital training set based on the sample set, the preset recognition model is trained into a target recognition model according to the digital training set. Finally, the target recognition model can be used to perform paragraph recognition on each text to be recognized stored in the terminal device 11.

[0076] The terminal device 11 can be any terminal capable of running the corresponding database, such as a computer, laptop, server, or server cluster. This embodiment does not limit the type of terminal device 11. Figure 1 The terminal device 11 is illustrated using a computer as an example. The electronic device 12 can be a computer, server, server cluster, or other device that can execute corresponding computer programs to implement the text paragraph recognition method provided in this application embodiment. This embodiment does not limit the scope of the application in this regard. Figure 1 Electronic device 12 is shown as an example of a computer.

[0077] It is understandable that terminal device 11 and terminal device 12 can be connected via network communication, which can be wired, wireless, or other methods.

[0078] It should be noted that the above application scenarios are merely illustrative, and the text paragraph recognition methods, devices, equipment, and storage media provided in this application include, but are not limited to, the above application scenarios.

[0079] Figure 2 This is a flowchart illustrating a text paragraph recognition method provided in an embodiment of this application. Figure 2 As shown, the text paragraph recognition method provided in this embodiment includes:

[0080] S101: Obtain the sample set.

[0081] The sample set includes a first sample set and a second sample set, which respectively include text paragraphs of the target text type and non-target text type.

[0082] A sample set is formed by collecting some text paragraphs from actual working conditions. For example, multiple text paragraphs with different text types, such as the target text type and non-target text type, are collected to form the sample set.

[0083] In this embodiment, text of the target text type is defined as funding text. A funding text paragraph could be, for example, "This paper is one of the research results of the National Natural Science Foundation of China Youth Science Fund Project 'Research on Automatic Review of Academic Literature Based on Deep Semantic Representation and Multi-Document Summarization' (Project No.: 71904058) and the Fundamental Research Funds for the Central Universities Project 'Research on the Evolution Path of Artificial Intelligence Algorithm Based on Dynamic Citation Network' (Project No.: KJ02072020-0200)." Correspondingly, text of a different type than the target text type is considered non-target text. A non-target text paragraph could be, for example, "This research was supported by the 'Sedimentology Strategy Seminar' project chaired by Academician Zhang San, and some content was shared at the 'International Symposium on the Development Strategy of Sedimentology in China' (571st Xiangshan Science Conference, Beijing, September 25-28, 2016). We express our gratitude." It is understood that this embodiment does not limit the text content expressed by each paragraph of the target text type and non-target text type.

[0084] In addition, each text paragraph of the target text type is treated as a subset, and the set of these subsets constitutes the first sample set. Similarly, each text paragraph of a non-target text type is treated as a subset, and the set of these subsets constitutes the second sample set.

[0085] S102: Obtain a digital training set based on the first sample set and the second sample set, and train a preset recognition model based on the digital training set to obtain a target recognition model.

[0086] After obtaining the sample set, a digital training set is further obtained based on the sample set. This digital training set is then used to train a pre-defined recognition model to obtain a target recognition model. The pre-defined recognition model can be a Support Vector Machine (SVM) model. Subsets from the digital training set are used as training samples for the pre-defined recognition model. The trained pre-defined recognition model is then used to perform paragraph recognition on the text to be recognized.

[0087] In one possible design, step S102 could be implemented as follows: Figure 3 As shown. Figure 3 This is a flowchart illustrating another text paragraph recognition method provided in an embodiment of this application. Figure 3 As shown, this embodiment includes:

[0088] S1021: Preprocess each text segment of the first and second sample sets respectively, and perform word segmentation on the preprocessed data.

[0089] Each text segment in the first and second sample sets is preprocessed to obtain the corresponding preprocessed result data, and then the preprocessed result data is segmented into words.

[0090] Alternatively, possible implementations for preprocessing each text segment included in the first sample set and the second sample set respectively can be as follows:

[0091] The text segments in the first and second sample sets are filtered out for content unrelated to the preset metadata, and preset hyphens are deleted. The text segments after the filtering and deletion operations are then determined as the preprocessed result data.

[0092] For fund text, the default metadata can be a description of the fund text, its name, or its fund number. For non-target type text paragraphs, the default metadata can be a description of the text paragraph, names within the text paragraph, or numerical identifiers within the text paragraph.

[0093] Filtering out content that is irrelevant to the preset metadata can include at least one of the following specific operations:

[0094] (1) Remove the content within quotation marks and book titles from the text paragraphs;

[0095] (2) Remove content such as "topic / subtopic";

[0096] (3) Remove content such as “cover page 1 / insert 1 after / person in charge”.

[0097] Furthermore, preset hyphens are deleted. These preset hyphens can be various symbols in the text paragraph that affect word segmentation, such as parentheses, square brackets, hyphens, forward slashes, backslashes, and spaces; this embodiment does not limit their use. Deleting preset hyphens means deleting the defined preset hyphens.

[0098] Thus, the text segments after the above filtering and deletion operations are determined as the preprocessed result data, completing the preprocessing of the text segments included in the first and second sample sets.

[0099] Furthermore, the preprocessed data is segmented into words.

[0100] Optionally, word segmentation can be implemented by running a preset word segmentation model. The preset word segmentation model can be any word segmentation software used for word segmentation; this embodiment does not limit the specific word segmentation software used.

[0101] Specifically, for example, if the preprocessed result data corresponding to each sample segment in the first sample set is defined as each first result data, then each first result data is the result data obtained after preprocessing each sample segment in the first sample set. Correspondingly, if the preprocessed result data corresponding to each sample segment in the second sample set is defined as each second result data, then each second result data is the result data obtained after preprocessing each sample segment in the second sample set. In other words, the preprocessed result data includes each first result data and each second result data. Therefore, each first result data and each second result data can be segmented using a preset word segmentation model, and the result data obtained after segmenting each first result data is determined as each third result data, while the result data obtained after segmenting each second result data is determined as each fourth result data. In other words, the result data obtained after segmentation using the preset word segmentation model includes each third result data and each fourth result data.

[0102] S1022: According to the preset segmentation rules, each word in the result data obtained from word segmentation is divided into word groups to form a paragraph recognition feature set.

[0103] The result data obtained after word segmentation, namely the third result data and the fourth result data, are divided into groups according to the preset segmentation rules to obtain multiple word groups. The set formed by each word group is determined as the paragraph recognition feature set.

[0104] Multiple word groups can be represented, for example, by the words enclosed in double quotes below, where each word within a double quote forms a word group. If a word group is considered a subset, then the set of all word groups constitutes the paragraph recognition feature set.

[0105] For example, the word groups obtained from the word divisions in each third result data can be "(fund|special project|support|funding|operating expenses|funding|plan|topic)", "(engineering|support|support|project|planning)", "(science and technology|scientific research|science|technology|application|innovation|research|development|development)", "(national|central|China)", "[az\d]{7,}", "(number|project number|id)", "(university|laboratory|university|education department|education ministry|school|center|research institute|college|company|industry and information technology department)", etc.

[0106] For example, the word groups obtained by dividing each word in each fourth result data can be "(specify|formulate|introduction|experimental report|purpose|communication|professor|lecturer|conclusion|essence|significance|weight|effect|result|impact|publisher|seminar|conference|postal code|telephone|magazine|production line|above|we|they)", "(in order|as|from|target|according to|for|for example|based on|according to|through)", "1[3-9]{1,1}\d{9,9}", etc.

[0107] It should be noted that the words in each of the third and fourth result data are not limited to text, but can also be numbers, such as serial numbers, mobile phone numbers, etc.

[0108] As described above, the words in the segmentation results are divided into multiple word groups, denoted by N, where N is a natural number greater than 0. A larger value for N results in more subsets of the paragraph recognition features, leading to a more accurate target recognition model and a longer recognition time. However, an excessively large value for N can cause overfitting in the target recognition model, affecting the recognition performance. Therefore, when dividing words into word groups, a pre-defined segmentation rule must be followed.

[0109] In one possible design, after performing word segmentation processing using a preset word segmentation model to obtain the third and fourth result data, it also includes, for example: Figure 4 The steps for determining the preset partitioning rules are shown. Figure 4 This is a flowchart illustrating another text paragraph recognition method provided in an embodiment of this application. Figure 4 As shown, this embodiment includes:

[0110] S201: Calculate the frequency of each word in each third result data and each fourth result data to obtain the mapping relationship between each third result data and each fourth result data and each frequency.

[0111] S202: Generate preset partitioning rules based on the mapping relationship.

[0112] After the preset word segmentation model performs word segmentation, the third and fourth result data obtained will determine the words and their parts of speech in each of the third and fourth result data. Therefore, the frequency of each word in each third and fourth result data can be further counted to obtain the correspondence between the word, its part of speech, and its corresponding frequency. In other words, by counting the frequency of each word, a mapping relationship is obtained between each third and fourth result data and the frequency of each word, and a preset segmentation rule is generated based on this mapping relationship.

[0113] When generating preset segmentation rules based on mapping relationships, these rules can be generated according to the frequency of each word in each third or fourth result data set, as well as the word's part of speech. For example, if a word appears frequently in each third result data set, that word, along with words similar in part of speech and meaning, can be identified as the preset segmentation rule for each third result data set—for example, words that are highly likely to appear in the basic text. Similarly, if a word appears frequently in each fourth result data set, that word, along with words similar in part of speech and meaning, can be identified as the preset segmentation rule for each fourth result data set—for example, words that are highly unlikely to appear in the basic text.

[0114] After dividing each word in the result data obtained by word segmentation into word groups through the above steps to form a paragraph recognition feature set, step S1023 is further executed.

[0115] S1023: Obtain the first feature sequence corresponding to the preprocessed result data based on the paragraph recognition feature set, so as to form a digital training set based on each first feature sequence.

[0116] After obtaining the paragraph recognition feature set, the preprocessed result data is obtained based on the paragraph recognition feature set, that is, the first feature sequence corresponding to each first result data and each second result data is obtained, and then a digital training set is formed based on each first feature sequence.

[0117] For example, based on the paragraph recognition feature set, the number of words in each word group included in each first result data and each second result data is first obtained, and the obtained number is determined as the corresponding first feature sequence for each first result data and each second result data. Then, each obtained first feature sequence is determined as a subset of the digital training set, thus forming the digital training set.

[0118] It is understood that in this embodiment, the vocabulary groups are divided according to each third result data to obtain the paragraph recognition feature set corresponding to each third result data. Accordingly, the number of words in each vocabulary group included in each first result data is obtained according to the paragraph recognition feature set. For example, the vocabulary groups obtained by dividing each word in each third result data are such as "(fund|special project|funding|funds|operating expenses|funding|plan|topic)", "(engineering|support|support|project|planning)", "(science and technology|scientific research|science|technology|application|innovation|research|development|development)", "(national|central|China)", "[az\d]{7,}", "(number|project number|id)", and "(university|laboratory|university|education department|education ministry|school|center|research institute|college|company|industry and information technology department)", that is, the value of N is 7. Suppose the first result data is "This paper is one of the research results of the National Natural Science Foundation of China Youth Science Fund Project (Project No.: 71904058) and the Fundamental Research Funds for the Central Universities Project (Project No.: KJ02072020X0200)". Then, this first result data contains the words "fund, fund, business expenses, funding, plan, project" in the first group "(fund|special project|funding|funds|business expenses|funding|plan|topic)", with a total of 4 words. If the first group is represented by 0, then "0:4" represents the number of words in the first word group in the first result data being 4. Similarly, if the second group "(engineering|support|support|project|planning)" is represented by 1, then this first result data contains the words "project, project, project, project", with a total of 4 words. Therefore, "1:4" represents the number of words in the second word group in the first result data being 4. Following this pattern, the number of words in the third, fourth, fifth, and sixth groups of the first result data can be obtained as 4, 2, 2, and 1 respectively. Therefore, according to the above representation rules, the number of words in each word group of the first result data can be represented as the numerical sequence "0:4 1:4 2:4 3:2 4:2 5:2 6:1" respectively. This numerical sequence is defined as the first feature sequence corresponding to the first result data.

[0119] Based on the description of the above embodiments, a first feature sequence corresponding to each first result data and each second result data can be obtained. Each first feature sequence is then considered a subset, and the set formed by all first feature sequences is determined as the digital training set. In other words, each first feature sequence is determined as a subset of the digital training set, thus obtaining the digital training set.

[0120] After obtaining the digitized training set, each first feature sequence in the digitized training set is used as a training sample to train the preset recognition model to obtain the target recognition model. Specifically, for each first feature data sequence, the first feature sequence corresponding to each first result data is a training sample of the fund text, which can also be considered a positive sample. Correspondingly, the first feature sequences corresponding to each second result data are training samples that are not fund text, and can be considered negative samples.

[0121] S103: Use the target recognition model to perform paragraph recognition on the text to be recognized.

[0122] The text to be identified is the target text that does not include prefix features. The prefix features are used to characterize the text type of the target text.

[0123] After training the target recognition model, it can be used to perform paragraph recognition on the text to be recognized. The text to be recognized is target text without prefix features, and the target text is fund text. Thus, the target recognition model can be used to perform batch paragraph recognition on fund text that does not include prefix features.

[0124] The text paragraph recognition method provided in this application first obtains a sample set, which includes a first sample set and a second sample set. Both the first and second sample sets contain text paragraphs of both the target text type and non-target text types. Then, a digitized training set is obtained based on the first and second sample sets, and a preset recognition model is trained using this training set to obtain a target recognition model. Finally, the target recognition model is used to perform paragraph recognition on the text to be recognized. The text to be recognized is target text that does not include prefix features, while prefix features are used to characterize the text type of the target text. This provides a method for text paragraph recognition without prefix features, avoiding the risk of misidentification and achieving high recognition accuracy, thus meeting the needs of text structure processing.

[0125] Based on the above embodiments, optionally, Figure 5 This is a flowchart illustrating another text paragraph recognition method provided in this application embodiment. This embodiment offers a possible implementation method for verifying and optimizing the target recognition model. Figure 5 As shown, this embodiment includes:

[0126] S301: Use the target recognition model to identify paragraphs in each verification sample in the verification sample set and determine whether the recognition results are correct.

[0127] A large number of validation samples were collected to form a validation sample set. A target recognition model was then used to identify paragraphs in each validation sample, and the accuracy of the recognition results was determined. The validation samples included multiple fund text paragraphs without prefix features or multiple non-fund text paragraphs.

[0128] S302: If not, adjust the word groups in the paragraph recognition feature set according to the recognition results.

[0129] If the recognition result is determined to be incorrect, then it is necessary to adjust each word in each word group of the paragraph recognition feature set used by the target recognition model according to the recognition result.

[0130] For example, if the target recognition model identifies a verification sample that should be fund text as non-fund text, or vice versa, it indicates an incorrect recognition result. Therefore, it is necessary to adjust the vocabulary in each word group of the paragraph recognition feature set used to determine the target recognition model based on the recognition results. For instance, when a verification sample of fund text is misidentified as non-fund text, it is necessary to determine whether any corresponding words representing fund text features have been collected in each word group of the paragraph recognition feature set, and to supplement the vocabulary in each word group. Conversely, when a verification sample of non-fund text is misidentified as fund text, some corresponding symbols belonging to non-fund text, such as special symbols, phone numbers, conjunctions, and prepositions, can be obtained and added to the vocabulary of each word group to supplement the corresponding features of non-fund text. This supplementation achieves the adjustment of each word group in the paragraph recognition feature set.

[0131] Typically, adjustments to each word group in the paragraph recognition feature set can be made until the prediction results obtained using the optimized target recognition model meet the preset threshold range. Conversely, if the recognition results are determined to be correct, the target recognition model can continue to be used for paragraph recognition of the text to be recognized.

[0132] S303: Obtain the second feature sequence corresponding to the preprocessed result data based on the adjusted paragraph recognition feature set, and form an optimized training set based on each second feature sequence.

[0133] After adjusting each word group in the paragraph recognition feature set to obtain a new paragraph recognition feature set, the adjusted paragraph recognition feature set is used to obtain the second feature sequence corresponding to the preprocessed result data. Each obtained second feature sequence is used as a subset to form an optimized training set.

[0134] The implementation method of this step is similar to that of the aforementioned step S1023, and will not be repeated here.

[0135] S304: Optimize the target recognition model based on the optimized training set until the prediction results meet the preset threshold range.

[0136] The prediction result is used to characterize the probability that the optimized target recognition model will produce a correct result.

[0137] The second feature sequences in the optimized training set are used as training samples for the target recognition model, and the model is trained continuously to optimize it. During training, the optimization process stops when the prediction result meets a preset threshold range.

[0138] The prediction results characterize the probability of obtaining a correct or incorrect result when using the optimized target recognition model for paragraph recognition. Tables 1 and 2 below list the prediction results for optimized positive and negative samples, respectively:

[0139] Table 1

[0140] Optimized positive samples Probability of the correct result Probability of incorrect results +1 0.973109954689844 0.0268900453101559 +1 0.925158246541171 0.0748417534588287 +1 0.949176513735674 0.0508234862643258 +1 0.998710149031425 0.0012898509685747 +1 0.973158442804405 0.026841557195595 +1 0.998710149031425 0.0012898509685747 +1 0.949212110483884 0.0507878895161161 +1 0.990665445840743 0.00933455415925666 +1 0.956670345329094 0.043329654670906

[0141] Table 2

[0142] Optimized negative samples Probability of incorrect results Probability of the correct result -1 0.00192156047395304 0.998078439526047 -1 1E-07 0.9999999 -1 0.00127480974702871 0.998725190252971 -1 0.23048184618887 0.76951815381113 -1 0.188962345823249 0.811037654176751 -1 0.00176927108642651 0.998230728913573 -1 0.277019145533432 0.722980854466568 -1 1E-07 0.9999999 -1 0.230452253540089 0.769547746459911

[0143] As shown in Tables 1 and 2, the larger the value of the correct recognition result, the more satisfactory the optimization of the target recognition model, and the higher the accuracy of paragraph recognition using the optimized target recognition model.

[0144] The text paragraph recognition method provided in this application uses a target recognition model to identify paragraphs in each verification sample in the verification sample set and determines whether the recognition result is correct. If incorrect, it further adjusts the words in each word group in the paragraph recognition feature set based on the recognition result to further optimize the target recognition model and improve its paragraph recognition accuracy. The text paragraph recognition method provided in this application not only provides a method for text paragraph recognition without prefix features, avoiding the risk of misidentification, but also has high recognition accuracy, meeting the needs of text structuring processing. Furthermore, it is easy to optimize and maintain, and can be widely applied in the field of text structuring technology.

[0145] The following are embodiments of the apparatus described in this application, which can be used to execute the corresponding method embodiments of this application. For details not disclosed in the apparatus embodiments of this application, please refer to the corresponding method embodiments of this application.

[0146] Figure 6This is a schematic diagram of the structure of a text paragraph recognition device provided in an embodiment of this application. Figure 6 As shown, the text paragraph recognition device 400 provided in this embodiment includes:

[0147] The sampling module 401 is used to acquire the sample set.

[0148] The sample set includes a first sample set and a second sample set, which respectively include text paragraphs of the target text type and non-target text type.

[0149] The processing module 402 is used to obtain a digital training set based on the first sample set and the second sample set, and to train a preset recognition model based on the digital training set to obtain a target recognition model.

[0150] The recognition module 403 is used to perform paragraph recognition on the text to be recognized using a target recognition model.

[0151] The text to be identified is the target text that does not include prefix features. The prefix features are used to characterize the text type of the target text.

[0152] Figure 7 This is a schematic diagram of the structure of a processing module provided in an embodiment of this application. Figure 7 As shown, the processing module 402 provided in this embodiment includes:

[0153] The first processing submodule 4021 is used to preprocess each text segment of the first sample set and the second sample set respectively, and to perform word segmentation on the preprocessed result data.

[0154] The second processing submodule 4022 is used to divide each word in the result data obtained from word segmentation into word groups according to the preset segmentation rules, and form a paragraph recognition feature set.

[0155] The third processing submodule 4023 is used to obtain the first feature sequence corresponding to the preprocessed result data based on the paragraph recognition feature set, so as to form a digital training set based on each first feature sequence.

[0156] In one possible design, the first processing submodule 4021 is specifically used for:

[0157] Filter out content in each text segment that is not related to the preset metadata and delete preset hyphens, so that the text segments after filtering and deletion operations are determined as the preprocessed result data.

[0158] In one possible design, the first processing submodule 4021 is also specifically used for:

[0159] Each first result data and each second result data is segmented using a preset word segmentation model to obtain the corresponding third result data and each fourth result data.

[0160] The preprocessed result data includes each first result data and each second result data, and the result data obtained from word segmentation includes each third result data and each fourth result data.

[0161] In one possible design, the third processing submodule 4023 is specifically used for:

[0162] The number of words in each word group included in each first result data and each second result data is obtained based on the paragraph recognition feature set, so as to obtain the first feature sequence corresponding to each first result data and each second result data.

[0163] Each first feature sequence is determined as a subset of the digital training set to obtain the digital training set.

[0164] In one possible design, the text paragraph recognition device 400 further includes a statistics and generation module. This statistics and generation module is used for:

[0165] The frequency of each word in each third and fourth result data is counted to obtain the mapping relationship between each third and fourth result data and each frequency.

[0166] Preset partitioning rules are generated based on the mapping relationship.

[0167] In one possible design, the text paragraph recognition device 400 further includes a verification and optimization module. This verification and optimization module is specifically used for:

[0168] The target recognition model is used to identify paragraphs in each verification sample in the verification sample set, and the accuracy of the recognition results is determined.

[0169] If not, adjust the word groups in the paragraph recognition feature set according to the recognition results;

[0170] The second feature sequence corresponding to the preprocessed result data is obtained based on the adjusted paragraph recognition feature set, and an optimized training set is formed based on each second feature sequence.

[0171] The target recognition model is optimized based on the training set until the prediction result meets the preset threshold range. The prediction result is used to characterize the probability that the optimized target recognition model's recognition result is correct or incorrect.

[0172] It is worth noting that the above Figure 6 and Figure 7The optional embodiments provide a text paragraph recognition device that can be used to perform each step of the text paragraph recognition method provided in any of the above embodiments. The specific implementation and technical effects are similar, and will not be described in detail here.

[0173] The device embodiments provided in this application are merely illustrative, and the module division is only a logical functional division. In actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system. The coupling between the modules can be achieved through some interfaces, which are usually electrical communication interfaces, but mechanical interfaces or other forms of interfaces are also possible. Therefore, the modules described as separate components may or may not be physically separated; they may be located in one place or distributed in different locations on the same or different devices.

[0174] Figure 8 This is a schematic diagram of the structure of an electronic device provided in this application. Figure 8 As shown, the electronic device 500 may include at least one processor 501 and a memory 502. Figure 8 The example shown is an electronic device using a processor.

[0175] The memory 502 is used to store the computer program of the processor 501. Specifically, the program may include program code, which includes computer operation instructions.

[0176] The memory 502 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0177] The processor 501 is configured to execute the computer program stored in the memory 502 to implement the steps of the text paragraph recognition method in the above method embodiments.

[0178] The processor 501 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0179] Optionally, the memory 502 can be either standalone or integrated with the processor 501. When the memory 502 is a device independent of the processor 501, the electronic device 500 may further include:

[0180] Bus 503 is used to connect processor 501 and memory 502. The bus can be an industry standard architecture (ISA) bus, a peripheral component (PCI) bus, or an extended industry standard architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc., but this does not mean there is only one bus or one type of bus.

[0181] Optionally, in a specific implementation, if the memory 502 and the processor 501 are integrated on a single chip, the memory 502 and the processor 501 can communicate through an internal interface.

[0182] This application also provides a computer-readable storage medium, which may include various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk. Specifically, the computer-readable storage medium stores a computer program. When at least one processor of an electronic device executes the computer program, the electronic device performs each step of the text paragraph recognition method provided in the above-described embodiments.

[0183] This application also provides a computer program product comprising a computer program stored in a readable storage medium. At least one processor of an electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program to cause the electronic device to perform the various steps of the text paragraph recognition method provided in the various embodiments described above.

[0184] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.

[0185] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A text paragraph recognition method, characterized in that, include: Obtain a sample set, which includes a first sample set and a second sample set. The first sample set and the second sample set respectively include text paragraphs of target text type and non-target text type. The target text type is fund text and the non-target text type is non-fund text. Each text segment of the first sample set and the second sample set is preprocessed to obtain each first result data and each second result data. Then, each first result data and each second result data is segmented using a preset word segmentation model to obtain the corresponding third result data and each fourth result data. The frequency of each word in each third result data and each fourth result data is statistically analyzed to obtain the mapping relationship between each third result data and each fourth result data and each frequency. A preset partitioning rule is generated based on the mapping relationship; and each word is divided into word groups according to the preset partitioning rule to form paragraph recognition feature sets corresponding to the first sample set and the second sample set respectively; a first feature sequence is obtained corresponding to each paragraph recognition feature set to form a digital training set based on each first feature sequence, wherein the first feature sequence includes the number of words in each word group corresponding to the first sample set or the second sample set; and a preset recognition model is trained based on the digital training set to obtain a target recognition model; The target recognition model is used to perform batch paragraph recognition on the text to be recognized. The text to be recognized is a fund text that does not include prefix features. The prefix features are used to characterize the text type of the target text.

2. The text paragraph recognition method according to claim 1, characterized in that, The preprocessing of each text segment in the first sample set and the second sample set includes: Content unrelated to preset metadata is filtered out from each text segment, and preset hyphens are deleted, so that each text segment after filtering and deletion operations is determined as the preprocessed result data.

3. The text paragraph recognition method according to claim 1, characterized in that, The step of obtaining the first feature sequence corresponding to each paragraph recognition feature set, and forming a digital training set based on each first feature sequence, includes: The number of words in the vocabulary groups included in each of the first result data and each of the second result data is obtained according to the paragraph recognition feature set, so as to obtain the first feature sequence corresponding to each of the first result data and each of the second result data; Each of the first feature sequences is determined as a subset of the digital training set to obtain the digital training set.

4. The text paragraph recognition method according to any one of claims 1-3, characterized in that, Also includes: The target recognition model is used to identify paragraphs in each verification sample in the verification sample set, and the accuracy of the recognition results is determined. If not, adjust the word groups in the paragraph recognition feature set according to the recognition results; The second feature sequence corresponding to the preprocessed result data is obtained based on the adjusted paragraph recognition feature set, and an optimized training set is formed based on each second feature sequence. The target recognition model is optimized based on the optimized training set until the prediction result meets the preset threshold range. The prediction result is used to characterize the probability that the optimized target recognition model's recognition result is a correct result or an incorrect result.

5. A text paragraph recognition device, characterized in that, include: A sampling module is used to acquire a sample set, which includes a first sample set and a second sample set. The first sample set and the second sample set respectively include text paragraphs of target text type and non-target text type. The target text type is fund text and the non-target text type is non-fund text. The processing module is used to preprocess each text segment of the first sample set and the second sample set to obtain each first result data and each second result data, and to perform word segmentation processing on each first result data and each second result data through a preset word segmentation model to obtain the corresponding third result data and each fourth result data. The frequency of each word in each third result data and each fourth result data is statistically analyzed to obtain the mapping relationship between each third result data and each fourth result data and each frequency. A preset partitioning rule is generated based on the mapping relationship; and each word is divided into word groups according to the preset partitioning rule to form paragraph recognition feature sets corresponding to the first sample set and the second sample set respectively; a first feature sequence is obtained corresponding to each paragraph recognition feature set to form a digital training set based on each first feature sequence, wherein the first feature sequence includes the number of words in each word group corresponding to the first sample set or the second sample set; and a preset recognition model is trained based on the digital training set to obtain a target recognition model; The recognition module is used to perform batch paragraph recognition of the text to be recognized using the target recognition model. The text to be recognized is a fund text that does not include prefix features. The prefix features are used to characterize the text type of the target text as the target text type.

6. An electronic device, characterized in that, include: processor; as well as, Memory for storing the computer program of the processor; The processor is configured to perform the text paragraph recognition method according to any one of claims 1 to 4 by executing the computer program.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the text paragraph recognition method according to any one of claims 1 to 4.

Images