Document analysis method and system based on natural language processing
Through natural language processing technology, the AI large language model and two-dimensional plane coordinate system are used to map entity relationships, dynamically optimize entity relationships in documents, and generate two-dimensional framework diagrams, solving the problem of complex document comprehension errors and improving information extraction and ability Reading effect.
Patent Information
- Application Number
- CN202410994271.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-23
- Publication Date
- 2024-11-12
AI Technical Summary
When the prior art deals with documents with complex structures and logic that are not strictly arranged, it is difficult to accurately understand the correlation between knowledge points, resulting in readers' understanding errors.
Using a document analysis method based on natural language processing, text preprocessing and entity information extraction are carried out through AI large language model, a two-dimensional plane coordinate system is established, the relationship characteristics of entity information and connection vocabulary are extracted, the entity relationship strength is dynamically optimized, and the two-dimensional Frame overview diagram.
It improves the accuracy and depth of information extraction, enhances the readability and understanding of text content, and helps users better understand the deep connections of complex texts.
Smart Images

Figure CN118940743A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of document processing, and is a document analysis method and system based on natural language processing. Background Art
[0002] Currently, the content structure of Word or PDF documents is mainly distinguished by the level of chapter titles, or summarized by the document reader after reading the entire document. If the document is very long, or the relationship between the knowledge points in the document is very complex, or the author of the document content does not strictly follow a certain logical arrangement to write the document, it will be difficult for the reader to understand the content of the document, and even misunderstandings may occur.
[0003] In the existing disclosed invention technologies, for example, the patent with application publication number CN117708315A discloses a method and system for simplifying long documents based on hierarchical generation, including screening the acquired documents, using a large language model to provide an interface to build a long document simplification prompt framework; performing syntactic analysis on the input document, dividing the document into multiple paragraphs, storing them in a list, performing an initial paragraph-level simplification, and performing a secondary lexical-level simplification on the document obtained after the initial paragraph-level simplification; finally simplifying the expression in the sentence through the constructed prompt framework to obtain the long document simplification processing result, and verifying the simplification processing effect.
[0004] The above patent lacks the extraction of connecting words between entity information. Although a large language model is used to provide an interface to build a long document simplification prompt framework, there are still certain limitations on the semantic understanding of the text. The simplification process may not fully consider the logical relationship and semantic coherence between sentences, resulting in the simplification results being not natural or coherent enough, which will cause readers' understanding deviations. Summary of the invention
[0005] The technical problem to be solved by the present invention is that in the prior art, the correlation between knowledge points in a document is complex or the author of the document content does not strictly follow a certain logical arrangement method to write the document, which makes it difficult for the reader to understand the content of the document and even causes misunderstandings. A document analysis method and system based on natural language processing are proposed.
[0006] In order to achieve the above object, the technical solution of the document analysis method based on natural language processing of the present invention includes the following steps:
[0007] S1: Perform text preprocessing on the document to be analyzed, build and train an AI big language model, and use the trained AI big language model to segment the document to be analyzed into a group of independent text segments according to paragraphs;
[0008] S2: By setting entity information prompt words for the AI large language model, the entity information in the document to be analyzed is identified, and the convolution features of various types of entity information are extracted to complete the entity information extraction process of the document to be analyzed;
[0009] S3: Establish a two-dimensional plane coordinate system, extract the occurrence count and location data of each entity information in the document to be analyzed, evaluate the initial relationship strength value between each entity and the entity information center, and annotate each entity information into the plane coordinate system;
[0010] S4: Identify the connection words in the document to be analyzed, synchronously record the connection feature data of each connection word, and confirm the relationship optimization coefficient between each entity information according to the connection feature data feedback;
[0011] S5: extract the final relationship strength value data between each entity information after the feedback confirmation processing is completed, arrange the entity information in order according to the final relationship strength value data, and output a two-dimensional framework overview diagram of the document to be analyzed.
[0012] Specifically, S1 includes the following specific steps:
[0013] S11: Perform text preprocessing on the document to be analyzed, remove special symbols, punctuation marks, HTML tags and noise information in the text, and screen and filter stop words;
[0014] S12: Build and train an AI big language model. The trained AI big language model divides the document to be analyzed into a group of independent text segments by paragraphs, and divides each independent text segment into a sequence of words and phrases.
[0015] Specifically, in S2, the entity information prompt words include: names of people, places, names of organizations, time, quantity, technical terms, names of events and other categories of nouns, with a total of R categories of entity information;
[0016] S2 includes the following specific steps:
[0017] S21: Map the words in the document to be analyzed into a low-dimensional vector space through the word embedding model, and use the Skip-gram algorithm to train the word embedding model to complete the semantic learning between word and phrase sequences;
[0018] S22: Establish a BERT (Bidirectional Encoder Representations from Transformers) model to convert the document to be analyzed into a character and number sequence X = (x1, x2, ..., x i , ..., x I) and input it into the BERT model to obtain the expression sequence of each character in the document to be analyzed as Y = (y1, y2, ..., y i , ..., y I ), where the total number of characters in the document to be analyzed is I, and i is a subscript, indicating the i-th character of the document to be analyzed;
[0019] S23: According to the entity information prompt words set for the BERT model, identify and calculate the feature vectors of each category of entity information in the document to be analyzed. The feature vector of each category of entity information is calculated as follows: Among them, T r is the feature vector of the r-th type of entity information, d1, d2 are both subscripts in the character expression sequence Y, d1, d2∈{1, 2, …, i, …I}, d1≠d2;
[0020] S24: Extract the feature vector of each category entity information to form a feature vector set of entity information: ST = (T1, T2, ..., T R );
[0021] S25: Set the feature vector of entity information as ST = (T1, T2, ..., T R ) is input into the convolution layer and entity features are extracted, and the extracted features are input into the pooling layer to output the final entity representation of various entity information, where the convolution kernel size is 10×10 and the height of each convolution kernel is h.
[0022] Specifically, S3 includes the following specific steps:
[0023] S31: extracting the entity information that appears most frequently in the document as the entity information center, and establishing a two-dimensional plane coordinate system with the entity information center as the origin of the two-dimensional coordinate system;
[0024] S32: extracting the occurrence number data and the occurrence position data of each type of entity information in the document to be analyzed at the same time;
[0025] S33: Evaluate the initial relationship strength value between each entity information and the entity information center. The evaluation strategy is as follows: Among them, r,u represents the u-th entity information in the r-th entity information, which is the entity information that appears most times in the document, that is, the entity information center; r′,v represents the v-th entity information in the r′-th entity information;
[0026] is the subscript, is the initial relationship strength value between the vth entity information in the r′th entity information and the entity information center; is the number of characters between the vth entity information in the r′th entity information and the entity information center;
[0027] CS r′,v , are the number of occurrences of the v-th entity information and the entity information center in the r′-th type of entity information in the document to be analyzed;
[0028] S34: extracting the initial relationship strength value between each entity and the entity information center, and marking each entity information into the plane coordinate system.
[0029] Specifically, S4 includes the following specific steps:
[0030] S41: Identify the connection words in the document to be analyzed, and synchronously record the connection features of each connection word; the connection features of the connection words include: comparative connection features and progressive connection features;
[0031] S42: extracting progressive connection words in the document to be analyzed, and performing progressive category judgment, wherein the progressive category includes: causal category and transition category, wherein the causal category is positively progressive, and the transition category is negatively progressive.
[0032] Specifically, the S4 further includes the following specific steps:
[0033] S43: Filtering two entity information closest to the connecting words in the document to be analyzed;
[0034] When the connecting word between two entity information is a comparative connecting word, according to the appearance position of the entity information in the document to be analyzed, the entity information appearing before the comparative connecting word is the front entity information, and the entity information appearing after the comparative connecting word is the back entity information, and the first type of optimization strategy is implemented;
[0035] When the connecting word between two entity information is a positive progressive connecting word, according to the appearance position of the entity information in the document to be analyzed, the entity information appearing before the comparison connecting word is the front entity information, and the entity information appearing after the comparison connecting word is the back entity information, and the second type of optimization strategy is implemented;
[0036] When the connective between two entity information is a negative connective, the entity information that appears before the comparison connective is the latter entity information, and the entity information that appears after the comparison connective is the former entity information, and the third type of optimization strategy is implemented;
[0037] S44: Calculate the k-th relationship optimization coefficient α between two entity information k , the specific calculation strategy is as follows:
[0038]
[0039] Wherein, ΔL1 and ΔL2 are the number of characters between the preceding entity information and the succeeding entity information and the connecting words respectively;
[0040] are the optimization factors of the k-th optimization strategy,
[0041] Specifically, in S5, the final relationship strength value between each entity information after the feedback confirmation process is completed is the product of the relationship optimization coefficient and the initial relationship strength value;
[0042] The pounds of the connection lines between the entity information in the two-dimensional framework overview diagram of the document to be analyzed match the final relationship strength value from large to small.
[0043] In addition, the document analysis system based on natural language processing of the present invention includes the following modules:
[0044] Text processing module, entity information recognition module, relationship strength value calculation module, relationship optimization module and framework generation module;
[0045] The text processing module is used to perform text preprocessing on the document to be analyzed, build and train an AI large language model, and divide the document to be analyzed into a group of independent text segments according to paragraphs through the trained AI large language model;
[0046] The entity information recognition module sets entity information prompt words for the AI large language model to identify entity information in the document to be analyzed, and extracts convolutional features of various types of entity information to complete the entity information extraction process of the document to be analyzed;
[0047] The relationship strength value calculation module is used to establish a two-dimensional plane coordinate system, extract the occurrence number data and the occurrence position data of each entity information in the document to be analyzed, evaluate the initial relationship strength value of each entity and the entity information center, and mark each entity information into the plane coordinate system;
[0048] The relationship optimization module is used to identify the connection words in the document to be analyzed, synchronously record the connection feature data of each connection word, and confirm the relationship optimization coefficient between each entity information according to the connection feature data feedback;
[0049] The framework generation module is used to extract the final relationship strength value data between each entity information and the entity center after completing the feedback confirmation processing, arrange the entity information in order according to the final relationship strength value data, and output a two-dimensional framework overview diagram of the document to be analyzed.
[0050] A storage medium stores instructions, and when a computer reads the instructions, the computer executes the document analysis method based on natural language processing.
[0051] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned document analysis method based on natural language processing when executing the computer program.
[0052] Compared with the prior art, the technical effects of the present invention are as follows:
[0053] 1. The present invention can more accurately identify and extract key information in text by using AI large language models for text preprocessing and entity information extraction. AI models can understand and analyze complex text structures, thereby improving the accuracy and depth of information extraction.
[0054] 2. The implementation of steps S3 and S4 of the present invention helps to visualize and understand the interrelationships and influences between entities by mapping entity relationships in a two-dimensional plane coordinate system and optimizing these relationships. This method can reveal the implicit structure and dynamics in the text and help users better understand the deep connections of the text content.
[0055] 3. The present invention can dynamically optimize the strength of the relationship between entities through feedback confirmation processing (steps S4 and S5), making the output entity relationship diagram more accurate and useful. This dynamic adjustment mechanism improves the adaptability and accuracy of the analysis results.
[0056] 4. The present invention displays complex text information in the form of a two-dimensional framework overview diagram, which not only improves the readability and comprehensibility of the information, but also facilitates users to quickly grasp and further analyze the information. BRIEF DESCRIPTION OF THE DRAWINGS
[0057] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.
[0058] in:
[0059] Figure 1 A schematic diagram of the process of the document analysis method based on natural language processing of the present invention;
[0060] Figure 2 It is a structural diagram of the document analysis system based on natural language processing of the present invention. DETAILED DESCRIPTION
[0061] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.
[0062] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.
[0063] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.
[0064] Embodiment 1:
[0065] like Figure 1 As shown, the document analysis method based on natural language processing in an embodiment of the present invention is as follows: Figure 1 As shown, the specific steps are as follows:
[0066] S1: Perform text preprocessing on the document to be analyzed, build and train an AI big language model, and use the trained AI big language model to segment the document to be analyzed into a group of independent text segments according to paragraphs;
[0067] S1 includes the following specific steps:
[0068] S11: Perform text preprocessing on the document to be analyzed, remove special symbols, punctuation marks, HTML tags and noise information in the text, and screen and filter stop words;
[0069] S12: Build and train an AI big language model. The trained AI big language model divides the document to be analyzed into a group of independent text segments by paragraphs, and divides each independent text segment into a sequence of words and phrases.
[0070] S2: By setting entity information prompt words for the AI large language model, the entity information in the document to be analyzed is identified, and the convolution features of various types of entity information are extracted to complete the entity information extraction process of the document to be analyzed;
[0071] In S2, the entity information prompt words include: names of people, places, names of organizations, time, quantity, technical terms, names of events and other categories of nouns, with a total of R categories of entity information;
[0072] S2 includes the following specific steps:
[0073] S21: Map the words in the document to be analyzed into a low-dimensional vector space through the word embedding model, and use the Skip-gram algorithm to train the word embedding model to complete the semantic learning between word and phrase sequences;
[0074] Exemplarily, in this embodiment, the word embedding model is a word2vec model.
[0075] S22: Establish a BERT (Bidirectional Encoder Representations from Transformers) model to convert the document to be analyzed into a character and number sequence X = (x1, x2, ..., x i , ..., x I ) and input it into the BERT model to obtain the expression sequence of each character in the document to be analyzed as Y = (y1, y2, ...y i , ..., y I ), where the total number of characters in the document to be analyzed is I, and i is a subscript, indicating the i-th character of the document to be analyzed;
[0076] S23: According to the entity information prompt words set for the BERT model, identify and calculate the feature vectors of each category of entity information in the document to be analyzed. The feature vector of each category of entity information is calculated as follows: Among them, T r is the feature vector of the r-th type of entity information, d1, d2 are both subscripts in the character expression sequence Y, d1, d2∈{1, 2, …, i, …, I}, d1≠d2;
[0077] S24: Extract the feature vector of each category entity information to form a feature vector set of entity information: ST = (T1, T2, ..., T R );
[0078] S25: Set the feature vector of entity information as ST = (T1, T2, ..., T R ) is input into the convolution layer and entity features are extracted, and the extracted features are input into the pooling layer to output the final entity representation of various entity information, where the convolution kernel size is 10×10 and the height of each convolution kernel is h.
[0079] S3: Establish a two-dimensional plane coordinate system, extract the occurrence count and location data of each entity information in the document to be analyzed, evaluate the initial relationship strength value between each entity and the entity information center, and annotate each entity information into the plane coordinate system;
[0080] S3 includes the following specific steps:
[0081] S31: extracting the entity information that appears most frequently in the document as the entity information center, and establishing a two-dimensional plane coordinate system with the entity information center as the origin of the two-dimensional coordinate system;
[0082] S32: extracting the occurrence number data and the occurrence position data of each type of entity information in the document to be analyzed at the same time;
[0083] S33: Evaluate the initial relationship strength value between each entity information and the entity information center. The evaluation strategy is as follows: Among them, r,u represents the u-th entity information in the r-th entity information, which is the entity information that appears most times in the document, that is, the entity information center; r′,v represents the v-th entity information in the r′-th entity information;
[0084] is the subscript, is the initial relationship strength value between the vth entity information in the r′th entity information and the entity information center; is the number of characters between the vth entity information in the r′th entity information and the entity information center;
[0085] CS r′,v , are the number of occurrences of the v-th entity information and the entity information center in the r′-th type of entity information in the document to be analyzed;
[0086] S34: extracting the initial relationship strength value between each entity and the entity information center, and marking each entity information into the plane coordinate system.
[0087] S4: Identify the connection words in the document to be analyzed, synchronously record the connection feature data of each connection word, and confirm the relationship optimization coefficient between each entity information according to the connection feature data feedback;
[0088] S4 includes the following specific steps:
[0089] S41: Identify the connection words in the document to be analyzed, and synchronously record the connection features of each connection word; the connection features of the connection words include: comparative connection features and progressive connection features;
[0090] S42: extracting progressive connection words in the document to be analyzed, and performing progressive category judgment, wherein the progressive category includes: causal category and transition category, wherein the causal category is positively progressive, and the transition category is negatively progressive.
[0091] The S4 further comprises the following specific steps:
[0092] S43: Filtering two entity information closest to the connecting words in the document to be analyzed;
[0093] When the connecting word between two entity information is a comparative connecting word, according to the appearance position of the entity information in the document to be analyzed, the entity information appearing before the comparative connecting word is the front entity information, and the entity information appearing after the comparative connecting word is the back entity information, and the first type of optimization strategy is implemented;
[0094] When the connecting word between two entity information is a positive progressive connecting word, according to the appearance position of the entity information in the document to be analyzed, the entity information appearing before the comparison connecting word is the front entity information, and the entity information appearing after the comparison connecting word is the back entity information, and the second type of optimization strategy is implemented;
[0095] When the connective between two entity information is a negative connective, the entity information that appears before the comparison connective is the latter entity information, and the entity information that appears after the comparison connective is the former entity information, and the third type of optimization strategy is implemented;
[0096] S44: Calculate the k-th relationship optimization coefficient α between two entity information k , the specific calculation strategy is as follows:
[0097]
[0098] Wherein, ΔL1 and ΔL2 are the number of characters between the preceding entity information and the succeeding entity information and the connecting words respectively;
[0099] are the optimization factors of the k-th optimization strategy,
[0100] S5: extract the final relationship strength value data between each entity information after the feedback confirmation processing is completed, arrange the entity information in order according to the final relationship strength value data, and output a two-dimensional framework overview diagram of the document to be analyzed.
[0101] In S5, the final relationship strength value between each entity information after the feedback confirmation process is completed is the product of the relationship optimization coefficient and the initial relationship strength value;
[0102] The pounds of the connection lines between the entity information in the two-dimensional framework overview diagram of the document to be analyzed match the final relationship strength value from large to small.
[0103] Embodiment 2:
[0104] like Figure 2 As shown, the document analysis system based on natural language processing in an embodiment of the present invention is as follows: Figure 2 As shown, it includes the following modules:
[0105] Text processing module, entity information recognition module, relationship strength value calculation module, relationship optimization module and framework generation module;
[0106] The text processing module is used to perform text preprocessing on the text to be analyzed, build and train an AI large language model, and divide the text to be analyzed into a group of independent text segments by paragraphs through the trained AI large language model;
[0107] The entity information recognition module sets entity information prompt words for the AI large language model to identify entity information in the document to be analyzed, and extracts convolutional features of various types of entity information to complete the entity information extraction process of the document to be analyzed;
[0108] The relationship strength value calculation module is used to establish a two-dimensional plane coordinate system, extract the occurrence number data and the occurrence position data of each entity information in the document to be analyzed, evaluate the initial relationship strength value of each entity and the entity information center, and mark each entity information into the plane coordinate system;
[0109] The relationship optimization module is used to identify the connection words in the document to be analyzed, synchronously record the connection feature data of each connection word, and confirm the relationship optimization coefficient between each entity information according to the connection feature data feedback;
[0110] The framework generation module is used to extract the final relationship strength value data between each entity information and the entity center after completing the feedback confirmation processing, arrange the entity information in order according to the final relationship strength value data, and output a two-dimensional framework overview diagram of the document to be analyzed.
[0111] Embodiment three:
[0112] This embodiment provides an electronic device, including: a processor and a memory, wherein the memory stores a computer program that can be called by the processor;
[0113] The processor executes the above-mentioned document analysis method based on natural language processing by calling the computer program stored in the memory.
[0114] The electronic device may have relatively large differences due to different configurations or performances, and may include one or more processors (Central Processing Units, CPU) and one or more memories, wherein the memory stores at least one computer program, which is loaded and executed by the processor to implement the document analysis method based on natural language processing provided by the above method embodiment. The electronic device may also include other components for implementing the functions of the device, for example, the electronic device may also have components such as a wired or wireless network interface and an input and output interface to input and output data. This embodiment will not be described in detail here.
[0115] Embodiment 4:
[0116] This embodiment provides a computer-readable storage medium having a rewritable computer program stored thereon;
[0117] When the computer program runs on a computer device, the computer device executes the above-mentioned document analysis method based on natural language processing.
[0118] For example, the computer readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, and an optical data storage device, etc.
[0119] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.
[0120] It should be understood that determining B based on A does not mean determining B only based on A. B can also be determined based on A and / or other information.
[0121] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When a computer instruction or computer program is loaded or executed on a computer, a process or function according to an embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center through a wired network or / and a wireless network. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.
[0122] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the present invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.
[0123] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0124] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only one, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0125] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0126] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0127] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.
[0128] In summary, compared with the prior art, the technical effects of the present invention are as follows:
[0129] 1. The present invention can more accurately identify and extract key information in text by using AI large language models for text preprocessing and entity information extraction. AI models can understand and analyze complex text structures, thereby improving the accuracy and depth of information extraction.
[0130] 2. The implementation of steps S3 and S4 of the present invention helps to visualize and understand the interrelationships and influences between entities by mapping entity relationships in a two-dimensional plane coordinate system and optimizing these relationships. This method can reveal the implicit structure and dynamics in the text and help users better understand the deep connections of the text content.
[0131] 3. The present invention can dynamically optimize the strength of the relationship between entities through feedback confirmation processing (steps S4 and S5), making the output entity relationship diagram more accurate and useful. This dynamic adjustment mechanism improves the adaptability and accuracy of the analysis results.
[0132] 4. The present invention displays complex text information in the form of a two-dimensional framework overview diagram, which not only improves the readability and comprehensibility of the information, but also facilitates users to quickly grasp and further analyze the information.
[0133] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.
Claims
1. A document analysis method based on natural language processing, characterized in that: The method comprises the following specific steps: S1: Perform text preprocessing on the document to be analyzed, build and train an AI big language model, and use the trained AI big language model to segment the document to be analyzed into a group of independent text segments according to paragraphs; S2: By setting entity information prompt words for the AI large language model, the entity information in the document to be analyzed is identified, and the convolution features of various types of entity information are extracted to complete the entity information extraction process of the document to be analyzed; S3: Establish a two-dimensional plane coordinate system, extract the occurrence count and location data of each entity information in the document to be analyzed, evaluate the initial relationship strength value between each entity and the entity information center, and annotate each entity information into the plane coordinate system; S4: Identify the connection words in the document to be analyzed, synchronously record the connection feature data of each connection word, and confirm the relationship optimization coefficient between each entity information according to the connection feature data feedback; S5: extract the final relationship strength value data between each entity information after the feedback confirmation processing is completed, arrange the entity information in order according to the final relationship strength value data, and output a two-dimensional framework overview diagram of the document to be analyzed.
2. The document analysis method based on natural language processing according to claim 1, characterized in that: S1 includes the following specific steps: S11: Perform text preprocessing on the document to be analyzed, remove special symbols, punctuation marks, HTML tags and noise information in the text, and screen and filter stop words; S12: Build and train an AI big language model. The trained AI big language model divides the document to be analyzed into a group of independent text segments by paragraphs, and divides each independent text segment into a sequence of words and phrases.
3. The document analysis method based on natural language processing according to claim 2 is characterized in that: In S2, the entity information prompt words include: names of people, places, names of organizations, time, quantity, technical terms, names of events and other categories of nouns, with a total of R categories of entity information; S2 includes the following specific steps: S21: Map the words in the document to be analyzed into a low-dimensional vector space through the word embedding model, and use the Skip-gram algorithm to train the word embedding model to complete the semantic learning between word and phrase sequences; S22: Establish a BERT model and convert the document to be analyzed into a character and number sequence X = (x1, x2, ..., x i , ..., x I ) and input it into the BERT model to obtain the expression sequence of each character in the document to be analyzed as Y = (y1, y2, ..., y i , ..., y I ), where the total number of characters in the document to be analyzed is I, and i is a subscript, indicating the i-th character of the document to be analyzed; S23: According to the entity information prompt words set for the BERT model, identify and calculate the feature vectors of each category of entity information in the document to be analyzed. The feature vector of each category of entity information is calculated as follows: Among them, T r is the feature vector of the r-th entity information, d1 and d2 are both subscripts in the character expression sequence Y, d1, d2∈{1, 2,…,i,…I}, d1≠d2; S24: Extract the feature vector of each category entity information to form a feature vector set of entity information: ST = (T1, T2, ..., T R ); S25: Set the feature vector of entity information as ST = (T1, T2, ..., T R ) is input into the convolution layer and entity features are extracted, and the extracted features are input into the pooling layer to output the final entity representation of various entity information, where the convolution kernel size is 10×10 and the height of each convolution kernel is h.
4. The document analysis method based on natural language processing according to claim 3 is characterized in that: S3 includes the following specific steps: S31: extracting the entity information that appears most frequently in the document as the entity information center, and establishing a two-dimensional plane coordinate system with the entity information center as the origin of the two-dimensional coordinate system; S32: extracting the occurrence number data and the occurrence position data of each type of entity information in the document to be analyzed at the same time; S33: Evaluate the initial relationship strength value between each entity information and the entity information center. The evaluation strategy is as follows: Among them, r,u represents the u-th entity information in the r-th entity information, which is the entity information that appears most times in the document, that is, the entity information center; r′,v represents the v-th entity information in the r′-th entity information; is the subscript, is the initial relationship strength value between the vth entity information in the r′th entity information and the entity information center; is the number of characters between the vth entity information in the r′th entity information and the entity information center; CS r′,v , is the number of occurrences of the v-th entity information and the entity information center in the r′-th entity information in the document to be analyzed; S34: extracting the initial relationship strength value between each entity and the entity information center, and marking each entity information into the plane coordinate system.
5. The document analysis method based on natural language processing according to claim 4 is characterized in that: S4 includes the following specific steps: S41: Identify the connection words in the document to be analyzed, and synchronously record the connection features of each connection word; the connection features of the connection words include: comparative connection features and progressive connection features; S42: extracting progressive connection words in the document to be analyzed, and performing progressive category judgment, wherein the progressive category includes: causal category and transition category, wherein the causal category is positively progressive, and the transition category is negatively progressive.
6. The document analysis method based on natural language processing according to claim 5 is characterized in that: The S4 further comprises the following specific steps: S43: Filtering two entity information closest to the connecting words in the document to be analyzed; When the connecting word between two entity information is a comparative connecting word, according to the position of the entity information in the document to be analyzed, the entity information appearing before the comparative connecting word is the front entity information, and the entity information appearing after the comparative connecting word is the back entity information, and the first type of optimization strategy is implemented; When the connecting word between two entity information is a positive progressive connecting word, according to the appearance position of the entity information in the document to be analyzed, the entity information appearing before the comparison connecting word is the front entity information, and the entity information appearing after the comparison connecting word is the back entity information, and the second type of optimization strategy is implemented; When the connective between two entity information is a negative connective, the entity information that appears before the comparison connective is the latter entity information, and the entity information that appears after the comparison connective is the former entity information, and the third type of optimization strategy is implemented; S44: Calculate the k-th relationship optimization coefficient α between two entity information k , the specific calculation strategy is as follows: Wherein, ΔL1 and ΔL2 are the number of characters between the preceding entity information and the succeeding entity information and the connecting words respectively; are the optimization factors of the k-th optimization strategy, 7. The document analysis method based on natural language processing according to claim 6 is characterized in that: In S5, the final relationship strength value between each entity information after the feedback confirmation process is completed is the product of the relationship optimization coefficient and the initial relationship strength value; The pounds of the connection lines between the entity information in the two-dimensional framework overview diagram of the document to be analyzed match the final relationship strength value from large to small.
8. A document analysis system based on natural language processing, which is used to implement the document analysis method based on natural language processing as described in any one of claims 1 to 7, characterized in that: The system includes the following modules: Text processing module, entity information recognition module, relationship strength value calculation module, relationship optimization module and framework generation module; The text processing module is used to perform text preprocessing on the document to be analyzed, build and train an AI large language model, and divide the text to be analyzed into a group of independent text segments by paragraphs through the trained AI large language model; The entity information recognition module identifies entity information in the document to be analyzed by setting entity information prompt words for the AI large language model, and extracts convolutional features of various types of entity information to complete the entity information extraction process of the document to be analyzed; The relationship strength value calculation module is used to establish a two-dimensional plane coordinate system, extract the occurrence number data and the occurrence position data of each entity information in the document to be analyzed, evaluate the initial relationship strength value between each entity and the entity information center, and mark each entity information into the plane coordinate system; The relationship optimization module is used to identify the connection words in the document to be analyzed, synchronously record the connection feature data of each connection word, and confirm the relationship optimization coefficient between each entity information according to the connection feature data feedback; The framework generation module is used to extract the final relationship strength value data between each entity information and the entity center after completing the feedback confirmation processing, arrange the entity information in order according to the final relationship strength value data, and output a two-dimensional framework overview diagram of the document to be analyzed.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the document analysis method based on natural language processing as described in any one of claims 1 to 7 is implemented.
10. An electronic device, characterized in that: include: A memory for storing instructions; A processor is used to execute the instructions so that the device performs operations to implement the document analysis method based on natural language processing as described in any one of claims 1 to 7.
