Document reference relation identification method and related equipment

Abstract

The invention provides a document reference relationship identification method and related equipment. The method comprises the following steps: performing data preprocessing and information extraction on a to-be-processed document to obtain an intermediate document in a preset format; identifying an entity reference relationship between entities in the intermediate document and the entities; verifying the entity reference relationship to obtain a verified target reference relationship; and generating a reference relation graph of the to-be-processed document based on the target reference relation and the corresponding target entity.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing, and in particular to a method for identifying document reference relationships and related equipment. Background Art

[0002] Standards are important reference materials in various industries and fields, and there are complex reference relationships between standards. Traditional standard document reference management often relies on manual work and is prone to omissions and errors. Summary of the invention

[0003] The present disclosure proposes a document reference relationship identification method and related equipment to solve the technical problem of low accuracy of standard document reference association to a certain extent.

[0004] In a first aspect, the present disclosure provides a method for identifying a document reference relationship, comprising:

[0005] Perform data preprocessing and information extraction on the documents to be processed to obtain intermediate documents in a preset format;

[0006] identifying entities in the intermediate document and entity reference relationships between the entities;

[0007] Verifying the entity reference relationship to obtain a verified target reference relationship;

[0008] Based on the target reference relationship and the corresponding target entity, a reference relationship graph of the document to be processed is generated.

[0009] In a second aspect of the present disclosure, a device for identifying a document reference relationship is provided, comprising:

[0010] The extraction module is used to perform data preprocessing and information extraction on the documents to be processed to obtain intermediate documents in a preset format;

[0011] An identification module, used for identifying entities in the intermediate document and entity reference relationships between the entities;

[0012] A verification module, used to verify the entity reference relationship and obtain a verified target reference relationship;

[0013] A graph module is used to generate a reference relationship graph of the document to be processed based on the target reference relationship and the corresponding target entity.

[0014] In a third aspect of the present disclosure, an electronic device is provided, comprising one or more processors, a memory; and one or more programs, wherein the one or more programs are stored in the memory and executed by the one or more processors, and the programs include instructions for executing the method described in the first aspect.

[0015] According to a fourth aspect of the present disclosure, a non-volatile computer-readable storage medium containing a computer program is provided. When the computer program is executed by one or more processors, the processors execute the method described in the first aspect.

[0016] In a fifth aspect of the present disclosure, a computer program product is provided, comprising computer program instructions, which, when executed on a computer, enable the computer to execute the method described in the first aspect.

[0017] As can be seen from the above, the present disclosure provides a method for identifying document reference relationships and related equipment, which use natural language processing and graph database technology to realize the automatic extraction and visualization of reference association relationships between standard documents. It solves the complexity of reference relationships between standard documents and the problem of visual analysis and query. By constructing a knowledge graph of standard documents, semantic modeling and automatic extraction of reference relationships between standards are realized. The knowledge graph is laid out using an improved force-directed graph algorithm to realize intuitive visualization of reference association relationships between standard documents. The system is suitable for fields such as standard formulation, standard retrieval, and standard knowledge management. It is suitable for fields such as standard formulation, standard retrieval, and standard knowledge management. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present disclosure or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 A schematic diagram of an identification architecture for document reference relationships according to an embodiment of the present disclosure.

[0020] Figure 2 The figure is a schematic diagram of the hardware structure of an exemplary electronic device according to an embodiment of the present disclosure.

[0021] Figure 3 The present invention is a flowchart of a method for identifying document reference relationships according to an embodiment of the present invention.

[0022] Figure 4 A schematic diagram of a method for identifying document reference relationships according to an embodiment of the present disclosure.

[0023] Figure 5 A schematic diagram of the knowledge graph of an embodiment of the present disclosure.

[0024] Figure 6 A schematic diagram of a device for identifying document reference relationships according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] In order to make the objectives, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0026] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the embodiments of the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0027] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0028] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0029] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0030] Figure 1 A schematic diagram of the document reference relationship recognition architecture of an embodiment of the present disclosure is shown. Figure 1, the identification architecture 100 of the document reference relationship may include a server 110, a terminal 120, and a network 130 that provides a communication link. The server 110 and the terminal 120 may be connected via a wired or wireless network 130. The server 110 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, security services, and CDN.

[0031] The terminal 120 may be implemented in hardware or software. For example, when the terminal 120 is implemented in hardware, it may be various electronic devices having a display screen and supporting page display, including but not limited to smart phones, tablet computers, e-book readers, laptop portable computers, and desktop computers, etc. When the terminal 120 device is implemented in software, it may be installed in the electronic devices listed above; it may be implemented as multiple software or software modules (such as software or software modules used to provide distributed services), or it may be implemented as a single software or software module, which is not specifically limited here.

[0032] It should be noted that the document reference relationship identification method provided in the embodiment of the present application can be executed by the terminal 120 or by the server 110. It should be understood that Figure 1 The number of terminals, networks and servers in the embodiment is only for illustration and is not intended to limit the number of terminals, networks and servers. Any number of terminals, networks and servers may be provided as required.

[0033] Figure 2 FIG. 2 shows a schematic diagram of the hardware structure of an exemplary electronic device 200 provided in an embodiment of the present disclosure. Figure 2 As shown, the electronic device 200 may include: a processor 202, a memory 204, a network module 206, a peripheral interface 208 and a bus 210. The processor 202, the memory 204, the network module 206 and the peripheral interface 208 are connected to each other in communication within the electronic device 200 through the bus 210.

[0034] Processor 202 may be a central processing unit (CPU), a neural network processor (NPU), a microcontroller (MCU), a programmable logic device, a digital signal processor (DSP), an application specific integrated circuit (ASIC), or one or more integrated circuits. Processor 202 may be used to perform functions related to the technology described in this disclosure. In some embodiments, processor 202 may also include multiple processors integrated into a single logical component. For example, Figure 2 As shown, processor 202 may include a plurality of processors 202a, 202b, and 202c.

[0035] The memory 204 may be configured to store data (eg, instructions, computer code, etc.). Figure 2 As shown, the data stored in the memory 204 may include program instructions (for example, program instructions for implementing the method for identifying the document reference relationship of the embodiment of the present disclosure) and data to be processed (for example, the memory may store configuration files of other modules, etc.). The processor 202 may also access the program instructions and data stored in the memory 204, and execute the program instructions to operate on the data to be processed. The memory 204 may include a volatile storage device or a non-volatile storage device. In some embodiments, the memory 204 may include a random access memory (RAM), a read-only memory (ROM), an optical disk, a magnetic disk, a hard disk, a solid-state drive (SSD), a flash memory, a memory stick, etc.

[0036] The network module 206 can be configured to provide communication with other external devices to the electronic device 200 via a network. The network can be any wired or wireless network capable of transmitting and receiving data. For example, the network can be a wired network, a local wireless network (e.g., Bluetooth, WiFi, near field communication (NFC) etc.), a cellular network, the Internet or a combination thereof. It is understood that the type of network is not limited to the above specific examples. In some embodiments, the network module 206 can include any number of network interface controllers (NICs), radio frequency modules, transceivers, modems, routers, gateways, adapters, cellular network chips, etc., in any combination.

[0037] The peripheral interface 208 can be configured to connect the electronic device 200 to one or more peripheral devices to achieve information input and output. For example, the peripheral devices can include input devices such as a keyboard, a mouse, a touch pad, a touch screen, a microphone, and various sensors, and output devices such as a display, a speaker, a vibrator, and an indicator light.

[0038] The bus 210 can be configured to transmit information between various components of the electronic device 200 (e.g., the processor 202, the memory 204, the network module 206, and the peripheral interface 208), such as an internal bus (e.g., a processor-memory bus), an external bus (USB port, PCI-E bus), etc.

[0039] It should be noted that, although the architecture of the electronic device 200 only shows the processor 202, the memory 204, the network module 206, the peripheral interface 208 and the bus 210, in the specific implementation process, the architecture of the electronic device 200 may also include other components necessary for normal execution. In addition, it can be understood by those skilled in the art that the architecture of the electronic device 200 may also only include the components necessary for implementing the embodiments of the present disclosure, and does not necessarily include all the components shown in the figure.

[0040] At present, the common methods for modeling, analyzing and calculating similarity of document citation relationships include: (1) Relation classification tasks. It is used to determine the relationship between two entities that appear simultaneously in a text. In knowledge graphs, this usually involves determining the standard citation relationship between two entities. Template matching methods are the most common methods in relationship classification tasks. It uses a template library to perform contextual matching on two given entities in the input text to determine whether the relationship between them conforms to a template in the template library. (2) Index-based methods. The names of the entities are segmented and keywords are retained as name indexes, and then an inverted index is established based on these indexes. By querying the inverted index, entities with similar names or attributes can be quickly screened out, and then further similarity calculations or relationship analysis can be performed. (3) Community networks (collaboration relationship graphs). Community networks (such as collaboration relationship graphs) can be constructed to represent the citation relationships between entities. It is usually used in scenarios such as paper author fusion, where similar nodes (authors or papers) are fused together to form a highly cohesive network. Traditional document citation association management is often limited to tables and text forms, which are difficult to understand and use, and rely on manual labor, which is prone to omissions and errors. Therefore, how to improve the accuracy and efficiency of identifying citation relationships between documents and visually and intuitively display citation relationships between documents have become technical problems that need to be solved urgently.

[0041] In view of this, the embodiments of the present disclosure provide a method for identifying document reference relationships and related devices. Natural language processing and graph database technology are used to realize automatic extraction and visualization of reference association relationships between standard documents. The complexity of reference relationships between standard documents and the problem of visual analysis and query are solved. By constructing a knowledge graph of standard documents, semantic modeling and automatic extraction of reference relationships between standards are realized. The knowledge graph is laid out using an improved force-directed graph algorithm to realize intuitive visualization of reference association relationships between standard documents. The system is suitable for fields such as standard formulation, standard retrieval, and standard knowledge management. Suitable for fields such as standard formulation, standard retrieval, and standard knowledge management.

[0042] See also Figure 3 , Figure 3 The schematic flow chart of the method for identifying a document reference relationship according to an embodiment of the present disclosure is shown. The method for identifying a document reference relationship according to an embodiment of the present disclosure can be deployed on a server or a terminal. Figure 3 In the method 300 for identifying document reference relationships, the method 300 may further include the following steps.

[0043] In step S310, data preprocessing and information extraction are performed on the document to be processed to obtain an intermediate document in a preset format.

[0044] Among them, the document to be processed may refer to the original document that needs to identify the reference relationship; for example, a technical standard document named "GB / T 12345-2023". Data preprocessing may refer to the cleaning and formatting of the original document to remove irrelevant formats, symbols and noise; for example, removing non-text information such as headers, footers, and pictures in the document and retaining the pure text content. Information extraction may refer to extracting the required key information from the preprocessed document, such as the drafting unit, drafter, standard number, standard name, etc. The intermediate document may be a document generated in a preset format after preprocessing and information extraction for further processing.

[0045] Specifically, in technical standards, standard documents can be identified and cleaned to remove irrelevant formats, symbols and noise, and the extracted information can be formatted into standard normative reference documents according to the selected reference format (for example, mandatory national standards GB, recommended national standards GB / T, etc.). Entities such as the drafting unit, drafter, standard chapter, standard number, and standard file name of the standard are identified, and the reference relationship between them is identified. Detailed metadata is generated for each reference file, including the above-mentioned extracted key information and the summary of the file, etc., to prepare basic data for subsequent analysis.

[0046] In step S320, entities in the intermediate document and entity reference relationships between the entities are identified.

[0047] Among them, an entity can refer to something that is distinguishable and exists independently. It can refer not only to an objective object that can be touched, but also to an abstract event. For example, a specific object mentioned in an intermediate document, such as a drafting unit, a drafter, a standard number, etc. Entity disambiguation can refer to a technology that solves the problem of polysemy for entities with the same name and maps the named entities appearing in the text to a known unambiguous structured knowledge base. An entity reference relationship can refer to a reference relationship between different entities in an intermediate document, such as a standard document citing another standard document.

[0048] Constructing the knowledge base dictionary: First, by collecting commonly used terms, definitions, standard clauses, etc. in technical standard documents, annotating the weights, parts of speech, and entity types of phrases, this part of data is stored in a structured manner in the database table; second, data captured from the Internet, such as books, academic papers, online resources, professional databases, etc., continuously improve and optimize the knowledge base dictionary.

[0049] In some embodiments, identifying an entity in the intermediate document and an entity reference relationship between the entities includes:

[0050] The intermediate document is input into the trained knowledge graph language model to obtain the entity and the entity reference relationship.

[0051] Among them, a semi-supervised learning model can be used for pre-training to form a knowledge graph language model, using a large amount of unlabeled data and labeled data to perform pattern recognition; through the semi-supervised learning model, relational facts can be extracted from the text and added to the knowledge graph to automatically build a large-scale knowledge graph. Specifically, active learning can be combined with semi-supervised learning to identify citation relationships. According to certain classification standards, the referenced files are classified and archived. Based on the conditional random field model, semi-supervised learning and active learning are combined. In the process of utilizing unlabeled samples, both machine annotation and manual annotation are utilized. Due to the participation of machine annotation in semi-supervised learning, the cost of manual annotation can be saved. At the same time, due to the participation of manual annotation in active learning, the impact of annotation errors in machine annotation on the model is reduced, such as Figure 4 As shown, Figure 4 A schematic diagram of a method for identifying document reference relationships according to an embodiment of the present disclosure is shown.

[0052] In some embodiments, method 300 further includes:

[0053] Acquire training data, wherein the training data includes labeled data;

[0054] Training the initial conditional random field model based on the training data to obtain an intermediate model;

[0055] Predicting the first unlabeled sample based on the intermediate model to obtain an intermediate prediction result;

[0056] Manually labeling unlabeled samples whose confidence level in the intermediate prediction results is lower than a threshold value to obtain a manual labeling result;

[0057] Update the intermediate model based on the manual annotation result to obtain an updated intermediate model;

[0058] Predicting the second unlabeled sample based on the updated intermediate model to obtain a target sample whose confidence is greater than or equal to a second threshold;

[0059] The updated intermediate model is trained based on the manual annotation results and the target samples to obtain a trained knowledge graph language model.

[0060] Specifically, the labeled data of some manually labeled reference files can be used as training data. The unlabeled data of a large number of reference files that have not been manually labeled can be used as supplementary data. Initialize the model and use a small amount of labeled data to train the initial conditional random field model. A batch of the most uncertain samples (i.e., samples with the lowest model prediction confidence) can be selected from the unlabeled data set. These uncertain samples are handed over to domain experts for manual labeling. The new manually labeled data is added to the training set and the model is retrained. The conditional random field model is used to predict a large amount of unlabeled data, and the samples with higher confidence in the prediction results are pseudo-labeled, that is, these samples are regarded as part of the labeled data. The pseudo-labeled data is merged with the real manually labeled data for the next round of model training.

[0061] After each round of active learning, the performance of the model on the validation set is evaluated, and the final trained model is used to classify and archive the reference documents. In this way, a large amount of unlabeled data is utilized through semi-supervised learning, reducing the reliance on expensive manual labeling. Active learning can reduce the impact of labeling errors on model training by selecting the most valuable samples for manual labeling. Through repeated iterations, a small amount of high-quality manually labeled data is added each time, gradually improving the accuracy and robustness of the model.

[0062] For example, a small amount of annotated citation relationship data can be collected to train the initial conditional random field model. The initial model is used to predict the unlabeled citation relationship data, and a batch of samples with the most uncertain prediction results are selected. These uncertain samples are handed over to domain experts for manual labeling, and the updated model is used to predict more unlabeled data, and the high-confidence prediction results are pseudo-labeled. The pseudo-labeled data is used together with the new manually labeled data for the next round of model training. After each round of iteration, the performance of the model on the validation set is evaluated, and the model parameters or features are adjusted as needed. Finally, the trained model is used to classify and archive new citation files. In this way, the accuracy and generalization ability of the model can be effectively improved, while the cost of manual labeling can be greatly reduced.

[0063] In step S330, the entity reference relationship is verified to obtain a verified target reference relationship.

[0064] Verification can refer to manual or automatic inspection of the extracted entity reference relationship to ensure its accuracy. For example, through manual or automatic means, verify that the document "GB / T 12345-2023" does reference "GB / T 56789-2022". The target reference relationship can refer to the entity reference relationship that has been confirmed to be correct after verification.

[0065] Specifically, manual error correction is a process of manually checking and correcting errors in text, data, or other information. Manual error correction can flexibly adjust the error correction strategy according to the actual situation, and can make more reasonable judgments for complex or ambiguous standard reference relationships. After correcting the errors, the text needs to be reviewed and verified to ensure that all errors have been corrected and no new errors have been introduced. The manually corrected data can be used as annotation data again to improve the accuracy of conditional random field model training.

[0066] Furthermore, using literature citation relationships to construct triples is an important part of text mining and knowledge graph construction. This method mainly analyzes the citation relationships between documents, extracts entities (such as authors, writing units, technical standards, etc.) and the relationships between them, and then forms triples. Cleaning and preprocessing the collected data, including removing duplicates, formatting data, parsing citation lists, etc., is the key to ensuring data quality.

[0067] In step S340, a reference relationship graph of the document to be processed is generated based on the target reference relationship and the corresponding target entity.

[0068] Among them, the reference relationship graph (or knowledge graph) can refer to a form of graph organization that associates various entities through semantic associations. The reference relationship graph extracts and merges structured and unstructured data, embodies the idea of ​​data governance and semantic connection, and is conducive to the use and migration of large-scale data.

[0069] After identifying the reference relationship, it can be stored in the knowledge graph. For each reference relationship, an edge can be created in the knowledge graph, and information such as the type of reference relationship, the referenced chapter or keyword can be used as the attributes of the edge. In the standard knowledge base, the reference association relationship of the knowledge graph can be used to discover new association points and intersections between standard documents. Through the encoding methods such as the size and color of the node, the user can intuitively understand the importance and number of citations of the standard document. The implementation method and system of the reference association relationship of standard documents based on the knowledge graph provide a new solution for the management and application of the standards field. Through automated reference relationship extraction and intelligent force-directed graph layout, the system can achieve efficient management and intuitive visual display of reference association relationships of standard documents. It has broad application value in standard formulation, standard retrieval and standard knowledge management, and brings more possibilities for the development and application of the standards field. For example, the reference relationship knowledge graph has applications such as standard abolition reminders, update reminders and standard Q&A, such as Figure 5 As shown, Figure 5 A schematic diagram of a method for identifying document reference relationships according to an embodiment of the present disclosure is shown.

[0070] Furthermore, the reference relationship graph can be stored in a graph database. A graph database can be a type of NoSQL database that applies graph theory to store relationship information between entities. A graph database is a non-relational database that applies graph theory to store relationship information between entities.

[0071] Knowledge representation can refer to the process of converting unstructured data into structured data and presenting and organizing it in a certain form, such as knowledge representation methods based on knowledge graphs. Through means such as entity alignment and entity disambiguation, different expressions of the same entity from different data sources are integrated to improve the comprehensiveness and accuracy of knowledge.

[0072] Therefore, according to the method of the embodiment of the present disclosure, by performing data preprocessing and information extraction on the document to be processed, an intermediate document in a preset format is generated, and the entities and their reference relationships therein are identified. By verifying these reference relationships, accurate target reference relationships are obtained, and finally a reference relationship map is generated to show the reference associations between documents. This process not only improves the accuracy and efficiency of information extraction, but also makes the relationship between documents visual, which is convenient for users to understand and analyze.

[0073] Specifically, the document D to be processed may be a standard document named "GB / T 12345-2023", which contains technical specifications of a certain product X. The document D to be processed may be cleaned and identified, for example, non-text information such as headers, footers, and images in the document may be removed, and the plain text content may be retained. The document may also be formatted in a standard format (such as GB / T format) to ensure that all information is presented in accordance with a unified standard.

[0074] Perform entity recognition on the document D to be processed, for example, drafting unit: Unit 1; drafters: Zhang San, Li Si; standard number: GB / T 12345-2023; standard name: "XX Product Technical Specification"; standard chapters: Chapter 3, Chapter 4, etc. Metadata of the document D to be processed can be further generated based on the above information, and the metadata can include information such as the drafting unit, drafter, standard number, standard name, and standard chapters. A document summary of the document D to be processed can also be generated to briefly introduce the main content of the document.

[0075] A certain number of annotated documents can be selected from existing standard documents as training data, and a preliminary model can be trained using the conditional random field (CRF) model. A batch of the most uncertain documents can be selected through active learning for manual annotation, and then these data can be added to the training set. The updated model can be used to predict the unlabeled documents, and the prediction results with high confidence can be selected for pseudo-annotation. The pseudo-annotated data can be merged with the manually annotated data, and the model can be trained again to finally obtain a trained annotation model.

[0076] The prediction results of the trained annotation model can be manually checked, especially the parts with low annotation confidence. For complex reference relationships, manual detailed proofreading can be performed to ensure correctness. All annotated documents can be fully reviewed to ensure that no new errors have been introduced, and the accuracy and completeness of the data can be ensured through cross-validation.

[0077] You can clean and format the annotated data to ensure data consistency and accuracy. Remove duplicates to ensure that each reference relationship is recorded only once. Extract entities and their relationships from the annotated data to form triples. For example, "GB / T 12345-2023 references GB / T 56789-2022".

[0078] The extracted reference relationships can be stored in the knowledge graph, creating nodes to represent standard documents and edges to represent reference relationships. Attributes can be added to each edge, such as reference type, reference chapter, etc. The knowledge graph can be displayed using visualization tools, and the importance and number of citations of the document can be displayed through the color, size and other attributes of the node. For example, important documents (such as core standards) are represented by larger nodes, and frequently cited documents are represented by dark nodes. In this way, users can quickly understand the associations between standard documents through the graph. Users can also use query tools to find all reference relationships of a specific document. In this way, not only can the reference relationships in standard documents be efficiently processed and annotated, but a visual knowledge graph can also be constructed to provide users with more intuitive information display and query functions. It combines the advantages of active learning and semi-supervised learning, which not only improves the efficiency of data processing, but also ensures the quality of data.

[0079] It can be seen that according to the method of the embodiment of the present disclosure, based on knowledge graph technology, combined with conditional random field model and natural language processing technology, through automatic reference relationship extraction and intelligent force-directed graph layout, it is possible to achieve efficient management and intuitive visual display of reference association relationships of standard documents. By analyzing the reference relationships in the knowledge graph, the relevance and importance between documents can be understood. When formulating new standards, references to existing documents can be referred to to avoid duplication and conflict. At the same time, for revising existing documents, the reference association information of the knowledge graph can also be used to better understand the differences and mutual influences between documents and improve the quality and applicability of documents. Application services based on natural language processing, Chinese word segmentation tools, conditional random field models, and graph databases have broad application value in areas such as document abolition reminders, document update reminders, and document Q&A, bringing more possibilities for the development and application of the document management field.

[0080] It should be noted that the method of the embodiment of the present disclosure can be performed by a single device, such as a computer or a server. The method of the present embodiment can also be applied in a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present disclosure, and the multiple devices will interact with each other to complete the described method.

[0081] It should be noted that the above describes some embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0082] Based on the same technical concept, corresponding to any of the above-mentioned embodiments and methods, the present disclosure also provides a document reference relationship identification device, see Figure 6 , the document reference relationship identification device, the device comprising:

[0083] The extraction module is used to perform data preprocessing and information extraction on the documents to be processed to obtain intermediate documents in a preset format;

[0084] An identification module, used for identifying entities in the intermediate document and entity reference relationships between the entities;

[0085] A verification module, used to verify the entity reference relationship and obtain a verified target reference relationship;

[0086] A graph module is used to generate a reference relationship graph of the document to be processed based on the target reference relationship and the corresponding target entity.

[0087] For the convenience of description, the above device is described by dividing it into various modules according to its functions. Of course, when implementing the present disclosure, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0088] The device of the above embodiment is used to implement the corresponding document reference relationship identification method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0089] Based on the same technical concept, corresponding to any of the above-mentioned embodiment methods, the present disclosure also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the document reference relationship identification method described in any of the above embodiments.

[0090] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0091] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the document reference relationship identification method described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0092] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present disclosure (including the claims) is limited to these examples. Based on the concept of the present disclosure, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present disclosure as described above, which are not provided in detail for the sake of simplicity.

[0093] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present disclosure difficult to understand, the known power / ground connections to the integrated circuit (IC) chips and other components may or may not be shown in the provided figures. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present disclosure difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which the embodiments of the present disclosure will be implemented (that is, these details should be fully within the scope of understanding of those skilled in the art). Where specific details (e.g., circuits) are set forth to describe exemplary embodiments of the present disclosure, it is apparent to those skilled in the art that the embodiments of the present disclosure can be implemented without these specific details or with changes in these specific details. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0094] Although the present disclosure has been described in conjunction with specific embodiments of the present disclosure, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0095] The embodiments of the present disclosure are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. A method for identifying document reference relations, comprising: Perform data preprocessing and information extraction on the documents to be processed to obtain intermediate documents in a preset format; identifying entities in the intermediate document and entity reference relationships between the entities; Verifying the entity reference relationship to obtain a verified target reference relationship; Based on the target reference relationship and the corresponding target entity, a reference relationship graph of the document to be processed is generated.

2. The method according to claim 1, wherein: Identifying an entity in the intermediate document and an entity reference relationship between the entities, including: The intermediate document is input into the trained knowledge graph language model to obtain the entity and the entity reference relationship.

3. The method according to claim 2, further comprising: Acquire training data, wherein the training data includes labeled data; Training the initial conditional random field model based on the training data to obtain an intermediate model; Predicting the first unlabeled sample based on the intermediate model to obtain an intermediate prediction result; Manually labeling unlabeled samples whose confidence level in the intermediate prediction results is lower than a threshold value to obtain a manual labeling result; The intermediate model is trained based on the manual annotation results to obtain a trained knowledge graph language model.

4. The method according to claim 3, wherein: The intermediate model is trained based on the manual annotation results to obtain a trained knowledge graph language model, including: Update the intermediate model based on the manual annotation result to obtain an updated intermediate model; Predicting the second unlabeled sample based on the updated intermediate model to obtain a target sample whose confidence is greater than or equal to a second threshold; The updated intermediate model is trained based on the manual annotation results and the target samples to obtain a trained knowledge graph language model.

5. The method according to claim 1, wherein: Verifying the entity reference relationship to obtain a verified target reference relationship includes: Verifying the entity reference relationship based on preset verification rules; In response to the entity reference relationship being verified, determining the entity reference relationship to be the target reference relationship; In response to the entity reference relationship verification failure, the entity reference relationship is manually corrected to obtain an updated reference relationship, and the updated reference relationship is verified until the verification passes and the updated reference relationship is determined to be the target reference relationship.

6. The method according to claim 1, wherein: Based on the target reference relationship and the corresponding target entity, a reference relationship graph of the document to be processed is generated, including: The document to be processed is determined as the first target node, the target entity is determined as the second target node, the target reference relationship is determined as the edge between the first target node and the second target node, and the attribute information of the target reference relationship is used as the attribute of the edge.

7. The method according to claim 1, wherein: The preprocessing includes: removing at least one of irrelevant formats, symbols and noises in the document to be processed; And / or, the information extraction includes extracting at least one of the following: a drafting unit, a drafting person, a document chapter, a document number or a document file name.

8. A device for identifying document reference relations, comprising: The extraction module is used to perform data preprocessing and information extraction on the documents to be processed to obtain intermediate documents in a preset format; An identification module, used for identifying entities in the intermediate document and entity reference relationships between the entities; A verification module, used to verify the entity reference relationship and obtain a verified target reference relationship; A graph module is used to generate a reference relationship graph of the document to be processed based on the target reference relationship and the corresponding target entity.

9. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 7 when executing the program. 10 . A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the method according to claim 1 .

Images