A knowledge graph construction method and device, electronic equipment and storage medium

Abstract

The application provides a knowledge graph construction method and device, electronic equipment and a computer readable storage medium. The knowledge graph construction method provided by the application comprises: acquiring a to-be-processed image material, identifying the to-be-processed image material, obtaining a text slice of the to-be-processed image material, inputting the text of the text slice, the coordinates corresponding to the text and the to-be-processed image material into a first multi-modal model which is trained and completed, and obtaining a sequence labeling result corresponding to the text; obtaining a first text relationship classification according to the sequence labeling result corresponding to the text and a second multi-modal model which is trained and completed; obtaining a text main entity, a text object and a second text relationship classification according to the text containing the first text relationship classification and a relationship extraction model which is trained and completed; and constructing a knowledge graph according to the text main entity, the text object and the second text relationship classification. The knowledge graph construction method can realize knowledge graph construction of insurance image materials.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a knowledge graph construction method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] In the era of artificial intelligence, traditional data processing methods can no longer meet human demands for information integration and knowledge acquisition. Knowledge graph technology, as one of the important foundations of artificial intelligence, has gained widespread attention in recent years due to its powerful semantic information processing capabilities and its ability to support knowledge reasoning and analysis.

[0003] Many application scenarios contain a wealth of business knowledge. For example, in the insurance sector, this rich business knowledge is primarily contained in the image materials uploaded by users for claims and underwriting. This valuable business knowledge only applies to the corresponding underwriting and claims cases and is not stored in a structured format. Knowledge graph technology can structure this business knowledge, making it an important reference for enterprise product design, risk control, and other applications. Currently, there is a lack of solutions for constructing knowledge graphs for insurance image materials. Summary of the Invention

[0004] The purpose of this invention is to provide a knowledge graph construction method, apparatus, electronic device, and computer-readable storage medium to solve the technical problem of the lack of knowledge graph construction schemes for insurance image materials in the prior art.

[0005] The technical solution of the present invention is as follows: a knowledge graph construction method is provided, including:

[0006] The image material to be processed is acquired, the image material to be processed is identified, and a text slice of the image material to be processed is obtained. The text of the text slice, the coordinates corresponding to the text, and the image material to be processed are input into a fully trained first multimodal model to obtain the sequence labeling result corresponding to the text.

[0007] Based on the sequence labeling results corresponding to the text and the fully trained second multimodal model, the first text relationship classification is obtained;

[0008] Based on the text containing the first text relation classification and the trained relation extraction model, we obtain the text main entity, text object, and second text relation classification.

[0009] A knowledge graph is constructed based on the main text entity, the text object, and the second text relationship classification.

[0010] Furthermore, the training process of the first multimodal model includes:

[0011] The text sample, the coordinates corresponding to the text sample, and the first image material sample are used as inputs to the first preset multimodal pre-trained model. The output text of the first preset multimodal pre-trained model is used as input to the linear layer. The representation output by the linear layer is used as input to the CRF layer. The CRF layer outputs the BIO sequence labeling result corresponding to the text sample.

[0012] Furthermore, the sequence labeling results corresponding to the text include text labeled with BIO sequences. Accordingly, based on the sequence labeling results corresponding to the text and the fully trained second multimodal model, a first text relation classification is obtained, including:

[0013] The text labeled with the BIO sequence, the coordinates corresponding to the text labeled with the BIO sequence, and the image material where the text labeled with the BIO sequence are located are input into the trained second multimodal model, and the relationship classification result is output. The first text relationship classification is obtained based on the relationship classification result.

[0014] Furthermore, the training process of the second multimodal model includes:

[0015] The text samples annotated with BIO sequences, the coordinates corresponding to the text samples annotated with BIO sequences, and the second image material samples are used as inputs to the second preset multimodal pre-trained model. The second preset multimodal pre-trained model outputs the representation vectors of the text characters. The representation vectors of the first characters in the two entities to be classified are transformed by linear layers to obtain the transformed representations. The transformed representations are used as inputs to the Biaffine layer, and the Biaffine layer outputs the relationship classification results.

[0016] Furthermore, based on the text containing the first text relation classification and the trained relation extraction model, the text subject entity, text object, and second text relation classification are obtained, including:

[0017] Fuzzy matching is performed on the key entities in the text containing the first text relation classification to obtain the fuzzy matched text. The fuzzy matched text is then input into the fully trained relation extraction model to obtain the text main entity, text object, and second text relation classification.

[0018] Furthermore, before obtaining the text subject entity, text object, and second text relation classification based on the text containing the first text relation classification and the trained relation extraction model, the method further includes constructing a relation extraction model based on the BERT text representation layer, subject entity extraction layer, object extraction and relation classification layer, using the ConditionalLayerNormalization network structure as the residual connection method of the relation extraction model.

[0019] The training process of the relation extraction model includes:

[0020] The text sample containing the first text relation classification is used as the input of the relation extraction model, and the corresponding text subject entity, text object, and second text relation classification are used as the output of the relation extraction model. The model is trained using the directional propagation algorithm, with the goal of reducing the value of the joint loss function of the text subject entity and text object. When the value of the joint loss function does not decrease after more than or equal to a preset number of training iterations, training is stopped, and a fully trained relation extraction model is obtained.

[0021] Furthermore, a knowledge graph is constructed based on the text subject entity, the text object, and the second text relation classification, including:

[0022] The main text entity, the object text, and the second text relationship are classified as texts to be standardized. The texts to be standardized are input into a well-trained similar text retrieval model, and the texts with standardized entity names are output. A knowledge graph is constructed based on the texts with standardized entity names.

[0023] The training process of the similar text retrieval model includes forming a training dataset using samples of the text to be standardized and the corresponding code library standard names of the samples of the text to be standardized; training the BERT model using the training dataset; using the trained BERT model to represent the corresponding code library standard names to generate a representation vector; and generating a Faiss index file based on the representation vector to obtain the similar text retrieval model.

[0024] Another technical solution of the present invention is as follows: a knowledge graph construction device is also provided, including a data preprocessing module, a first relation classification module, a second relation classification module, and a graph construction module;

[0025] The data preprocessing module is used to acquire the image material to be processed, identify the image material to be processed, obtain text slices of the image material to be processed, and input the text of the text slices, the coordinates corresponding to the text, and the image material to be processed into a fully trained first multimodal model to obtain the sequence labeling results corresponding to the text.

[0026] The first relationship classification module is used to obtain a first text relationship classification based on the sequence labeling results corresponding to the text and the trained second multimodal model;

[0027] The second relation classification module is used to obtain the main entity of the text, the object of the text, and the second text relation classification based on the text containing the first text relation classification and the trained relation extraction model.

[0028] The graph construction module is used to construct a knowledge graph based on the text main entity, the text object, and the second text relationship classification.

[0029] Another technical solution of the present invention is as follows: an electronic device is also provided, including a memory and a processor. The memory stores a computer program that can be executed by the processor. When the processor executes the computer program, it implements the knowledge graph construction method as described in any of the above technical solutions.

[0030] Another technical solution of the present invention is as follows: a computer-readable storage medium is also provided, wherein the computer-readable storage medium stores a computer program, and the computer program, when executed by a processor, implements the knowledge graph construction method as described in any of the above technical solutions.

[0031] The beneficial effects of this invention are as follows: It acquires image material to be processed, identifies the image material to be processed to obtain text slices of the image material to be processed, inputs the text of the text slices, the coordinates corresponding to the text, and the image material to be processed into a fully trained first multimodal model to obtain sequence labeling results corresponding to the text; based on the sequence labeling results corresponding to the text and a fully trained second multimodal model, it obtains a first text relation classification; based on the text containing the first text relation classification and a fully trained relation extraction model, it obtains text main entities, text objects, and a second text relation classification; it constructs a knowledge graph based on the text main entities, the text objects, and the second text relation classification; through the above method, a knowledge graph for insurance image material can be constructed, and a high-quality knowledge graph can be constructed comprehensively and quickly. Attached Figure Description

[0032] Figure 1 A flowchart illustrating the knowledge graph construction method provided in an embodiment of the present invention;

[0033] Figure 2 A schematic diagram of the original image material provided in an embodiment of the present invention;

[0034] Figure 3 This is a schematic diagram of an image material containing multiple text slices provided in an embodiment of the present invention;

[0035] Figure 4 A schematic diagram of text slice content provided in an embodiment of the present invention;

[0036] Figure 5 This is a schematic diagram of the network structure of the first multimodal model provided in an embodiment of the present invention;

[0037] Figure 6 A schematic diagram of the annotation results provided in an embodiment of the present invention;

[0038] Figure 7 This is a schematic diagram of image material containing sequence annotations provided in an embodiment of the present invention;

[0039] Figure 8 This is a schematic diagram of the network structure of the second multimodal model provided in an embodiment of the present invention;

[0040] Figure 9 A schematic diagram of image material including a first text relationship classification provided in an embodiment of the present invention;

[0041] Figure 10 This is a schematic diagram of the network structure of the CASREL model provided in an embodiment of the present invention;

[0042] Figure 11 This is a schematic diagram of the knowledge graph construction device provided in an embodiment of the present invention;

[0043] Figure 12 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0044] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0045] In the description of this application, the terms "first," "second," etc., are used only for distinguishing purposes and should not be construed as indicating or implying relative importance or order. In this specification, the terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0046] It should be noted that in the embodiments of this application, "and / or" describes the relationship between associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. In addition, the character " / ", unless otherwise specified, generally indicates that the associated objects before and after it are in an "or" relationship.

[0047] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0048] Figure 1 This is a flowchart illustrating the knowledge graph construction method according to an embodiment of the present invention. It should be noted that if substantially the same result is achieved, the knowledge graph construction method of the present invention does not necessarily follow the same approach. Figure 1 The illustrated process sequence is limited. For example... Figure 1 As shown, this knowledge graph construction method mainly includes the following steps:

[0049] S1, acquire the image material to be processed, identify the image material to be processed to obtain a text slice of the image material to be processed, input the text of the text slice, the coordinates corresponding to the text, and the image material to be processed into the first multimodal model that has been trained, and obtain the sequence labeling result corresponding to the text;

[0050] The image materials to be processed can be medical history materials uploaded by users for underwriting and / or claims, mainly including physical examination reports, outpatient medical records, admission and discharge summaries, examination reports, laboratory reports, etc. These medical history materials contain rich medical knowledge and treatment processes, such as which examinations or tests a user underwent at the hospital due to discomfort in a specific area, or the need for surgery or chemotherapy for a certain disease. Using OCR (optical character recognition) technology, text slices in the image materials are located, and the text within these slices is recognized, obtaining the coordinates and text of all text lines in the image materials.

[0051] S2, based on the sequence labeling results corresponding to the text and the fully trained second multimodal model, a first text relationship classification is obtained; wherein, the first multimodal model and the second multimodal model can be different multimodal models;

[0052] S3, based on the text containing the first text relation classification and the trained relation extraction model, obtain the text main entity, text object and second text relation classification;

[0053] S4. Construct a knowledge graph based on the text main entity, the text object, and the second text relationship classification.

[0054] In this embodiment of the invention, by acquiring image material to be processed, identifying the image material to be processed to obtain text slices of the image material to be processed, inputting the text of the text slices, the coordinates corresponding to the text, and the image material to be processed into a fully trained first multimodal model to obtain the sequence labeling results corresponding to the text; based on the sequence labeling results corresponding to the text and a fully trained second multimodal model, a first text relation classification is obtained; based on the text containing the first text relation classification and a fully trained relation extraction model, text main entities, text objects, and a second text relation classification are obtained; a knowledge graph is constructed based on the text main entities, the text objects, and the second text relation classification; this can realize the construction of a knowledge graph for insurance image material, and can comprehensively and quickly construct a high-quality knowledge graph.

[0055] In an optional implementation, the training process of the first multimodal model includes:

[0056] The text sample, the coordinates corresponding to the text sample, and the first image material sample are used as inputs to the first preset multimodal pre-trained model. The output text of the first preset multimodal pre-trained model is used as input to the linear layer. The representation output by the linear layer is used as input to the CRF layer. The CRF layer outputs the BIO sequence labeling result corresponding to the text sample.

[0057] It should be noted that plain text-based sequence annotation performs poorly in image material sequence annotation tasks. This is mainly because it ignores the layout information of the image material; for example, keyword entities such as "name" and "discharge diagnosis" often have different fonts and positions than the main text. Reconstructing image material paragraphs based on text and coordinate information rules has low accuracy, especially for cases outside the rule coverage. While OCR parsing can effectively obtain the coordinates of text slices and their corresponding text strings, it may lose paragraph information. Using only text information to reconstruct image report paragraphs has limited effectiveness. However, jointly modeling the layout information and text information of the image material can more accurately and effectively reconstruct paragraphs.

[0058] In one specific embodiment, paragraph merging includes two parts: sequence labeling and relationship classification. The original image materials from the actual user's claim and / or underwriting process, and a schematic diagram of the original image materials, such as... Figure 2 As shown; after the original image material is recognized by OCR, the text in the original image material is divided into multiple text slices. A schematic diagram of an image material containing multiple text slices is shown below. Figure 3 As shown; a schematic diagram of the text slice content, as follows. Figure 4 As shown, the text slice includes the text content "content", the text coordinates in the image "position", and the slice number "id". Using a first multimodal network structure, key, value, hospital, and other entities in the image material are identified based on BIO annotation patterns. Slices are then segmented based on the entity recognition results. A schematic diagram of the network structure of the first multimodal model is shown below. Figure 5 As shown, the network structure of the first multimodal model includes a LayoutLMv3 network structure (the first preset multimodal pre-trained model network structure) and a CRF layer. During model training, the directional propagation algorithm is used to reduce the value of the loss function (crf-loss function) as the training objective. An early-stop mechanism is added: when the value of the crf-loss function does not decrease after a preset number of training iterations (e.g., 3 rounds), training stops, and the model with the smallest crf-loss function value during training is output as the optimal model.

[0059] like Figure 5 As shown, in actual annotation, the text to be extracted, the coordinates of the text, and the image (picture, original image material) are used as inputs to layoutLMv3. layoutLMv3 outputs the text representation. The text representation is transformed by the Linear layer and used as the emission probability of the CRF layer. The CRF layer decodes the text to obtain the BIO sequence annotation result corresponding to the input text.

[0060] In another specific embodiment, the entities in the sequence-annotated image material are shown in Table 1.

[0061] Table 1 shows the entities marked in the image materials.

[0062] Entity type Entity description key Keywords value The value corresponding to the keyword hospital Hospital other other

[0063] Slicing and labeling the OCR recognition results yields the labeled results, as illustrated in the diagram below. Figure 6 As shown, Figure 6 In the text, the "label" tag marks the sliced ​​text as the "key" entity. One type of BIO annotation result is shown in Table 2.

[0064] Table 2 BIO annotation results

[0065] surname name certain certain certain B-key I-key B-value I-value I-value

[0066] When performing BIO annotation, the input text is "Name [Name]", and the output sequence labeling result is "B-key I-key otherB-value I-value I-value".

[0067] In specific implementation, such as Figure 3 As shown, the text in the original image material is divided into multiple text slices. Only the slice sequence can be reconstructed based on the slice coordinate order information, but paragraph and structural information cannot be reconstructed. Sequence annotation based on the first multimodal model yields image material containing sequence annotations. A schematic diagram of image material containing sequence annotations is shown below. Figure 7 As shown, the first multimodal model annotation can be used to annotate key type entities such as "name", "admission date" and "admission diagnosis", as well as value type entities such as "male" and "37 years old".

[0068] In an optional implementation, the sequence labeling result corresponding to the text includes the text after BIO sequence labeling. Accordingly, based on the sequence labeling result corresponding to the text and the fully trained second multimodal model, a first text relationship classification is obtained, including:

[0069] The text labeled with the BIO sequence, the coordinates corresponding to the text labeled with the BIO sequence, and the image material where the text labeled with the BIO sequence are located are input into the trained second multimodal model, and the relationship classification result is output. The first text relationship classification is obtained based on the relationship classification result.

[0070] In one specific embodiment, a schematic diagram of the network structure of the second multimodal model is shown below. Figure 8 As shown, the network structure of the second multimodal model includes a LayoutLMv3 network (the second pre-trained multimodal model) and a Biaffine layer. The second multimodal model is used for a relation classification task.

[0071] In an optional implementation, the training process of the second multimodal model includes:

[0072] The text samples annotated with BIO sequences, the coordinates corresponding to the text samples annotated with BIO sequences, and the second image material samples are used as inputs to the second preset multimodal pre-trained model. The second preset multimodal pre-trained model outputs the representation vectors of the text characters. The representation vectors of the first characters in the two entities to be classified are transformed by linear layers to obtain the transformed representations. The transformed representations are used as inputs to the Biaffine layer, and the Biaffine layer outputs the relationship classification results.

[0073] In one specific embodiment, during the training and prediction process of the second multimodal model, after the entity type corresponding to each text slice is known, several (three) nearest key entities are sampled for each value entity to form a pair of entities to be classified. The Euclidean distance between the coordinates of the key entity (bottom right corner point) and the key entity (top left corner point) is calculated, and the Euclidean distance is used as an evaluation index for the proximity of the value entity and the key entity.

[0074] The second multimodal model is trained using the directional propagation algorithm, with the goal of reducing the value of the cross-entropy loss (CE-loss) function. An early-stop mechanism is added: when the CE-loss value does not decrease after a preset number of training rounds (3 rounds), training is stopped, and the model with the smallest value of the cross-entropy loss function during training is output as the optimal model.

[0075] like Figure 8 As shown, in practical applications, the text of the text slice in the image, the corresponding coordinates of the text, and the image itself are used as inputs to layoutLMv3 (the second preset multimodal pre-trained model). The output is the representation vector of the text characters. The representation vectors of the first character of the two entities to be classified (entity 1 and entity 2) are extracted and transformed through a Linear layer. The representations after the Linear layer transformation are used as inputs to the Biaffine layer, which outputs a classification result of 1 or 0. A classification result of 1 or 0 (relationship classification result) indicates whether the two entities constitute a key-value relationship. Based on the relationship classification result, the first text relationship classification can be obtained, that is, the classification of whether the entities in the text constitute a key-value relationship.

[0076] In specific implementation, based on the sequence labeling results corresponding to the text and the fully trained second multimodal model, a first text relationship classification can be obtained, along with a schematic diagram of the image material containing the first text relationship classification, as shown below. Figure 9 As shown, Figure 9 The lines connecting the boxes in the image represent the first text relationship classification. For example, the key "gender" corresponds to the value "male". The first text relationship classification realizes the structuring of the text sequence, thereby enabling paragraph understanding of the video material.

[0077] In an optional implementation, based on the text containing the first text relation classification and the trained relation extraction model, the text subject entity, text object, and second text relation classification are obtained, including:

[0078] Fuzzy matching is performed on the key entities in the text containing the first text relation classification to obtain the fuzzy matched text. The fuzzy matched text is then input into the fully trained relation extraction model to obtain the text main entity, text object, and second text relation classification.

[0079] In one specific embodiment, the text containing the first text relationship classification is the text in which each entity constitutes a key-value relationship classification, i.e., a structured text paragraph. The paragraph topic classification can be achieved based on the text content of the key entity using a fuzzy matching algorithm. The fuzzywuzzy toolkit is used to perform fuzzy matching on the key entity text. For example, the description of "department" includes departments, outpatient departments, execution departments, patient departments, admission departments, etc. When OCR recognition contains typos, resulting in the key entity being "admission department," or when the accumulated corpus does not have a classification for "admission department," the similarity between the department and related descriptions can be obtained based on fuzzywuzzy. For example, if "admission diagnosis" and "past medical history" have a high similarity, then the key entity "department" can be classified as a department. Key-value pairs can directly return the paragraph topic classification extraction results, such as hospital stay days: 7 days, gender: male, etc. Paragraph types, such as admission status and hospitalization process, require further refinement based on text relationship extraction tasks.

[0080] In an optional implementation, before obtaining the text main entity, text object and second text relation classification based on the text containing the first text relation classification and the trained relation extraction model, the method further includes constructing a relation extraction model based on the BERT text representation layer, main entity extraction layer, object extraction and relation classification layer, and using the Conditional Layer Normalization network structure as the residual connection method of the relation extraction model.

[0081] The training process of the relation extraction model includes:

[0082] The text sample containing the first text relation classification is used as the input of the relation extraction model, and the corresponding text subject entity, text object, and second text relation classification are used as the output of the relation extraction model. The model is trained using the directional propagation algorithm, with the goal of reducing the value of the joint loss function of the text subject entity and text object. When the value of the joint loss function does not decrease after more than or equal to a preset number of training iterations, training is stopped, and a fully trained relation extraction model is obtained.

[0083] In one specific embodiment, the paragraph text is extracted to obtain a relation table, as shown in Table 3.

[0084] Table 3 Relationship Table

[0085]

[0086]

[0087]

[0088] User case information with rich node relationships, as shown in Table 3, can be structured from paragraph text (including text classified under the first text relationship category). For example, "Laboratory tests and auxiliary examinations: Ultrasound examination at Jizhou District Central Hospital: Fatty liver (moderate) and left kidney stone" can be extracted to obtain "the results corresponding to the examination". That is, the examination is "ultrasound", and the corresponding examination results are "fatty liver (moderate)" and "left kidney stone".

[0089] In another specific embodiment, the CASREL model is used as the base model for the relation extraction model. A schematic diagram of the CASREL model's network structure is shown below. Figure 10 As shown, Figure 10 It includes the BERT text representation layer (BERT Encoder), the subject entity extraction layer (Subject), and the object extraction and relation classification layer (Relations). N For the output of the BERT text representation layer, v sub The main entity is output by the main entity extraction layer, where k represents the k-th main entity. The relation extraction model includes a BERT text representation layer, which outputs the text representation vector; a main entity extraction layer, which identifies the main entities in the text; and an object extraction and relation classification layer, which extracts the objects corresponding to the main entities and performs relation classification. The residual connection method in the basic model is changed to Conditional Layer Normalization to optimize the network structure. During model training and prediction, each text in the paragraph (containing the text of the first text relation classification) is used as input to identify the main entities (text main entities) in the text. For each main entity, the corresponding object (text object) and relation classification (second text relation classification) are extracted.

[0090] The relation extraction model is trained using the directional propagation algorithm, with the goal of reducing the value of the joint loss function of the subject and object. The loss functions of the subject and object are both poor entropy loss functions, and the joint loss function is loss = w1 * subject_loss + w2 * object_loss. An early-stop mechanism is added: if the value of the joint loss function does not decrease after more than or equal to a preset number of training rounds (e.g., 3 rounds), training is stopped, and the model with the smallest value of the joint loss function during training is output as the optimal model. Here, subject_loss is the loss function of the subject, object_loss is the loss function of the object, w1 and w2 are the weights of the loss function, and w1 + w2 = 1.

[0091] In practice, the text input to the relation extraction model is "Past medical history: underwent 'gastric polyp removal' several months ago, followed by a gastroscopy showing erosive gastritis, with occasional discomfort below the xiphoid process." After identifying the principal entities, the relation extraction model obtains two principal entities: the surgery name "gastric polyp removal" and the examination name "gastroscopy." The principal entity identification results are shown in Table 4. After extracting objects and classifying relations for the two principal entities, the objects and relation classifications corresponding to principal entity 1 are shown in Table 5, and the objects and relation classifications corresponding to principal entity 2 are shown in Table 6.

[0092] Table 4. Main Entity Recognition Results

[0093]

[0094] Table 5 Classification of Subject Entity 1, Corresponding Objects, and Relationships

[0095]

[0096] Table 6 Classification of Subject Entity 2, Corresponding Objects, and Relationships

[0097]

[0098] In an optional implementation, a knowledge graph is constructed based on the text subject entity, the text object, and the second text relation classification, including:

[0099] The main text entity, the object text, and the second text relationship are classified as texts to be standardized. The texts to be standardized are input into a well-trained similar text retrieval model, and the texts with standardized entity names are output. A knowledge graph is constructed based on the texts with standardized entity names.

[0100] The training process of the similar text retrieval model includes forming a training dataset using samples of the text to be standardized and the corresponding code library standard names of the samples of the text to be standardized; training the BERT model using the training dataset; using the trained BERT model to represent the corresponding code library standard names to generate a representation vector; and generating a Faiss index file based on the representation vector to obtain the similar text retrieval model.

[0101] It should be noted that the claims and / or underwriting images may come from different hospitals, and different hospitals may describe the same entity in different ways. For example, "left kidney stone" and "right kidney stone" are both coded to the ICD code "N20.000", which corresponds to the disease "kidney stone". This requires the name standardization of entity names such as disease, surgery, examination / test name, and hospital name. That is, the extracted entity names need to be coded to the corresponding code library standard names.

[0102] In one specific embodiment, a similar text retrieval model is constructed using the SimBERT model and the Faiss toolkit. Taking diseases as an example, a Faiss index is built from all disease texts in ICD-10. By inputting the extracted disease name, the corresponding disease in ICD-10 and its corresponding ICD code are output, thus achieving disease name standardization. The input to the SimBERT model is text pairs, as shown in Table 7.

[0103] Table 7 Text Pairs

[0104] text Squamous cell tumors of the esophagus and adenocarcinomas of the gastroesophageal junction synonyms Malignant tumor at the esophagogastric junction

[0105] In Table 7, `text` represents the disease description extracted from the image material (text main entity, text object, and the second text relation classification), and `synonyms` represents the expected disease name after coding (code library standard name). Based on the SimBERT model, SimBERT is fine-tuned based on business scenario data. That is, a training dataset is formed using samples of the text to be standardized and the corresponding code library standard names of the samples of the text to be standardized. The BERT model (SimBERT model) is trained using the training dataset. All disease standard names, after fine-tuning, are represented by the SimBERT representation (trained BERT model) as sentence vectors, and a Faiss index file is generated based on the Faiss toolkit. The text to be standardized is input into the Faiss index file, and the text with standardized entity names is output.

[0106] In another specific embodiment, the structured results of the case data (text with standardized entity names) can be written into a graph database. This allows the graph database to contain graph data, including basic user information such as name, gender, and age; disease information such as the year a certain disease was diagnosed; examination / test information; surgical information; medication information; chemotherapy information, etc., which can be used to construct a knowledge graph. It should be noted that before writing the structured results of the case data into the graph database, results with low confidence can be reviewed to improve the accuracy of the graph data in the database. The confidence level of the structured results of the case data (image materials) can be obtained based on the accuracy of various model algorithms, such as the first multimodal model, the second multimodal model, the relation extraction model, and the similar text retrieval model.

[0107] The knowledge graph construction method provided in this invention involves acquiring image material to be processed, identifying the image material to be processed to obtain text slices of the image material to be processed, inputting the text of the text slices, the coordinates corresponding to the text, and the image material to be processed into a fully trained first multimodal model to obtain sequence labeling results corresponding to the text; obtaining a first text relation classification based on the sequence labeling results corresponding to the text and a fully trained second multimodal model; obtaining text subject entities, text object entities, and a second text relation classification based on the text containing the first text relation classification and a fully trained relation extraction model; and constructing a knowledge graph based on the text subject entities, the text object entities, and the second text relation classification. This method can realize the construction of a knowledge graph for insurance image material and can comprehensively and quickly construct a high-quality knowledge graph.

[0108] It should be noted that for multi-line paragraph text or form-structured text, direct OCR recognition followed by text structuring makes it difficult to reconstruct the document structure. OCR recognition of image materials often results in typos or punctuation errors; if document structure is reconstructed solely based on text features, the reconstruction quality is significantly affected by the OCR recognition performance. The knowledge graph construction method provided in this invention constructs image materials using multimodal features, integrating image layout features, coordinate information, and text features. This effectively reconstructs the text structure of image materials, proving more efficient than relying solely on text features or coordinate information rules. This invention also utilizes a relation extraction model including the CASREL network and ConditionalLayer Normalization to transform the discrete relation classification task into a triplet-finding task, allowing for the simultaneous extraction of multiple relations from the text and effectively resolving relation overlap issues. Since the same entity can have multiple descriptions, and since OCR recognition may contain typos, this invention employs a similar text retrieval model constructed using the SimBERT model and the Faiss toolkit to standardize entity names such as hospitals, diseases, examinations, and lab tests. Nodes or relationships with low confidence in the structuring results of image materials can be reviewed and corrected; finally, the structured image materials are written into the graph database to update the spectral data.

[0109] Video materials from insurance scenarios can serve as crucial reference data for insurance companies in areas such as product design, risk control, claims automation, and underwriting automation, representing a significant intellectual asset. By constructing domain knowledge graphs from structured video materials, insurance companies can alleviate many of the pressures they face in risk management, preventing speculators from exploiting loopholes in their operations to harm their interests. Furthermore, knowledge graphs can optimize insurance industry supervision, allowing regulators to more accurately grasp the overall operational status of insurance companies and proactively address regulatory challenges. In addition, knowledge graphs can solve problems such as information asymmetry for consumers in transactions and the high cost of obtaining product information, providing new ideas for optimizing the consumer service experience.

[0110] The knowledge graph construction method provided in this invention can be built based on artificial intelligence. It acquires and processes relevant data using AI technology, enabling unattended knowledge graph construction. Artificial intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0111] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0112] Figure 11 This is a schematic diagram of the knowledge graph construction device according to an embodiment of the present invention, such as... Figure 11 As shown, the knowledge graph construction device 110 includes a data preprocessing module 111, a first relation classification module 112, a second relation classification module 113, and a graph construction module 114;

[0113] The data preprocessing module 111 is used to acquire the image material to be processed, identify the image material to be processed, obtain text slices of the image material to be processed, input the text of the text slices, the coordinates corresponding to the text, and the image material to be processed into a fully trained first multimodal model to obtain the sequence labeling result corresponding to the text.

[0114] The first relationship classification module 112 is used to obtain a first text relationship classification based on the sequence labeling results corresponding to the text and the trained second multimodal model;

[0115] The second relation classification module 113 is used to obtain the text main entity, text object and second text relation classification based on the text containing the first text relation classification and the trained relation extraction model.

[0116] The graph construction module 114 is used to construct a knowledge graph based on the text main entity, the text object, and the second text relationship classification.

[0117] In an optional implementation, the knowledge graph construction device 110 further includes a first training module, which is used to train a first multimodal model. The training process of the first multimodal model includes:

[0118] The text sample, the coordinates corresponding to the text sample, and the first image material sample are used as inputs to the first preset multimodal pre-trained model. The output text of the first preset multimodal pre-trained model is used as input to the linear layer. The representation output by the linear layer is used as input to the CRF layer. The CRF layer outputs the BIO sequence labeling result corresponding to the text sample.

[0119] In an optional implementation, the sequence labeling result corresponding to the text includes the text after BIO sequence labeling. Accordingly, the first relationship classification module 112 is further configured to input the text after BIO sequence labeling, the coordinates corresponding to the text after BIO sequence labeling, and the image material where the text after BIO sequence labeling is located into the trained second multimodal model, output the relationship classification result, and obtain the first text relationship classification based on the relationship classification result.

[0120] In an optional implementation, the knowledge graph construction device 110 further includes a second training module, which is used to train a second multimodal model. The training process of the second multimodal model includes:

[0121] The text samples annotated with BIO sequences, the coordinates corresponding to the text samples annotated with BIO sequences, and the second image material samples are used as inputs to the second preset multimodal pre-trained model. The second preset multimodal pre-trained model outputs the representation vectors of the text characters. The representation vectors of the first characters in the two entities to be classified are transformed by linear layers to obtain the transformed representations. The transformed representations are used as inputs to the Biaffine layer, and the Biaffine layer outputs the relationship classification results.

[0122] In an optional implementation, the second relation classification module 113 is further configured to perform fuzzy matching on the key entities in the text containing the first text relation classification to obtain the fuzzy-matched text, and input the fuzzy-matched text into the trained relation extraction model to obtain the text main entity, text object and second text relation classification.

[0123] In an optional implementation, the knowledge graph construction device 110 further includes a relation extraction model construction module and a third training module. The relation extraction model construction module is used to construct a relation extraction model based on the BERT text representation layer, main entity extraction layer, object extraction and relation classification layer, and uses the Conditional Layer Normalization network structure as the residual connection method of the relation extraction model.

[0124] The third training module is used to train the relation extraction model, and the training process of the relation extraction model includes:

[0125] The text sample containing the first text relation classification is used as the input of the relation extraction model, and the corresponding text subject entity, text object, and second text relation classification are used as the output of the relation extraction model. The model is trained using the directional propagation algorithm, with the goal of reducing the value of the joint loss function of the text subject entity and text object. When the value of the joint loss function does not decrease after more than or equal to a preset number of training iterations, training is stopped, and a fully trained relation extraction model is obtained.

[0126] In an optional implementation, the graph construction module 114 is further configured to classify the text main entity, the text object, and the second text relationship as text to be standardized, input the text to be standardized into a well-trained similar text retrieval model, output the text after entity name standardization, and construct a knowledge graph based on the text after entity name standardization.

[0127] The training process of the similar text retrieval model includes forming a training dataset using samples of the text to be standardized and the corresponding code library standard names of the samples of the text to be standardized; training the BERT model using the training dataset; using the trained BERT model to represent the corresponding code library standard names to generate a representation vector; and generating a Faiss index file based on the representation vector to obtain the similar text retrieval model.

[0128] The knowledge graph construction apparatus provided in this embodiment of the invention acquires image material to be processed through a data preprocessing module 111, identifies the image material to be processed to obtain text slices of the image material to be processed, and inputs the text of the text slices, the coordinates corresponding to the text, and the image material to be processed into a fully trained first multimodal model to obtain the sequence labeling result corresponding to the text; a first relation classification module 112 obtains a first text relation classification based on the sequence labeling result corresponding to the text and a fully trained second multimodal model; a second relation classification module 113 obtains text main entities, text objects, and a second text relation classification based on the text containing the first text relation classification and a fully trained relation extraction model; and a graph construction module 114 constructs a knowledge graph based on the text main entities, the text objects, and the second text relation classification; this apparatus can realize the construction of a knowledge graph for insurance image material and can comprehensively and quickly construct a high-quality knowledge graph.

[0129] Figure 12 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Figure 12 As shown, the electronic device 120 includes a processor 121 and a memory 122 communicatively connected to the processor 121.

[0130] The memory 122 stores program instructions for implementing the knowledge graph construction method of any of the above embodiments.

[0131] The processor 121 is used to execute program instructions stored in the memory 122 to construct a knowledge graph.

[0132] The processor 121 can also be referred to as a CPU (Central Processing Unit). The processor 121 may be an integrated circuit chip with signal processing capabilities. The processor 121 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor.

[0133] This invention provides a storage medium that stores program instructions capable of implementing all the methods described above. The storage medium can be non-volatile or volatile. These program instructions can be stored in the storage medium as a software product, including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks, or terminal devices such as computers, servers, mobile phones, and tablets.

[0134] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0135] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. The above are merely embodiments of the present invention and do not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

[0136] The above description is merely an embodiment of the present invention. It should be noted that those skilled in the art can make improvements without departing from the inventive concept of the present invention, but these improvements all fall within the protection scope of the present invention.

Claims

1. A method for constructing a knowledge graph, characterized in that, The method comprises the following steps: Obtaining image material to be processed, identifying the image material to be processed, obtaining a text slice of the image material to be processed, inputting text of the text slice, coordinates corresponding to the text and the image material to be processed into a trained first multi-modal model, obtaining a sequence labeling result corresponding to the text, the sequence labeling result corresponding to the text comprising a text after BIO sequence labeling, the coordinates corresponding to the text being coordinates of the text in the image material to be processed, a training process of the first multi-modal model comprising inputting a text sample, coordinates corresponding to the text sample and a first image material sample into a first preset multi-modal pre-training model, inputting output text of the first preset multi-modal pre-training model into a linear layer, inputting a representation output by the linear layer into a CRF layer, the CRF layer outputting a BIO sequence labeling result corresponding to the text sample, a network structure of the first preset multi-modal pre-training model being a LayoutLMv3 network structure; Obtaining a first text relationship classification according to the sequence labeling result corresponding to the text and a trained second multi-modal model, a training process of the second multi-modal model comprising inputting a text sample after BIO sequence labeling, coordinates corresponding to the text sample after BIO sequence labeling and a second image material sample into a second preset multi-modal pre-training model, inputting a representation vector of a text character output by the second preset multi-modal pre-training model, inputting a representation vector of a first character in two relationship entities to be classified into a linear layer respectively to obtain a transformed representation, inputting the transformed representation into a Biaffine layer, the Biaffine layer outputting a relationship classification result, a network structure of the second preset multi-modal pre-training model being a LayoutLMv3 network structure; Obtaining a text main entity, a text object and a second text relationship classification according to a text containing the first text relationship classification and a trained relationship extraction model; Constructing a knowledge graph according to the text main entity, the text object and the second text relationship classification. 2.The knowledge graph construction method of claim 1, wherein, Obtaining a first text relationship classification according to the sequence labeling result corresponding to the text and a trained second multi-modal model, comprising: Inputting the text after BIO sequence labeling, coordinates corresponding to the text after BIO sequence labeling and an image material where the text after BIO sequence labeling is located into the trained second multi-modal model to output a relationship classification result, and obtaining the first text relationship classification according to the relationship classification result. Obtaining a text main entity, a text object and a second text relationship classification according to a text containing the first text relationship classification and a trained relationship extraction model, comprising: 3.The knowledge graph construction method of claim 1, wherein, Performing fuzzy matching on key entities in the text containing the first text relationship classification to obtain a text after fuzzy matching, inputting the text after fuzzy matching into the trained relationship extraction model to obtain the text main entity, the text object and the second text relationship classification. ​ 4.The knowledge graph construction method of claim 1, wherein, Before obtaining the text main entity, the text object and the second text relation classification according to the text containing the first text relation classification and the trained relation extraction model, further comprising: constructing a relation extraction model according to a BERT text representation layer, a main entity extraction layer, an object extraction and relation classification layer, and taking a Conditional Layer Normalization network structure as a residual connection mode of the relation extraction model; The training process of the relation extraction model comprises: Taking the text sample containing the first text relation classification as the input of the relation extraction model, and taking the corresponding text main entity, text object and second text relation classification as the output of the relation extraction model; using a direction propagation algorithm to train the model, taking the value of the joint loss function of the text main entity and the text object as the target of the model training, and stopping the training when the value of the joint loss function does not decrease when greater than or equal to a preset training number, to obtain a trained relation extraction model. 5.The knowledge graph construction method of claim 1, wherein, According to the text main entity, the text object and the second text relation classification, a knowledge graph is constructed, comprising: Taking the text main entity, the text object and the second text relation classification as a text to be standardized, inputting the text to be standardized into a trained similar text retrieval model, outputting a text after standardizing the entity name, and constructing a knowledge graph according to the text after standardizing the entity name; The training process of the similar text retrieval model comprises: forming a training data set by taking the sample of the text to be standardized and the corresponding code library standard name of the sample of the text to be standardized, training a BERT model by taking the training data set, representing the corresponding code library standard name by the trained BERT model, generating a representation vector, generating a faiss index file according to the representation vector, and obtaining a similar text retrieval model.

6. A knowledge graph construction apparatus, characterized by comprising: It comprises a data preprocessing module, a first training module, a first relation classification module, a second training module, a second relation classification module and a graph construction module. The data preprocessing module is used to obtain image materials to be processed, identify the image materials to be processed, obtain text slices of the image materials to be processed, input the text of the text slices, the coordinates corresponding to the text and the image materials to be processed into a trained first multi-modal model, and obtain sequence labeling results corresponding to the text, wherein the sequence labeling results corresponding to the text comprise text after BIO sequence labeling, and the coordinates corresponding to the text are coordinates of the text in the image materials to be processed; the first training module is used to train the first multi-modal model, and the training process of the first multi-modal model comprises: The text sample, the coordinates corresponding to the text sample, and the first image material sample are input into a first preset multi-modal pre-training model, an output text of the first preset multi-modal pre-training model is input into a linear layer, and a representation output by the linear layer is input into a CRF layer, and the CRF layer outputs a BIO sequence labeling result corresponding to the text sample, and a network structure of the first preset multi-modal pre-training model is a LayoutLMv3 network structure. The first relationship classification module is configured to obtain a first text relationship classification according to the sequence labeling result corresponding to the text and a second multi-modal model that is trained completely. The second training module inputs a text sample after BIO sequence labeling, coordinates corresponding to the text sample after BIO sequence labeling, and a second image material sample into a second preset multi-modal pre-training model, and makes the second preset multi-modal pre-training model output a representation vector of a text character, and the representation vector of a first character in two relationship entity classes is respectively transformed by a linear layer to obtain a transformed representation, and the transformed representation is input into a Biaffine layer, and the Biaffine layer outputs a relationship classification result, and a network structure of the second preset multi-modal pre-training model is a LayoutLMv3 network structure. The second relationship classification module is configured to obtain a text subject, a text object, and a second text relationship classification according to a text containing the first text relationship classification and a relationship extraction model that is trained completely. The graph construction module is configured to construct a knowledge graph according to the text subject, the text object, and the second text relationship classification.

7. An electronic device comprising a memory, a processor, the memory storing a computer program, characterized in that, The processor executes the computer program to implement the knowledge graph construction method in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 7. The computer program is executed by the processor to implement the knowledge graph construction method in any one of claims 1 to 5.

Images