Knowledge graph construction method and device based on large model

Abstract

The embodiment of the invention discloses a knowledge graph construction method and device based on a large model, belongs to the technical field of artificial intelligence, and is used for solving the problems of low data utilization rate and low knowledge graph accuracy of knowledge graph construction. Comprising the steps of obtaining original data of a multi-source service, performing knowledge extraction processing on the original data to obtain a data set, mapping the data set to a unified vector space through a pre-trained large model to obtain a to-be-aligned entity, and performing entity alignment processing on the to-be-aligned entity and a target domain dictionary to obtain a target candidate entity set of the to-be-aligned entity; performing entity disambiguation processing on the target candidate entity set and the to-be-aligned entities to obtain an entity knowledge set; and according to the entity knowledge set, performing relation alignment processing on the entity relation of the entities in the entity knowledge set through the large model, outputting a knowledge triple after the relation alignment processing, and constructing a knowledge graph corresponding to the original data based on the knowledge triple.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a knowledge graph construction method and device based on a large model. BACKGROUND

[0002] Currently, the knowledge base of operator customer service has various sources and rich data types, lacks unified specifications and standards, and each province independently maintains business knowledge, which brings great difficulty to establishing a unified knowledge graph. It is difficult to ensure the accuracy and consistency of data.

[0003] In related technologies, the knowledge graph needs to be updated and maintained regularly, and operators need to perform a large amount of manual operations, data cleaning and auditing, etc., resulting in high management and maintenance costs. The developers use the offline data processing and database import method, which cannot meet the real-time and interactive requirements of the knowledge graph. Therefore, the data processing process mainly depends on manual guidance and the knowledge management mode is static, which has the problems of low data utilization rate of constructing the knowledge graph and low accuracy of the knowledge graph. SUMMARY

[0004] The purpose of the embodiments of the present application is to provide a knowledge graph construction method and device based on a large model, to solve the problems of low data utilization rate of constructing the knowledge graph and low accuracy of the knowledge graph.

[0005] To solve the above technical problems, the embodiments of the present application are implemented as follows: In a first aspect, the embodiments of the present application provide a knowledge graph construction method based on a large model, which includes: obtaining original data of multi-source business, the original data including: structured data, semi-structured data and unstructured data; performing knowledge extraction processing on the original data to obtain a data set, the data set including: a structured sub-data set, an entity text sub-data set and a visual text sub-data set; mapping the data set to a unified vector space through a pre-trained large model to obtain a to-be-aligned entity, and performing entity alignment processing on the to-be-aligned entity and a target field dictionary to obtain a target candidate entity set of the to-be-aligned entity; performing entity disambiguation processing on the target candidate entity set and the to-be-aligned entity to obtain an entity knowledge set; performing relationship alignment processing on the entity relationships of the entities in the entity knowledge set through the large model according to the entity knowledge set, outputting a knowledge triple after relationship alignment processing, and constructing a knowledge graph corresponding to the original data based on the knowledge triple.

[0006] In a second aspect, an embodiment of the present application provides a knowledge graph construction device based on a large model, the device comprising: an acquisition module configured to acquire original data of a plurality of sources of business, the original data comprising structured data, semi-structured data and unstructured data; an extraction module configured to perform knowledge extraction processing on the original data to obtain a data set, the data set comprising: a structured sub-data set, an entity text sub-data set and a visual text sub-data set; a mapping module configured to map the data set to a unified vector space by using a pre-trained large model to obtain to-be-aligned entities, and perform entity alignment processing on the to-be-aligned entities and a target field dictionary to obtain a target candidate entity set of the to-be-aligned entities; a disambiguation module configured to perform entity disambiguation processing on the target candidate entity set and the to-be-aligned entities to obtain an entity knowledge set; and an output module configured to perform relation alignment processing on entity relations of entities in the entity knowledge set by using the large model according to the entity knowledge set, output knowledge triples after the relation alignment processing, and construct a knowledge graph corresponding to the original data based on the knowledge triples.

[0007] In a third aspect, an embodiment of the present application provides an electronic device, which comprises a processor, a memory, and a program or instructions stored on the memory and executable on the processor, and the program or instructions are executed by the processor to implement the knowledge graph construction method based on a large model as described above.

[0008] In a fourth aspect, an embodiment of the present application provides a readable storage medium, and the readable storage medium stores a program or instructions, and the program or instructions are executed by a processor to implement the knowledge graph construction method based on a large model as described above.

[0009] In a fifth aspect, an embodiment of the present application provides a chip, which comprises a processor and a communication interface, the communication interface and the processor are coupled, and the processor is configured to run a program or instructions to implement the knowledge graph construction method based on a large model as described above.

[0010] In a sixth aspect, an embodiment of the present application provides a computer program product, and the computer program product comprises a computer program, and the computer program is executed by a processor to implement the knowledge graph construction method based on a large model as described above.

[0011] The technical scheme provided in the embodiment of the application comprises the following steps: obtaining original data of multi-source services, wherein the original data comprises structured data, semi-structured data and unstructured data; performing knowledge extraction processing on the original data to obtain a data set, wherein the data set comprises a structured sub-data set, an entity text sub-data set and a visual text sub-data set; mapping the data set to a unified vector space by using a pre-trained large model to obtain to-be-aligned entities, and performing entity alignment processing on the to-be-aligned entities and a target field dictionary to obtain a target candidate entity set of the to-be-aligned entities; performing entity disambiguation processing on the target candidate entity set and the to-be-aligned entities to obtain an entity knowledge set; performing relation alignment processing on entity relations of entities in the entity knowledge set by using a large model according to the entity knowledge set, outputting knowledge triples after the relation alignment processing, and constructing a knowledge graph corresponding to the original data based on the knowledge triples. In the technical scheme, knowledge is extracted from multi-modal and multi-source heterogeneous original data to obtain a data set, the data set is mapped to a unified vector space by using a pre-trained large model to obtain to-be-aligned entities, the to-be-aligned entities are standardized and disambiguated by using a target field dictionary to obtain an entity knowledge set, the entity relations of each entity are aligned according to the entity knowledge set to obtain knowledge triples, and a knowledge graph is constructed according to the knowledge triples. The technical scheme can perform entity alignment and disambiguation processing on multi-source heterogeneous data sets, perform relation alignment processing on entity relations, fully utilize original data, automatically construct a knowledge graph, does not need manual data screening and database importing operations, and can solve the problems of low data utilization rate and low accuracy of a knowledge graph. BRIEF DESCRIPTION OF DRAWINGS

[0012] Figure 1 FIG. 1 is a structural schematic diagram of a knowledge graph construction system based on a large model according to an embodiment of the application; Figure 2 FIG. 2 is a flow schematic diagram of a knowledge graph construction method based on a large model according to an embodiment of the application; Figure 3 FIG. 3 is a schematic diagram of a cross-modal semantic alignment method according to an embodiment of the application; Figure 4 FIG. 4 is a flow schematic diagram of another knowledge graph construction method based on a large model according to an embodiment of the application; Figure 5 FIG. 5 is a structural schematic diagram of a knowledge graph construction apparatus based on a large model according to an embodiment of the application; Figure 6 FIG. 6 is a structural schematic diagram of an electronic device according to an embodiment of the application. DETAILED DESCRIPTION

[0013] With reference to the accompanying drawings, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only some of the embodiments of the present application, but not all of them. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of the present application.

[0014] The terms "first", "second", and the like in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally a category and do not limit the number of objects, for example, the first object can be one or more. In addition, "and / or" in the specification and claims indicates at least one of the connected objects, and the character " / ", generally indicates that the front and rear associated objects are in an "or" relationship.

[0015] Figure 1 is a structural schematic diagram of a large model-based knowledge graph construction system according to an embodiment of the present application. As shown in Figure 1 The structure of the large model-based knowledge graph construction system involves four modules: data layer, processing layer, model layer and application layer. The data layer includes input and preprocessing of raw data. The processing layer includes knowledge extraction / multimodal alignment processing of raw data, correction of knowledge graph and storage / reading of knowledge. It also includes knowledge alignment processing through user feedback / knowledge update of the application layer, and dynamic updating of data. The updated data is input into the model layer. The model layer is used for model training of the large model. The application layer is used to output the constructed knowledge graph for user use, and to receive user feedback / knowledge update, etc.

[0016] The large model-based knowledge graph construction method provided by the embodiments of the present application will be described in detail below in combination with the accompanying drawings and specific embodiments and their application scenarios.

[0017] Figure 2 An embodiment of the present application provides a large model-based knowledge graph construction method. The method can be executed by an electronic device, which can include a server and / or a terminal device, such as a vehicle-mounted terminal or a mobile phone terminal. In other words, the method can be executed by software or hardware installed in a large model-based knowledge graph construction device. The method includes the following steps: S202, obtaining raw data of a multi-source service, the raw data including structured data, semi-structured data and unstructured data.

[0018] The original data of the multi-source service includes original data based on the integration of various services in various customer service systems, and the data types cover structured data, semi-structured data and unstructured data. For example, structured service knowledge (such as operator policy files, package fee tables, etc.), customer service common question and answer data, new e-commerce knowledge, complaint work orders, customer service and customer historical dialogue, fault handling steps, and the like.

[0019] The structured data refers to data with a clear data structure and format, which is usually organized in the form of a two-dimensional table. For example, an employee information table.

[0020] The semi-structured data is a data type between structured data and unstructured data, which is used to describe the structure and meaning of data. For example, data in Extensible Markup Language (XML) and JavaScript Object Notation (JSON) formats.

[0021] The unstructured data refers to data without fixed structure and format, which usually includes text, pictures, audio, video and the like. For example, a news article, a photo, a piece of music, and the like are all unstructured data.

[0022] In S204, the original data is subjected to knowledge extraction processing to obtain a data set, which includes a structured sub-data set, an entity text sub-data set and a visual text sub-data set.

[0023] The knowledge extraction processing refers to extracting triple information in the original data. For example, triple information in structured data and semi-structured data is extracted to obtain a structured sub-data set; triple information in text data in unstructured data is extracted to obtain an entity text sub-data set; and triple information in visual branch data in unstructured data is extracted to obtain a visual text sub-data set.

[0024] The structured sub-data set includes a knowledge collection extracted from structured data and semi-structured data.

[0025] The entity text sub-data set includes a knowledge collection extracted from text knowledge in unstructured data.

[0026] The visual text sub-data set includes a knowledge collection extracted from visual knowledge in unstructured data.

[0027] In S206, the data set is mapped to a unified vector space through a pre-trained large model to obtain a to-be-aligned entity, and the to-be-aligned entity and a target field dictionary are subjected to entity alignment processing to obtain a target candidate entity set of the to-be-aligned entity.

[0028] The pre-trained large model is a lightweight model used for cross-modal alignment processing of the data set and generation of knowledge triples. The pre-trained large model is a model obtained by fine-tuning through a knowledge distillation technique, taking the open-source multi-modal large model Qwen2.5-VL-32B-Instruct as a teacher model and its corresponding open-source base model Qwen2.5-VL-7B-Instruct as a student model. Among them, the teacher model generates labeled data and soft labels to guide the student model to learn multi-modal knowledge extraction and alignment capabilities, and finally form a lightweight model suitable for knowledge graph construction.

[0029] The target field dictionary is used to standardize and normalize the to-be-aligned entities, thereby improving data utilization. For example, “package A” and “A package” are unified as “package A”.

[0030] Specifically, the data set is input into the pre-trained large model, and the data set is mapped to a unified vector space by the pre-trained large model to obtain entities of the data set in the unified vector space as to-be-aligned entities, which include one or more entities. By performing entity alignment processing on the to-be-aligned entities and the target field dictionary, a target candidate entity set of the to-be-aligned entities is obtained, which includes multiple candidate entities corresponding to each entity in the to-be-aligned entities. For example, one of the to-be-aligned entities includes a package combination A, and the entities aligned with the target field dictionary include a package combination A1 and a package combination A2. Then, the package combination A1 and the package combination A2 are taken as candidate entities of the package combination A, and the set of candidate entities corresponding to each entity in the to-be-aligned entities is taken as the target candidate entity set of the to-be-aligned entities.

[0031] S208, performing entity disambiguation processing on the target candidate entity set and the to-be-aligned entities to obtain an entity knowledge set.

[0032] The entity disambiguation processing includes: calculating the semantic vector similarity of the context by the large model, outputting the most similar entity between the entity in the to-be-aligned entity and the entity in the target candidate entity set as the entity knowledge set. For example, the entity is apple, which can be a brand or a fruit. The entity needs to be disambiguated. If both represent fruits, only one entity is retained in the entities with the same semantics.

[0033] The target candidate entity set in the target candidate entity set corresponding to each entity in the to-be-aligned entity can be obtained. The entity knowledge set is the set of entities in the data set after standardization and entity disambiguation processing.

[0034] S210, according to the entity knowledge set, the entity relationship of the entity in the entity knowledge set is processed by the large model, and the knowledge triple after the relationship alignment processing is output, and the knowledge graph corresponding to the original data is constructed based on the knowledge triple.

[0035] The entity relationship refers to the relationship between entities in a triple, such as in a triple, in addition to entity A and entity B, the relationship between the two, such as compatibility, replacement or association, etc.

[0036] The entity relationship between the entities in the entity knowledge set is processed by the relationship alignment, and the unique entity relationship between the entities in the triple is obtained as the knowledge triple, and the knowledge graph is constructed according to the knowledge triple after the entity relationship alignment processing of the entity in the entity knowledge set.

[0037] The technical scheme of the embodiment of the present application is adopted, the original data of multiple sources is obtained, the original data includes: structured data, semi-structured data and unstructured data; the original data is processed by knowledge extraction to obtain a data set, the data set includes: a structured sub-data set, an entity text sub-data set and a visual text sub-data set; the data set is mapped to a unified vector space by a pre-trained large model to obtain a to-be-aligned entity, and the to-be-aligned entity and a target field dictionary are processed by entity alignment to obtain a target candidate entity set of the to-be-aligned entity; the target candidate entity set and the to-be-aligned entity are processed by entity disambiguation to obtain an entity knowledge set; according to the entity knowledge set, the entity relationship of the entity in the entity knowledge set is processed by the large model, and the knowledge triple after the relationship alignment processing is output, and the knowledge graph corresponding to the original data is constructed based on the knowledge triple. Among them, the knowledge of the multi-modal and multi-source heterogeneous original data is extracted to obtain a data set, the data set is mapped to a unified vector space by a pre-trained large model to obtain a to-be-aligned entity, the to-be-aligned entity is standardized and disambiguated by a target field dictionary to obtain an entity knowledge set; then the entity relationship of each entity is aligned according to the entity knowledge set to obtain a knowledge triple, and the knowledge graph is constructed according to the knowledge triple, which can perform entity alignment and disambiguation on the multi-source heterogeneous data set, and perform relationship alignment on the entity relationship, fully utilize the original data, automatically construct the knowledge graph, without manual data screening and database import operations, and can solve the problems of low data utilization rate and low accuracy of the knowledge graph.

[0038] In one embodiment, the original data is processed by knowledge extraction to obtain a data set (i.e. S204), which can perform the following steps A1-A7: Step A1, extracting the triple of structured data and semi-structured data in the original data, and generating a structured sub-data set.

[0039] Triples are used to represent entities and the relationships between them.

[0040] Specifically, the original data can be represented as a set D:

[0041] in, Indicates the first Structured data, such as operator policy documents and package tariff tables; This represents semi-structured data, such as business knowledge in JSON format; This represents unstructured data, such as complaint ticket records, voice conversations, customer service Q&A center data, video screenshots, etc. N represents the number of data points in the original data.

[0042] Extract structured data from the raw data, including: directly parsing field values ​​of tabular data (such as package tariff tables and policy documents), using Structured Query Language (SQL) to extract triples, and storing the extracted triples as entity-entity relationship-entity data.

[0043] For example, extracting triples can be represented as:

[0044] The extracted triples are stored as entity-entity relation-entity data, and the structured data can be represented as: .

[0045] e represents an entity, and r represents an entity relationship.

[0046] Extract semi-structured data from the raw data, including: business knowledge in JSON format, defining XML Path Language (XPath) path rules to extract key fields, and verifying data integrity through JSON Schema, which is used to verify whether JSON data conforms to the expected structure, type, format, and constraints.

[0047] For example: {"policy_id": "P001", "title": "5G Family Sharing Package A", "content": {"Price": "500 yuan / month", "Data usage": "100GB"} The key-value pairs of the `content` dictionary are combined with names to form entity-relationship-entity data, which is semi-structured data. may be represented as:

[0048] wherein e represents an entity, r represents an entity relationship, and represent different entities.

[0049] The triples of the structured data and the semi-structured data extracted from the original data are preprocessed, and the preprocessing process includes noise filtering and redundancy elimination, missing value filling, unified data format, and entity disambiguation, which can improve the data quality. Specifically, (1) noise filtering and redundancy elimination: a statistical (term frequency-inverse document frequency, TF-IDF) algorithm is used to calculate the word frequency weight of the triples of the structured data and the semi-structured data, and a threshold is set to filter high-frequency meaningless words, such as stop words and invalid punctuation; regular expressions are used to unify numerical units, such as standardizing “500 yuan / year” to “500 yuan / year” and “500 MB” to “0.5 GB / month”.

[0050] (2) missing value filling: a K-Nearest Neighbors (KNN) algorithm is used to interpolate and fill the missing fields in the triples of the structured data and the semi-structured data, such as filling the missing “package traffic” field according to the average value of similar packages.

[0051] (3) unified data format and entity disambiguation: a standardized target field dictionary is constructed

[0052] , and based on the context embedding vector similarity calculation, the same entities are merged, such as unifying “package A” and “A package” to “package A”. The similarity calculation can use a semantic vector model to obtain a sentence vector, and then use cosine to calculate the similarity. By extracting the triples in the structured data and the semi-structured data in the original data, and through preprocessing for data cleaning, the structured sub-data set after merging the triples in the structured data and the semi-structured data is obtained. The structured sub-data set can be represented as:

[0053] , and the standardized entity library corresponding to the structured sub-data set can be represented as: .

[0054] Step A2, converting the voice dialogue data in the unstructured data into text voice data through a voice recognition system.

[0055] ​​Unstructured data in raw data, such as image / video data (visual branch data) Word and PDF text data (such as equity products, package handling process diagrams, device fault handling step videos, etc.) And voice dialogue data between customers and customer service Store unstructured data as multi-modal data. Unstructured data can be represented as:

[0056] Due to the insufficient coverage of single modal data, the multi-modal data after preliminary processing can be associated with structured knowledge data sets And standardized entity library And standardize entity library To build a unified multi-modal knowledge representation space, eliminate business conflicts and form a global knowledge base.

[0057] First, the voice dialogue data in the unstructured data is converted into text voice data Through the speech recognition system .

[0058] Step A3, input the text voice data and text data in the unstructured data into the large model, and determine the entity text sub-data set through the large model.

[0059] The large model is used to extract triples of text data in unstructured data.

[0060] Specifically, the converted text voice data and text data in the unstructured data are input into the large model. The large model can extract the business entities and entity relationships corresponding to the entities from and to form triples.

[0061] For example, the large model uses the prompt word template : Please identify the business entities and entity relationships in the operator customer service scenario from the following text and classify the output: Entity categories: devices (such as routers, optical modems), fault types (such as network disconnection, poor signal), service types (such as broadband installation, package change), user identity (such as enterprise customers, home users).

[0062] Entity output format: {"entity": ["optical modem", "router"]} The relationship output format is: {"relationship": [("router", "causes", "network outage"), ("optical modem", "configuration error", "network outage")]} The text content is as follows: {word_content} Output: The final output of the large model includes entities, entity relationships, and event results, for example: {"entity": ["user", "package A", "network delay"], "relationship": [("user", "complaint reason", "network delay"), ("user", "unsubscribe", "package A")]}.

[0063] The processed entity text sub-data set is a set of triples of entities , which can be represented as:

[0064] E is the number of entities in the entity text sub-data set, and R is the number of entity relationships in the entity text sub-data set. is represented as the vector of head entity h; is represented as the vector of tail entity t; is represented as the entity relationship.

[0065] Step A4: Through the large model, the visual branch data in the unstructured data is parsed and processed to obtain the first visual text data.

[0066] The large model is also used to extract triples of visual branch data in unstructured data.

[0067] The pre-trained large model is used to parse the content of the visual knowledge branch data and generate the first visual text data of structured information . Specifically, the image / video is input into the large model, and through the prompt words of the large model, the key words in the image / video are recognized to generate the first visual text data.

[0068] For example, the following prompt word template can be used to guide the large model to output: Visual content analysis: Please identify the key objects, scenes, layouts, or text in the image / video, and describe the details such as the position, color, shape, and number of objects.

[0069] For device failure images, please analyze and extract key information: Input: Device failure screenshot (e.g., router indicator light abnormal).

[0070] Output format: ​{"device" : "router", "exception features" : {"indicator light" : "red blinking", "interface" : "loose network cable"}, "suggested actions" : "Check the network cable connection and restart the router."}

[0071] Step A5, the visual branch data in the unstructured data is identified by optical character recognition technology to obtain second visual text data.

[0072] The visual branch data in the unstructured data is identified again by optical character recognition technology, such as the text content in the image / video is identified by optical character recognition technology to obtain second visual text data , which is used to supplement the results generated by the large model. The visual branch data in the unstructured data can be identified again.

[0073] Step A6, generating a visual text sub-data set according to the triplets in the first visual text data, the second visual text data and the visual branch data.

[0074] The first visual text data , the second visual text data result identified by optical character recognition technology and the modal information of the original image / video data, i.e., the visual knowledge branch data are merged to form triplets, which constitute a visual text sub-data set , which can be represented as:

[0075] Step A7, the structured sub-data set, the entity text sub-data set and the visual text sub-data set are used as a data set.

[0076] The above structured sub-data set , the entity text sub-data set and the visual text sub-data set are determined as a data set.

[0077] In this embodiment, the structured data and the triples in the semi-structured data in the original data are extracted, and data cleaning processing such as disambiguation processing is performed to obtain a structured sub-data set; then, through a speech recognition system and a large model, the triples in the text data corresponding to the voice dialogue data in the unstructured data are extracted to obtain an entity text sub-data set; at the same time, the visual branch data in the unstructured data is identified multiple times, such as through a pre-trained large model and through an optical character recognition technology, to obtain first visual text data and second visual text data, and the twice identification results and the visual branch data in the unstructured data are taken as a visual text sub-data set. Through multiple identifications of the visual branch data in the unstructured data, disambiguation processing exists in the subsequent alignment process, so that there is no need to consider the problem of repeatedly obtaining entities and the like, and the accuracy of determining the triples of the visual branch data in the unstructured data can be improved, and unrecognized data can be prevented. The structured sub-data set, the entity text sub-data set, and the visual text sub-data set are taken as a data set.

[0078] In one embodiment, the entity alignment processing is performed on the to-be-aligned entity and the target domain dictionary to obtain a target candidate entity set of the to-be-aligned entity (i.e., S206), and the following steps B1-B4 can be executed: Step B1, the cosine similarity of the to-be-aligned entity and the entities in the target domain dictionary is calculated, and based on the cosine similarity, a preset number of entities in the target domain dictionary are determined, and based on the preset number of entities, a first candidate entity set is determined.

[0079] Before the entity alignment processing is performed on the to-be-aligned entity and the target domain dictionary, the data set is input into a pre-trained large model, and the data set is mapped to a unified vector space by the large model to obtain the to-be-aligned entity.

[0080] Specifically, the multi-modal entity data set in the data set is input into a pre-trained large model, and the entity data set can be represented as . Wherein m e {text, image, audio} represents a modal type, e is the original feature thereof, and N is the number of entities. Through the encoder , different modalities are mapped to a unified semantic space, i.e., a shared semantic space: , to obtain the entity in the shared semantic space, which can be represented as:

[0081] Wherein, e is a normalized embedding vector, i.e., a vector mapped to the shared semantic space, represents the semantic vector of the i-th entity, and m is the m-th modal type, represents the semantic vector of the jth entity, n is the nth type of modality.

[0082] The target domain dictionary can be represented as: , j represents the jth entity, and a total of M entities are included. The entities in the target domain dictionary can be represented as .

[0083] The calculation method of the cosine similarity of the entities to be aligned and the entities in the target domain dictionary can be represented by the following formula:

[0084] , wherein represents the entity to be aligned obtained from the unified vector space, and the threshold value can be set by the user, such as a threshold value of 0.9. , which is used to filter the matching results to obtain a preset number of first candidate entity sets in the target domain dictionary, for example, the top 5 entities in the target domain dictionary and the entities in the entity to be aligned are taken as the first candidate entities, and the first candidate entity set of all entities in the entity to be aligned is the first candidate entity set. By calculating the cosine similarity of the entities in the entity to be aligned and the entities in the target domain dictionary, the first candidate entity corresponding to the entity in the target domain dictionary is obtained. According to the cosine similarity of the entities in the entity to be aligned and the entities in the target domain dictionary, the entities in the target domain dictionary corresponding to each entity in the entity to be aligned are ranked from high to low, and the top preset number of entities are taken as the first candidate entities. According to the candidate entities, the first candidate entity set is determined, wherein the first candidate entity set includes one or more first candidate entities of each entity in the entity to be aligned in the target domain dictionary. For example, the entity to be aligned includes package A and package B, and the entities in the target domain dictionary include package A1, package A2, package A3, and package B1, package B2, and package B3. By calculating the cosine similarity of package A and package A1, package A2, and package A3, the top 2 package A1 and package A2 are obtained, and then package A1 and package A2 are both first candidate entities of package A. By calculating the cosine similarity of package B and package B1, package B2, and package B3 by formula A, the top 2 package B1 and package B3 are obtained, and then package B1 and package B3 are both first candidate entities of package B. Wherein, package A1 and package A2 of package A, and package B1 and package B3 of package B are taken as the first candidate entity set of the entity to be aligned. Generally, each entity in the entity to be aligned corresponds to its own first candidate entity.

[0085] Step B2, obtaining the entities in the entity to be aligned that are not matched with the entities in the target domain dictionary as the entities to be matched.

[0086] In the similarity matching of the entity to be aligned and the target field dictionary in step B1, when the entity to be aligned does not have similarity with the entity in the target field dictionary, the entity to be aligned that does not match the entity in the target field dictionary is taken as the entity to be matched. may be represented as: , is a temperature parameter, and the remaining letters are consistent with the above and will not be described in detail.

[0087] In step B3, the entity alignment processing is performed between the entities to be matched by using the contrast learning function, and a second candidate entity set is obtained.

[0088] The contrast learning function is used to optimize the loss function when the entities to be matched are matched. The contrast learning function The specific calculation is shown in the following formula:

[0089] wherein is a negative sample set, i.e., a set of entities to be matched, is a temperature parameter. By optimizing the loss function, the semantic alignment capability between heterogeneous modalities is enhanced, and at the same time, the clustering algorithm is used to merge the entities with high similarity in the set of entities to be matched to generate a second candidate entity set. The letters are consistent with the above and will not be described in detail.

[0090] In step B4, the first candidate entity set and the second candidate entity set are taken as a target candidate entity set.

[0091] The candidate entities obtained by matching the entity to be aligned and the target field dictionary are taken as a first candidate entity set, the alignment processing is performed between the entities to be matched that are not matched with the target field dictionary to obtain a second candidate entity set, and the first candidate entity set and the second candidate entity set are taken as a target candidate entity set.

[0092] In this embodiment, by matching the entity to be aligned and the entity in the target field dictionary, a first candidate entity set is obtained, the entity to be aligned that is not matched with the entity in the target field dictionary is taken as an entity to be matched, the entity alignment processing is performed between the entities to be matched by using the contrast learning function, a second candidate entity set is obtained, and the first candidate entity set and the second candidate entity set are taken as a target candidate entity set. The entity to be aligned can be accurately and preliminarily aligned, and the entity to be aligned is more standardized.

[0093] In one embodiment, the entity disambiguation processing is performed on the target candidate entity set and the entity to be aligned to obtain an entity knowledge set (i.e., S208), and the following steps C1-C2 can be executed: Step C1, construct a prompt engineering template based on the target candidate entity set and the entity to be aligned, and determine the target candidate entity set after entity disambiguation processing. The prompt engineering template is used to guide the large model to complete the entity disambiguation processing.

[0094] Specifically, the prompt words can be as follows: [Role setting] You are a professional knowledge graph engineer responsible for completing entity standardization alignment tasks through semantic analysis and business rules.

[0095] [Input] Entity to be aligned: {entity description}; Target candidate entity set: {(entity identifier (ID), semantic vector, attribute feature)}.

[0096] [Task description] Select the most suitable entity ID from the target candidate entity set according to the following rules: 1. Semantic matching degree: Calculate the semantic vector similarity (such as cosine similarity) between the entity to be aligned and the entities in the target candidate entity set.

[0097] 2. Business rules: Preferentially match entities with completely consistent attribute features (such as device model, service type).

[0098] 3. Conflict resolution: If there are multiple high-matching-degree entities in the target candidate entity set, select the most reasonable entity ID based on the context business logic.

[0099] [Output requirement] Return the result in strict JSON format, and prohibit adding any additional text or annotations.

[0100] JSON field requirements: `"matched_entity"`: matched entity ID (string). `"confidence_score"`: matching confidence (numeric value, range [0, 1]). `"matching_reason"`: matching reason (string). `"candidate_scores"`: candidate entity matching score (object, key is entity ID, value is score).

[0101] Step C2, through the prompt engineering template, make the large model output the probability value of each target candidate entity in the target candidate entity set, and determine the target candidate entity corresponding to each entity in the entity to be aligned based on the probability value.

[0102] Specifically, in the target candidate entity set after entity disambiguation processing obtained through step C1, determine the target candidate entity corresponding to each entity. The calculation of the target candidate entity output by the large model can be represented as: probability distribution, i.e. probability value The final target candidate entity can be represented by the following formula table:

[0103] wherein, represents the target candidate entity, C represents the target candidate entity set, that is, in the target candidate entity set corresponding to the entity pair to be aligned, the candidate entity with the maximum probability in the target candidate entity set corresponding to each entity of the entity pair to be aligned is taken as the target candidate entity of the entity pair to be aligned.

[0104] Step C3, taking the target candidate entity corresponding to each entity in the entity pair to be aligned as the entity knowledge set.

[0105] The entity knowledge set can be represented as the target candidate entity after the entity disambiguation processing. 、 and , including: the standardized business entities aligned according to the target field dictionary, forming a new knowledge set .

[0106] In the embodiment, the prompt engineering template is constructed through the target candidate entity set and the entity pair to be aligned, so that the large model can perform entity disambiguation processing on the target candidate entity set and the entity pair to be aligned, obtain the target candidate entity set and the entity pair to be aligned after the disambiguation processing, and output the target candidate entity corresponding to each entity in the entity pair to be aligned through the large model, and determine the entity knowledge set. The entities with logical problems in the target candidate entity set and the entity pair to be aligned can be eliminated for standardization processing, and the target candidate entity of each entity in the entity pair to be aligned is determined by calculating the probability value, and the target candidate entity is taken as the entity knowledge set.

[0107] In one embodiment, according to the entity knowledge set, the entity relationship of the entity in the entity knowledge set is processed by the large model, and the knowledge triple after the relationship alignment processing (namely S210) is output, which can be executed as follows: Step D1, according to the entity knowledge set, determining the entity in the entity knowledge set and the entity relationship of the entity in the entity knowledge set.

[0108] According to the entity knowledge set output by the large model in step C3, the entity relationship of each data pair (entity-entity relationship-entity) in the entity knowledge set is further aligned and fused.

[0109] Step D2, determining the candidate similar relationship corresponding to the entity relationship of the entity in the entity knowledge set through the attribute feature similarity function in the large model.

[0110] The attribute feature similarity function is used to represent the corresponding candidate similar relationship of the entity in the entity knowledge set. The candidate similar relationship includes one or more.

[0111] The relationship pair of the target candidate entity can be represented as: The relationship pairs are processed by cross-modal alignment. Specifically, an attribute feature similarity function is defined In combination with the Euclidean distance and the cosine similarity, it can be expressed as:

[0112] wherein, is used to balance the structural difference and the semantic similarity weight, and the value of a can be adjusted. The top 5 candidate similar relationships are selected according to the similarity, and a threshold is set The matching results are screened. If the threshold is less than 0.6, it is determined as a separate relationship, and if it is greater than 0.6, it indicates that the relationship is entangled. h and t represent two entities, and r represents the entity relationship between h and t. The Euclidean distance represents the actual distance between two entities. The cosine similarity determines the similarity of two vectors by calculating the cosine value of the included angle between the two vectors. The values of the similarity ranking and the set threshold are not specifically limited.

[0113] According to the attribute feature similarity function, the relationship alignment of the entity relationship is completed by constructing a prompt engineering template to guide the large model, and a plurality of candidate similar relationships are obtained.

[0114] As an example, the prompt words of the large model can be as follows: [Role setting] You are a professional knowledge graph engineer responsible for determining whether there is a mergable relationship between the source entity and the target entity based on entity description and business rule judgment, and outputting standardized triples.

[0115] [Input] The source entity is the first entity, such as h: {"entity_id": "E123", "description": "XX brand X model mobile phone"} The target entity is the first entity, such as t: {"entity_id": "E456", "description": "XX brand X model mobile phone case"} The candidate similar relationship set: ["compatible", "replace", "associated"] [Task description] 1. Semantic matching degree analysis: Compare the description texts of the source entity and the target entity, calculate the semantic similarity (such as cosine similarity), and extract the key attributes (such as brand, model) for accurate matching.

[0116] 2. Business rule verification: According to the relationship type (such as "compatible") in the candidate similar relationship set, the rationality is judged in combination with the business logic (such as commodity recommendation rules).

[0117] 3. Relationship decision: If the semantic matching degree is greater than or equal to 0.7 and the business rules support, output the triple (h, r, t), where h is the identifier of the source entity, t is the identifier of the target entity, and r is the matched entity relationship type.

[0118] [Output requirements] Return the result in strict JSON format, and prohibit adding any additional text or comments.

[0119] JSON field requirements: `"is_related"`: Whether to establish a relationship (Boolean value).

[0120] `"relation_type"`: Relationship type (string, must belong to the set of candidate similar relationships).

[0121] `"confidence_score"`: Decision confidence (numeric value, range [0, 1]).

[0122] `"reasoning_details"`: Decision reason (object, containing semantic matching and business rule analysis).

[0123] Step D3, according to the attention mechanism of the large model, calculate the confidence of the candidate similar relationship, through the confidence, determine the target entity relationship of the entity in the entity knowledge set.

[0124] Specifically, the attention mechanism of the large model calculates the confidence of the candidate similar relationship, which can be represented as follows:

[0125] where, is the relationship projection matrix, represents the semantic embedding vector of the head entity h; represents the semantic embedding vector of the tail entity t; combine and in dimension, after splicing, it is 2k-dimensional, and finally take triple (threshold , numerical value is not specifically limited). That is, the confidence of the candidate similar relationship in step D2 is greater than the preset threshold, and the entity relationship is taken as the target entity relationship.

[0126] As an example, the large model output result: {"is_related": true, "relation_type": "compatible", "confidence_score": 0.85, "reasoning_details": { "semantic_match_score": 0.80, "semantic_analysis": "The source entity description contains 'wireless router', and the target entity description contains 'mesh extender'. They belong to the same device category and their functions are complementary.", "attribute_match": { "Brand": {"source": "TP-Link", "target": "TP-Link", "match": true}, Model number: {"source": "AX1810", "target": "TL-WPA4220", "match":false}} "business_rule_support": "According to network device compatibility rules, different models of devices from the same brand are defined as 'compatible' if their functions are complementary and they have no conflicting parameters (such as inconsistent frequency bands)." The results of the large model are integrated with entity-entity-relationship-entity pairs that are not intertwined to form the final knowledge triples. .

[0127] Step D4: Based on the target entity relation and the entity corresponding to the target entity relation, output the knowledge triples corresponding to the original data after relation alignment processing.

[0128] Based on the target entity relationships of the entities in the entity knowledge set output by the large model in step D3, determine the target relationship pairs (entity-target entity relationship-entity), where the entity relationships between entities are unique, and determine the target entity relationship pairs as the knowledge triples corresponding to the original data.

[0129] A knowledge triple can be represented as: A knowledge graph constructed based on knowledge triples can be represented as: .in, For a collection of entities, For a set of entity relations, .

[0130] Where E and R represent the sets of entities and entity relationships in the knowledge graph, respectively. R E: In mathematics, the "Cartesian product" represents all possible combinations of "entity-entity relation-entity" triples. : indicates that T is a valid subset of the Cartesian product, containing only verified entity-entity relation-entity triples in the knowledge graph that conform to business facts.

[0131] In summary, as shown in Figure 3 Fig. 1 is a schematic diagram of a cross-modal semantic alignment method. In a pre-trained large model, input multi-modal data and a target domain dictionary. The multi-modal data is a data set corresponding to the original data. The multi-modal data and the target domain dictionary are semantically embedded and mapped to a unified semantic space. The aligned entities obtained after the semantic vector embedding of the multi-modal data are standardized and disambiguated. It is determined whether the aligned entities match the target domain dictionary. If not, the unmatched entities are treated as matching entities, and the matching entities are clustered to obtain a second candidate entity set. If yes, output the first candidate entity set matched with the target domain dictionary. The first candidate entity set and the second candidate entity set are output as the target candidate entity set. The entity disambiguation process is performed by the large model, and then the relationship alignment process is performed to obtain the knowledge triple.

[0132] In this embodiment, the entity relationship between the entities in the entity knowledge set is determined by the large model. After the entity alignment process, the relationship alignment process is performed on the entity relationship between the entities. All the data in the triple can be fully aligned to obtain the knowledge triple corresponding to the original data, so as to construct the knowledge graph based on the knowledge triple. The knowledge triple obtained through multiple alignment processes fully utilizes the original data and has standardization and uniqueness, thereby improving the data utilization rate of the constructed knowledge graph and the accuracy of the knowledge graph.

[0133] In one embodiment, the method further comprises verifying the knowledge graph.

[0134] Specifically, the knowledge graph constructed based on the knowledge triple is verified by the two preset verification mechanisms to obtain the verified knowledge graph. The two preset verification mechanisms include first-order logic rule verification and path consistency algorithm.

[0135] (1) First-order logic rule verification: obtain the entities of the knowledge triple in the knowledge graph, predefine the entity attributes and constraint conditions of the entities of the knowledge triple in the knowledge graph, and judge the semantic consistency of the entity attributes based on the constraint conditions.

[0136] For example, the entity attribute is the fee, the constraint condition is the data type, and the predefined first-order logic rule is , which means that all entities marked with the "fee" attribute must satisfy the constraint condition of "numeric type" (such as amount, percentage, etc. quantifiable form). If a non-numeric type attribute entity (such as text free) is detected, a correction process is triggered to correct the knowledge triple in the knowledge graph.

[0137] (2) Path consistency algorithm: obtain the knowledge triple in the knowledge graph, and detect the logical conflicts between the knowledge triples.

[0138] For example, if the knowledge triples in the knowledge graph are obtained as: (for example, "A is a package, A is a package, A is a price, A is 50 yuan"), (for example, "A is a package, A is a package, A is a price, A is not 50 yuan"), it is determined that there is a contradiction between the two knowledge triples, and the two are marked as conflicts. By traversing the adjacency relationship of the knowledge graph through a graph traversal algorithm, combinations of triples that violate transitivity, symmetry or antisymmetry are dynamically identified, and the knowledge graph is modified.

[0139] It should be noted that the knowledge graph or the verified knowledge graph is stored in a graph database to store the knowledge graph, and a Cython query interface is designed to achieve efficient access. For example, an index is established for high-frequency query relationships, and a query interface is designed to support multi-modal queries and provide a batch export interface.

[0140] In this embodiment, the dual verification mechanism of the finally constructed knowledge graph can improve the accuracy of constructing the knowledge graph.

[0141] In one embodiment, the method can further perform steps E1-E3 as follows: Step E1, data monitoring processing is performed on the original data, when a new entity is monitored, the new entity and the new entity relationship corresponding to the new entity are obtained, and a new knowledge fragment is generated.

[0142] Specifically, a real-time event stream processing pipeline is constructed based on a distributed stream computing framework Apache Flink to monitor the millisecond-level of the multi-source heterogeneous data stream of the original data, which is used for dynamic capture of business knowledge state and instant synchronization of the knowledge graph.

[0143] The monitored new entity can be an entity generated when user feedback, operation and maintenance instructions or data source changes.

[0144] The data monitoring processing of the original data includes: performing feature analysis and pattern recognition on the real-time input data, extracting the entity in the real-time input data as a new entity The new entity relationship is represented as E, forming a new knowledge fragment .

[0145] Step E2, the entities in the new knowledge fragment and the entities in the knowledge graph are processed for semantic association matching, and the unmatched entities that are not matched between the entities in the new knowledge fragment and the entities in the knowledge graph are obtained.

[0146] The newly added knowledge fragment in step E1 is matched with the entity in the knowledge graph based on semantic association matching processing to determine whether there is a naming conflict and attribute heterogeneity between the newly added knowledge fragment and the entity in the knowledge graph.

[0147] The semantic association matching processing includes the entity alignment processing, the relationship alignment processing, and the disambiguation processing.

[0148] When there is an unmatched entity between the entity in the newly added knowledge fragment and the entity in the knowledge graph, or the entity relationship in the newly added knowledge fragment and the entity relationship in the knowledge graph are not matched, the corresponding entity is also determined as an unmatched entity.

[0149] In step E3, the similarity value between the unmatched entity and the entity in the knowledge graph is calculated to obtain a target entity, and the target entity includes an entity in the knowledge graph with the highest similarity value, which is used to dynamically update the knowledge graph.

[0150] Specifically, based on the context embedding vector, the similarity value between the unmatched entity and the entity in the knowledge graph in step E2 is calculated, and according to the similarity value, the entity node with the highest similarity value of the entity in the knowledge graph is determined as the optimal target entity. The unmatched entity is replaced by the target entity, and the newly added relationship is verified through the two preset verification mechanisms to filter noise.

[0151] The knowledge graph is dynamically updated through the target entity. Specifically, the state of the knowledge graph at time t is The real-time updating process follows the following incremental merging formula:

[0152] represents the incremental set of the newly added knowledge fragment composed of the newly added entity , the newly added entity relationship E, and the node weight adjustment. is the dynamically updated knowledge graph.

[0153] The updated knowledge graph automatically generates a structured training sample for subsequent training of a large model to perform real-time online fine-tuning on the parameters of the large model, and realize lightweight incremental updating of the model parameters. At the same time, the evolution log of the dynamic knowledge graph is injected into a streaming computing framework to optimize the feature extraction strategy of the subsequent event stream.

[0154] ​​In this embodiment, through detection processing of the original data, based on the stream computing framework and the multi-modal knowledge alignment technology, the knowledge graph is automatically and dynamically updated at a second level. Through the detection of real-time incremental updating mechanism, the knowledge correction updating link is triggered, and compared with the related art, the limitation of the knowledge graph updating relying on manual maintenance is solved, the problem of outdated knowledge and updating not in time is solved, and the knowledge in the knowledge base can be updated in real time, and the system response timeliness is improved.

[0155] In one embodiment, the large model is trained by the following method, and the following steps F1-F4 can be executed: Step F1, obtain a sample data set and a sample knowledge graph, and input the sample data set into a large model to be trained, the large model to be trained comprising: an open source basic model corresponding to an open source multi-modal large model, wherein the open source multi-modal large model is used as a teacher model, and the open source basic model is used as a student model, to generate labeled data and probability distribution labels based on the teacher model to guide the fine-tuning of the student model.

[0156] The large model to be trained adopts a "teacher-student" knowledge distillation architecture, and the open source multi-modal large model Qwen2.5-VL-32B-Instruct is used as the teacher model, and the lightweight open source basic model Qwen2.5-VL-7B-Instruct corresponding thereto is used as the student model. During the training process, the soft labels of the labeled data and the probability distribution generated by the teacher model guide the fine-tuning of the student model Qwen2.5-VL-7B-Instruct, so that the student model inherits the multi-modal understanding and knowledge extraction capability of the teacher model. The labeled data include data labeled after the data output by the teacher model is processed.

[0157] Obtain a plurality of sample data sets, wherein the sample data set comprises: a sample structured sub-data set, a sample entity text sub-data set and a sample visual text sub-data set.

[0158] The sample structured sub-data set is obtained based on SQL query, content dictionary key value, etc.

[0159] The sample entity text sub-data set and the sample visual text sub-data set adopt a two-stage knowledge driven generation strategy.

[0160] In the first stage, the text large model Qwen3-235B-A22B with a parameter amount of 235B of a hybrid expert architecture is used to generate basic data of text information extraction (such as entity and entity relationship labeling), and at the same time, the open source multi-modal large model Qwen2.5-VL-32B-Instruct (teacher model) is used to generate labeled data of visual knowledge extraction (such as image / video content analysis), so as to ensure that the data covers business scenarios such as package benefits and fault handling and has uniqueness.

[0161] In the second stage, the teacher model Qwen2.5-VL-32B-Instruct converts the above-mentioned labeled data into soft labels of probability distribution, as a training guidance signal of the student model Qwen2.5-VL-7B-Instruct, and further obtains a sample entity text sub-data set and a sample visual text sub-data set.

[0162] The obtained sample data set and the soft label generated by the teacher model are input into the large model to be trained (the student model Qwen2.5-VL-7B-Instruct), and the parameters of the student model are fine-tuned by jointly optimizing the loss function (including data cross-entropy loss and knowledge graph similarity loss), so that the student model learns the sample data set while inheriting the multi-modal semantic alignment ability of the teacher model through the soft label, and finally obtains the trained large model. The soft label represents the probability distribution.

[0163] In step F2, the sample data set is mapped to a unified vector space by the large model to be trained, and the sample to-be-aligned entity corresponding to the sample data set is obtained.

[0164] Specifically, the Translation Embedding (TransE) model and the learning model TransR are used to jointly embed the sample data set of structured knowledge in the knowledge graph into a low-dimensional vector space. The TransE model maps the triplets into vector relationships , where , and , respectively represent the knowledge embedding vectors of the head entity, the entity relationship, and the tail entity. The TransR model introduces a relationship space mapping matrix , , which satisfies .

[0165] where h represents the original embedding vector of the head entity; represents the space mapping matrix of the entity relationship r, with a dimension of d*k, where d is the entity space dimension and k is the relationship space dimension, used to project the entity vector from the entity space to the relationship space dedicated to the entity relationship r; represents the relationship space embedding vector obtained after the head entity h is projected; t represents the original embedding vector of the tail entity; represents the relationship space embedding vector obtained after the tail entity t is projected; r represents the embedding vector of the entity relationship r; The core logic of TransR is that the projection vector of the head entity in the entity relationship r space plus the vector of the entity relationship r should be approximately equal to the projection vector of the tail entity in the entity relationship r space, ensuring that the logic of "entity-entity relationship-entity" is reasonable, for example: the logic of Package A (after projection) + tariff + 500 yuan / month (after projection) is consistent.

[0166] The knowledge embedding vector is used as an external feature, i.e., the sample entity to be aligned, which is concatenated with the word vector in the target field dictionary, such as the word vector of the customer service dialogue text, and then input into the large model to be trained.

[0167] Step F3: Input the sample entity to be aligned and the probability distribution label into the large model to be trained, and output the sample knowledge triple through the fusion formula of the embedding layer of the large model to be trained.

[0168] In the large model of the Transformer architecture, the vector of the knowledge graph is integrated into the embedding layer, so that the model can understand and utilize the entity relationship and attribute information specific to the customer service field while learning language knowledge. Specifically, the embedding layer fusion formula is:

[0169] Embedding layer fusion is to integrate the business knowledge of the knowledge graph and the language knowledge of the text in the bottom layer of the large model to be trained, i.e., the embedding layer, to ensure that the large model to be trained has business understanding ability from the input stage. By inputting the sample entity to be aligned into the large model to be trained and using the fusion formula of the embedding layer of the large model to be trained, the large model learns language knowledge and inherits the multi-modal semantic alignment ability of the teacher model through the probability distribution label of the teacher model, and outputs the sample knowledge triple generated by the sample data set.

[0170] Step F4: Train the large model to be trained by using the sample knowledge triple, the sample knowledge graph, and the joint loss function, to obtain the trained large model.

[0171] The joint optimization loss function L can be represented as:

[0172] wherein, represents the large model to be trained The cross-entropy loss on the training data set is used to measure the difference between the model prediction result and the true label, ensuring that the large model to be trained can accurately understand and generate natural language. is based on the knowledge graph loss of the loss function is a weight parameter for balancing the data-based loss and the knowledge graph-based loss, and the best value thereof is determined through experiments to ensure that the basic ability of the large model to natural language processing is not impaired while improving the understanding of the large model to be trained for customer service business knowledge. is a weight parameter for balancing the data-based loss and the knowledge graph-based loss, and the best value thereof is determined through experiments to ensure that the basic ability of the large model to natural language processing is not impaired while improving the understanding of the large model to be trained for customer service business knowledge.

[0173] It should be noted that the pre-trained large model can also be used to optimize the knowledge graph. Specifically, the trained large model is applied to real-time inference services, such as extracting text information, entity disambiguation processing, and relationship alignment processing, which can improve the accuracy of multi-modal dialogue understanding. On the other hand, it is applied to dynamic updating of the knowledge graph, such as extracting incremental knowledge and detecting and correcting conflicts in the knowledge graph.

[0174] The inference result generated by the user query (such as base station signal problem, device setting problem) is extracted by the pre-trained large model to extract new entities and new entity relationships. If the inference result (such as sample knowledge triple) of the pre-trained large model conflicts with the existing knowledge triple of the knowledge graph, a confidence threshold filtering mechanism is triggered, as follows: If Confidence( )> , then update the knowledge graph: (h, , t), where is a manually set threshold value that can ensure the reliability of the new knowledge, represents an entity relationship, h represents a head entity, and t represents a tail entity.

[0175] In this embodiment, the large model to be trained is trained by extracting the sample data set. The architecture of the large model to be trained inherits the knowledge transfer ability of the teacher model, and the large model to be trained is trained by optimizing the loss function, so that the joint optimization loss function converges, and when the result output by the large model to be trained based on the sample data set, for example, the sample knowledge triple and the sample knowledge graph, reaches the expected effect, for example, they are consistent, the training of the large model to be trained is completed, and the pre-trained large model is obtained.

[0176] In summary, by storing the constructed knowledge into the graph database through the processing layer, the operator customer service scenario-oriented agent-assisted robot question-answering system, knowledge base search recommendation platform and other applications can be supported to provide intelligent services for users. Specific application examples are as follows: (1) Agent-assisted robot question-answering system: by generating a Cython query statement, an interface is called to respond to the business consultation needs of customer service agents in real time. For example, when a user asks about "package traffic transfer rules", the system first matches the entity "package-traffic transfer" in the knowledge graph, retrieves the associated relationships (such as "applicable conditions", "exceptional cases") through a graph traversal algorithm, and generates a natural language explanation (such as "traffic transfer is only applicable to primary card users, and vice card traffic cannot be transferred") in combination with a large model. For multi-modal queries (such as user uploading of device fault screenshots), the system calls a large model to analyze image content and extract key information such as "router indicator light abnormality", and then outputs a graphic and text guide to the solution through the fault type and solution associated by the knowledge graph. (2) Knowledge base search recommendation platform: based on the semantic association of user historical consultation records and the knowledge graph, a collaborative filtering algorithm is used to recommend related business knowledge. For example, a user has recently consulted about "broadband installation process", and the system will recommend related content such as "router configuration guide" and "network speed testing method", and generate an abstract through a large model to improve the retrieval efficiency of customer service personnel. At the same time, the platform supports multi-modal search (such as voice query and picture upload), and matches the most relevant knowledge items through cross-modal semantic alignment technology. (3) Dynamic knowledge evolution monitoring: through the Flink streaming computing framework, user feedback (such as "recommended content is inaccurate"), customer service agent correction suggestions and new business policy change events are collected in real time to trigger the incremental update process of the knowledge graph. For example, if a user complains that "the fee adjustment of a certain package is not synchronized", the system automatically identifies the conflicting entities and starts the entity disambiguation process and relationship alignment process driven by the large model, corrects the knowledge triples in the knowledge graph, and optimizes the parameters of the large model through online fine-tuning, forming a closed-loop mechanism of "data backflow-large model iteration-service optimization". This can ensure that the knowledge graph is highly synchronized with the business dynamics, significantly reducing the cost of manual maintenance.

[0177] Figure 4 is a flowchart of another method for constructing a knowledge graph based on a large model according to an embodiment of the present application, as shown in Figure 4 , the method comprises the following steps: S401, obtaining original data of multi-source business, the original data comprising: structured data, semi-structured data and unstructured data.

[0178] S402, extracting triples of structured data and semi-structured data in the original data, and generating a structured sub-data set.

[0179] S403, convert the voice dialogue data in the unstructured data into text voice data through a voice recognition system, input the text voice data and the text data in the unstructured data into a large model, and determine an entity text sub-data set through the large model.

[0180] S404, analyze and process the visual branch data in the unstructured data through the large model to obtain first visual text data, and identify the visual branch data in the unstructured data through an optical character recognition technology to obtain second visual text data.

[0181] S405, generate a visual text sub-data set according to the first visual text data, the second visual text data, and the triple in the visual branch data.

[0182] S406, take the structured sub-data set, the entity text sub-data set, and the visual text sub-data set as a data set.

[0183] S407, calculate the cosine similarity of the entity to be aligned and the entity in the target field dictionary, determine a preset number of entities in the target field dictionary according to the cosine similarity, and determine a first candidate entity set based on the preset number of entities.

[0184] S408, obtain an entity that is not matched with the entity in the target field dictionary in the entity to be aligned as a to-be-matched entity, and perform entity alignment processing on the to-be-matched entities through a contrast learning function to obtain a second candidate entity set.

[0185] S409, take the first candidate entity set and the second candidate entity set as a target candidate entity set.

[0186] S410, construct a prompt engineering template according to the target candidate entity set and the entity to be aligned, and the prompt engineering template is used to guide the large model to complete entity disambiguation processing.

[0187] S411, make the large model output a probability value of each target candidate entity in the target candidate entity set through the prompt engineering template, and determine the target candidate entity corresponding to each entity in the entity to be aligned based on the probability value.

[0188] S412, take the target candidate entity corresponding to each entity in the entity to be aligned as an entity knowledge set.

[0189] S413, determine the entity in the entity knowledge set and the entity relationship of the entity in the entity knowledge set according to the entity knowledge set.

[0190] S414, perform relationship alignment processing on the entity relationship of the entity in the entity knowledge set through the large model, output a knowledge triple after the relationship alignment processing, and construct a knowledge graph corresponding to the original data based on the knowledge triple.

[0191] S415, time data monitoring processing is performed on the original data, when a new entity is monitored, the new entity and the new entity corresponding to the new entity relationship are obtained, and a new knowledge fragment is generated.

[0192] S416, the entities in the new knowledge fragment and the entities in the knowledge graph are processed for semantic association matching, and the unmatched entities of the entities in the new knowledge fragment and the entities in the knowledge graph are obtained.

[0193] S417, the similarity value of the unmatched entity and the entity in the knowledge graph is calculated, and the target entity is obtained, which is used for dynamically updating the knowledge graph.

[0194] The specific process of S401 to S417 is described in detail in the above embodiment, which will not be repeated here.

[0195] The technical scheme of the embodiment of the application is adopted, the original data of the multi-source service is obtained, the original data includes: structured data, semi-structured data and unstructured data; the original data is processed for knowledge extraction, and a data set is obtained, the data set includes: a structured sub-data set, an entity text sub-data set and a visual text sub-data set; the data set is mapped to a unified vector space by a pre-trained large model, and a to-be-aligned entity is obtained, and the to-be-aligned entity and a target field dictionary are processed for entity alignment, and a target candidate entity set of the to-be-aligned entity is obtained; the target candidate entity set and the to-be-aligned entity are processed for entity disambiguation, and an entity knowledge set is obtained; according to the entity knowledge set, the entity relationship of the entity in the entity knowledge set is processed for relationship alignment by a large model, and a knowledge triple after relationship alignment is output, and a knowledge graph corresponding to the original data is constructed based on the knowledge triple. Wherein, for multi-modal and multi-source heterogeneous original data, knowledge extraction is performed to obtain a data set, a pre-trained large model is used to map the data set to a unified vector space to obtain a to-be-aligned entity, a target field dictionary is used to standardize and disambiguate the to-be-aligned entity to obtain an entity knowledge set; then the entity relationship of each entity is aligned according to the entity knowledge set to obtain a knowledge triple, and a knowledge graph is constructed according to the knowledge triple, which can perform entity alignment and disambiguation on multi-source heterogeneous data sets, and perform relationship alignment on entity relationships, fully utilize the original data, automatically construct a knowledge graph, without manual data screening and database import operations, and can solve the problems of low data utilization rate and low accuracy of the knowledge graph.

[0196] It should be noted that the execution entity of the knowledge graph construction method based on a large model provided in this application embodiment can be a knowledge graph construction device based on a large model, or a control module in the knowledge graph construction device based on a large model for executing the knowledge graph construction method based on a large model. This application embodiment uses the execution of the knowledge graph construction method based on a large model by the knowledge graph construction device based on a large model as an example to illustrate the knowledge graph construction device based on a large model provided in this application embodiment.

[0197] Figure 5 This is a schematic diagram of a knowledge graph construction device based on a large model according to an embodiment of the present invention. Figure 5 As shown, the knowledge graph construction device based on a large model includes: an acquisition module 51, an extraction module 52, a mapping module 53, a disambiguation module 54, and an output module 55. The acquisition module 51 is used to acquire raw data from multi-source business, including structured data, semi-structured data and unstructured data. Extraction module 52 is used to perform knowledge extraction processing on the original data to obtain a dataset, which includes: a structured sub-dataset, an entity text sub-dataset, and a visual text sub-dataset; The mapping module 53 is used to map the dataset to a unified vector space through a pre-trained large model to obtain entities to be aligned, and to perform entity alignment processing on the entities to be aligned and the target domain dictionary to obtain the target candidate entity set of the entities to be aligned. The disambiguation module 54 is used to perform entity disambiguation processing on the target candidate entity set and the entities to be aligned to obtain the entity knowledge set; Output module 55 is used to perform relation alignment processing on the entity relations of entities in the entity knowledge set through a large model, output the relation-aligned knowledge triples, and construct the knowledge graph corresponding to the original data based on the knowledge triples.

[0198] In one embodiment, the extraction module 52 includes: The first extraction unit is used to extract triples from the structured and semi-structured data in the original data and generate a structured subset. The triples are used to represent entities and the entity relationships between entities. The conversion unit is used to convert speech dialogue data in unstructured data into text speech data through a speech recognition system; The second extraction unit is used to input text speech data and unstructured text data into the large model, and the large model determines the entity text subset. The large model is used to extract triples from the unstructured text data. The analysis unit is configured to analyze the visual branch data in the unstructured data by using a large model to obtain first visual text data, and the large model is further configured to extract triples of the visual branch data in the unstructured data. The recognition unit is configured to recognize the visual branch data in the unstructured data by using an optical character recognition technology to obtain second visual text data, and generate a visual text sub-data set according to the first visual text data, the second visual text data, and the triples of the visual branch data. The determination unit is configured to determine the structured sub-data set, the entity text sub-data set, and the visual text sub-data set as the data set.

[0199] In one embodiment, the mapping module 53 is configured to calculate cosine similarity of entities to be aligned and entities in a target domain dictionary, determine a preset number of entities in the target domain dictionary according to the cosine similarity, determine a first candidate entity set based on the preset number of entities, obtain entities in the entities to be aligned that are not matched with the entities in the target domain dictionary as entities to be matched, and perform entity alignment processing on the entities to be matched through a contrast learning function to obtain a second candidate entity set, and determine the first candidate entity set and the second candidate entity set as a target candidate entity set.

[0200] In one embodiment, the output module 55 is configured to determine entities in the entity knowledge set and entity relationships of the entities in the entity knowledge set according to the entity knowledge set, determine a candidate similar relationship corresponding to the entity relationship of the entities in the entity knowledge set through an attribute feature similarity function in the large model, calculate a confidence of the candidate similar relationship according to an attention mechanism of the large model, determine a target entity relationship of the entities in the entity knowledge set through the confidence, and output a knowledge triple corresponding to the original data after the relationship alignment processing according to the target entity relationship and entities corresponding to the target entity relationship.

[0201] In one embodiment, the device further includes a detection module configured to perform data monitoring processing on the original data, obtain a new entity and a new entity relationship corresponding to the new entity when a new entity is monitored, and generate a new knowledge fragment, perform semantic association matching processing on the entities in the new knowledge fragment and the entities in the knowledge graph to obtain unmatched entities that are not matched between the entities in the new knowledge fragment and the entities in the knowledge graph, calculate a similarity value of the unmatched entities and the entities in the knowledge graph to obtain a target entity, and update the knowledge graph dynamically, wherein the target entity includes an entity in the knowledge graph with the highest similarity value.

[0202] In one embodiment, the apparatus further comprises a detection module for obtaining a sample data set and a sample knowledge graph, inputting the sample data set into a large model to be trained, the large model to be trained comprising: an open source basic model corresponding to an open source multi-modal large model, wherein the open source multi-modal large model serves as a teacher model, and the open source basic model serves as a student model, to generate labeled data and probability distribution labels based on the teacher model to guide the fine-tuning of the student model; mapping the sample data set to a unified vector space through the large model to be trained to obtain sample entities to be aligned corresponding to the sample data set; inputting the sample entities to be aligned and the probability distribution labels into the large model to be trained, and outputting sample knowledge triples through a fusion formula of an embedding layer of the large model to be trained; training the large model to be trained through the sample knowledge triples, the sample knowledge graph, and a joint loss function to obtain a trained large model.

[0203] By adopting the technical solutions of the embodiments of the present application, the original data of the multi-source service is obtained, the original data including: structured data, semi-structured data, and unstructured data; the original data is subjected to knowledge extraction processing to obtain a data set, the data set including: a structured sub-data set, an entity text sub-data set, and a visual text sub-data set; the data set is mapped to a unified vector space through a pre-trained large model to obtain entities to be aligned, and the entities to be aligned and a target domain dictionary are subjected to entity alignment processing to obtain a target candidate entity set of the entities to be aligned; the target candidate entity set and the entities to be aligned are subjected to entity disambiguation processing to obtain an entity knowledge set; according to the entity knowledge set, the entity relationships of the entities in the entity knowledge set are subjected to relationship alignment processing through the large model, knowledge triples after the relationship alignment processing are output, and a knowledge graph corresponding to the original data is constructed based on the knowledge triples. In this way, for the multi-modal and multi-source heterogeneous original data, the knowledge extraction is performed to obtain the data set, the data set is mapped to the unified vector space through the pre-trained large model to obtain the entities to be aligned, the standardization processing and the disambiguation processing are performed on the entities to be aligned through the target domain dictionary to obtain the entity knowledge set, the entity relationships of each entity are subjected to alignment processing according to the entity knowledge set to obtain the knowledge triples, and the knowledge graph is constructed according to the knowledge triples. This can perform the entity alignment and disambiguation processing on the multi-source heterogeneous data set, perform the relationship alignment processing on the entity relationships, fully utilize the original data, automatically construct the knowledge graph, and does not need manual data screening and database importing operations, thereby being capable of solving the problems of low data utilization rate and low accuracy of the knowledge graph in constructing the knowledge graph.

[0204] The device based on a large model in the embodiments of the present application can be a device, or a component, an integrated circuit, or a chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. Exemplarily, the mobile electronic device can be a mobile phone, a tablet computer, a notebook computer, a palm computer, a vehicle-mounted electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), and the like, and the non-mobile electronic device can be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine, or a self-service machine, and the like, which are not specifically limited in the embodiments of the present application.

[0205] The device based on a large model in the embodiments of the present application can be a device with an operating system. The operating system can be an Android operating system, an ios operating system, or other possible operating systems, which are not specifically limited in the embodiments of the present application.

[0206] The device based on a large model provided in the embodiments of the present application can implement the method embodiments. Figures 1 to 4 The processes implemented in the method embodiments are not repeated here to avoid repetition.

[0207] Based on the same technical concept, the embodiments of the present application further provide an electronic device for executing the knowledge graph construction method based on a large model, Figure 6 A structural schematic diagram of an electronic device for implementing the embodiments of the present application. The electronic device can have great differences due to different configurations or performances, and can include a processor 610, a communications interface 620, a memory 630, and a communications bus 640. The processor 610, the communications interface 620, and the memory 630 can complete mutual communication through the communications bus 640. The processor 610 can invoke a computer program stored on the memory 630 and executable on the processor 610 to perform the following steps: Obtaining original data of a plurality of services, the original data including structured data, semi-structured data, and unstructured data; Performing knowledge extraction processing on the original data to obtain a data set, the data set including a structured sub-data set, an entity text sub-data set, and a visual text sub-data set; The data set is mapped to a unified vector space by a pre-trained large model to obtain the to-be-aligned entity, and entity alignment processing is performed on the to-be-aligned entity and a target field dictionary to obtain a target candidate entity set of the to-be-aligned entity; Entity disambiguation processing is performed on the target candidate entity set and the to-be-aligned entity to obtain an entity knowledge set. According to the entity knowledge set, the entity relationship of the entity in the entity knowledge set is processed by a large model, and a knowledge triple after relationship alignment processing is output, and a knowledge graph corresponding to the original data is constructed based on the knowledge triple.

[0208] The technical scheme of the embodiment of the application is adopted, the original data of the multi-source service is obtained, the original data includes: structured data, semi-structured data and unstructured data; the original data is subjected to knowledge extraction processing to obtain a data set, the data set includes: a structured sub-data set, an entity text sub-data set and a visual text sub-data set; the data set is mapped to a unified vector space by a pre-trained large model to obtain the to-be-aligned entity, and entity alignment processing is performed on the to-be-aligned entity and a target field dictionary to obtain a target candidate entity set of the to-be-aligned entity; entity disambiguation processing is performed on the target candidate entity set and the to-be-aligned entity to obtain an entity knowledge set; according to the entity knowledge set, the entity relationship of the entity in the entity knowledge set is processed by a large model, and a knowledge triple after relationship alignment processing is output, and a knowledge graph corresponding to the original data is constructed based on the knowledge triple. Among them, for the multi-modal and multi-source heterogeneous original data, knowledge extraction is performed to obtain a data set, a pre-trained large model is used to map the data set to a unified vector space to obtain a to-be-aligned entity, and standardization processing and disambiguation processing are performed on the to-be-aligned entity by a target field dictionary to obtain an entity knowledge set; then the entity relationship of each entity is aligned according to the entity knowledge set to obtain a knowledge triple, and a knowledge graph is constructed according to the knowledge triple, which can perform entity alignment and disambiguation processing on the multi-source heterogeneous data set, and perform relationship alignment processing on the entity relationship, fully utilize the original data, automatically construct a knowledge graph, without manual data screening and database import operations, and can solve the problems of low data utilization rate and low accuracy of the knowledge graph.

[0209] The specific execution steps can refer to the steps of the above-mentioned knowledge graph construction method based on a large model, and the same technical effects can be achieved. To avoid repetition, they will not be described here.

[0210] It should be noted that the electronic device in the embodiments of the application includes: a server, a terminal or other devices other than a terminal.

[0211] The above electronic device structure does not constitute a limitation on the electronic device, which can include more or fewer components than those shown, or combine some components, or have different arrangements of components, for example, the input unit can include a Graphics Processing Unit (GPU) and a microphone, and the display unit can be configured with a display panel in the form of a liquid crystal display, an organic light-emitting diode, etc. The user input unit includes at least one of a touch panel and other input devices. The touch panel is also referred to as a touch screen. Other input devices can include, but are not limited to, a physical keyboard, function keys (such as volume control keys, on-off keys, etc.), trackballs, mice, joysticks, and the like, which are not described here.

[0212] The memory can be used to store software programs and various data. The memory can mainly include a first storage area storing programs or instructions and a second storage area storing data, wherein the first storage area can store an operating system, application programs or instructions required by at least one function (such as a sound playing function, an image playing function, etc.), etc. In addition, the memory can include a volatile memory or a non-volatile memory, or the memory can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically EPROM (EEPROM), or a flash memory. The volatile memory can be a Random Access Memory (RAM), a Static RAM (SRAM), a Dynamic RAM (DRAM), a Synchronous DRAM (SDRAM), a Double Data Rate SDRAM (DDR SDRAM), an Enhanced SDRAM (ESDRAM), a Synchlink DRAM (SLDRAM), and a Direct Rambus RAM (DRRAM).

[0213] The processor can include one or more processing units; optionally, the processor integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and an application program, and the modem processor mainly processes a wireless communication signal, such as a baseband processor. It can be understood that the above-mentioned modem processor can also not be integrated into the processor.

[0214] The embodiment of the present application further provides a readable storage medium, and the readable storage medium stores a program or instructions, the program or instructions are executed by a processor to realize each process of the above-mentioned knowledge graph construction method based on a large model, and the same technical effects can be achieved. To avoid repetition, details are not repeated here.

[0215] The processor is the processor in the electronic device in the above-mentioned embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disk, etc.

[0216] The embodiment of the present application further provides a chip, and the chip includes a processor and a communication interface, the communication interface is coupled with the processor, and the processor is used to run a program or instructions to realize each process of the above-mentioned knowledge graph construction method based on a large model, and the same technical effects can be achieved. To avoid repetition, details are not repeated here.

[0217] It should be understood that the chip mentioned in the embodiment of the present application can also be referred to as a system-level chip, a system chip, a chip system, or a system-on-chip chip, etc.

[0218] The embodiment of the present application further provides a computer program product, and the computer program product includes a computer program. When the computer program is executed by a processor, the processor is used to run a program or instructions to realize each process of the above-mentioned product recommendation method embodiment, and the same technical effects can be achieved. To avoid repetition, details are not repeated here.

[0219] It should be noted that, in the present document, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the presence of additional identical elements in the process, method, article, or apparatus that comprises the element. Furthermore, it is to be understood that the method and apparatus of the present application can be carried out by more than one process, method, article, or apparatus either simultaneously, concurrently, or with intervening action that are carried out at the same time, either in a simultaneous fashion or in a fashion that is interleaved in time. For example, the described methods can be performed in a different order from that described, and / or various steps can be combined or omitted, and / or additional steps can be added, without departing from the scope of the present application. Also, features described with respect to certain examples can be combined in other examples.

[0220] From the above description of the embodiments, it is apparent that the above-described method of the embodiments can be realized by means of software and general-purpose hardware platforms, and of course, can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such an understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as a ROM / RAM, a magnetic disk, or an optical disk) and includes a number of instructions for causing a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device) to execute the methods described in the various embodiments of the present application.

[0221] The embodiments of the present application are described above with reference to the accompanying drawings, but the present application is not limited to the above-described specific embodiments, and the above-described specific embodiments are merely illustrative, rather than limiting, and a person of ordinary skill in the art can make many forms under the inspiration of the present application without departing from the scope of the present application and the scope protected by the claims.

Claims

1. A method for constructing a knowledge graph based on a large model, characterized in that, The method includes: Obtain raw data from multiple sources, including structured data, semi-structured data, and unstructured data. The original data is subjected to knowledge extraction processing to obtain a dataset, which includes: a structured sub-dataset, an entity text sub-dataset, and a visual text sub-dataset; The dataset is mapped to a unified vector space by a pre-trained large model to obtain entities to be aligned, and entity alignment processing is performed on the entities to be aligned and the target domain dictionary to obtain the target candidate entity set of the entities to be aligned. Entity disambiguation processing is performed on the target candidate entity set and the entities to be aligned to obtain an entity knowledge set; Based on the entity knowledge set, the entity relationships in the entity knowledge set are aligned using the large model, and knowledge triples with aligned relationships are output. A knowledge graph corresponding to the original data is constructed based on the knowledge triples.

2. The method according to claim 1, characterized in that, The step of performing knowledge extraction processing on the original data to obtain a dataset includes: Extract the triples of the structured data and the semi-structured data from the original data, and generate the structured subset dataset. The triples are used to represent the entities and the entity relationships between the entities. The speech dialogue data in the unstructured data is converted into text speech data using a speech recognition system; The text speech data and the text data in the unstructured data are input into the large model, and the entity text subset is determined by the large model. The large model is used to extract the triples of the text data in the unstructured data. The large model is used to parse and process the visual branch data in the unstructured data to obtain the first visual text data. The large model is also used to extract the triples of the visual branch data in the unstructured data. The second visual text data is obtained by recognizing the visual branch data in the unstructured data using optical character recognition technology. The visual text subset is generated based on the triples in the first visual text data, the second visual text data, and the visual branch data. The structured sub-dataset, the entity text sub-dataset, and the visual text sub-dataset are used as the dataset.

3. The method according to claim 1, characterized in that, The step of performing entity alignment processing on the entities to be aligned and the target domain dictionary to obtain the target candidate entity set of the entities to be aligned includes: Calculate the cosine similarity between the entity to be aligned and the entity in the target domain dictionary; determine a preset number of entities in the target domain dictionary based on the cosine similarity; and determine a first candidate entity set based on the preset number of entities. The entities that do not match the entities in the target domain dictionary are selected from the entities to be aligned and are used as entities to be matched. By comparing the learning function, the entities to be matched are aligned to obtain a second candidate entity set; The first candidate entity set and the second candidate entity set are used as the target candidate entity set.

4. The method according to claim 1, characterized in that, The step of performing relation alignment processing on the entity relations of entities in the entity knowledge set using the large model, based on the entity knowledge set, and outputting relation-aligned knowledge triples includes: Based on the entity knowledge set, determine the entity in the entity knowledge set, and the entity relationship of the entity in the entity knowledge set; By using the attribute feature similarity function in the large model, candidate similarity relations corresponding to the entity relations of the entities in the entity knowledge set are determined; Based on the attention mechanism of the large model, the confidence level of the candidate similarity relationship is calculated, and the target entity relationship of the entity in the entity knowledge set is determined by the confidence level. Based on the target entity relationship and the entity corresponding to the target entity relationship, output the knowledge triple corresponding to the original data after relationship alignment processing.

5. The method according to claim 1, characterized in that, The method further includes: The original data is subjected to data monitoring processing. When a new entity is detected, the new entity and the new entity relationship corresponding to the new entity are obtained, and a new knowledge fragment is generated. Semantic association matching is performed on the entities in the newly added knowledge fragment and the entities in the knowledge graph to obtain unmatched entities that do not match the entities in the newly added knowledge fragment and the entities in the knowledge graph; The similarity value between the unmatched entity and the entity in the knowledge graph is calculated to obtain the target entity, which includes the entity in the knowledge graph with the highest similarity value, and is used to dynamically update the knowledge graph.

6. The method according to claim 1, characterized in that, The large model was trained using the following method: Obtain a sample dataset and a sample knowledge graph, and input the sample dataset into the large model to be trained. The large model to be trained includes: an open-source base model corresponding to the open-source multimodal large model, wherein the open-source multimodal large model serves as the teacher model and the open-source base model serves as the student model, so as to generate labeled data and probability distribution labels based on the teacher model to guide the fine-tuning of the student model. The large model to be trained maps the sample dataset to a unified vector space to obtain the sample entities to be aligned corresponding to the sample dataset. The sample entities to be aligned and the probability distribution labels are input into the large model to be trained, and the sample knowledge triples are output through the fusion formula of the embedding layer of the large model to be trained. The large model to be trained is obtained by using the sample knowledge triples, the sample knowledge graph, and the joint loss function.

7. A knowledge graph construction device based on large model technology, characterized in that, include: The acquisition module is used to acquire raw data from multi-source services, including structured data, semi-structured data, and unstructured data. The extraction module is used to perform knowledge extraction processing on the original data to obtain a dataset, which includes: a structured sub-dataset, an entity text sub-dataset, and a visual text sub-dataset; The mapping module is used to map the dataset to a unified vector space through a pre-trained large model to obtain entities to be aligned, and to perform entity alignment processing on the entities to be aligned and the target domain dictionary to obtain the target candidate entity set of the entities to be aligned. The disambiguation module is used to perform entity disambiguation processing on the target candidate entity set and the entity to be aligned to obtain an entity knowledge set. The output module is used to perform relation alignment processing on the entity relations of the entities in the entity knowledge set through the large model, output the relation-aligned knowledge triples, and construct the knowledge graph corresponding to the original data based on the knowledge triples.

8. An electronic device, characterized in that, The system includes a processor and a memory electrically connected to the processor, the memory storing a computer program, and the processor being used to call and execute the computer program from the memory to implement a knowledge graph construction method based on large model technology as described in claims 1-6.

9. A computer-readable storage medium, characterized in that, The storage medium is used to store a computer program that can be executed by a processor to implement a knowledge graph construction method based on large model technology as described in claims 1-6.

10. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the knowledge graph construction method based on large model technology as described in claims 1-6.

Images