Knowledge graph construction method and device and storage medium
By processing the original image and text data separately and then fusing image triples and text triples, the problem of decreased accuracy in knowledge graphs in existing technologies is solved, and higher construction accuracy is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHINA UNITED NETWORK COMM GRP CO LTD
- Filing Date
- 2026-01-13
- Publication Date
- 2026-05-15
AI Technical Summary
Existing knowledge graph construction methods suffer from low accuracy due to multiple rounds of entity judgment.
By processing the original image data and text data separately, image triples and text triples are identified and fused together. A knowledge graph is constructed using a pre-trained model, reducing the context accumulation effect and the homogenization problem of single modality.
It improves the accuracy of knowledge graph construction, reduces homogeneous results, and enhances the accuracy of knowledge graphs.
Smart Images
Figure CN122047438A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of knowledge graph technology, and in particular to a knowledge graph construction method, apparatus and storage medium. Background Technology
[0002] With the rapid development of artificial intelligence and big data technologies, knowledge graphs, as structured semantic networks, are widely used in various industries. However, the construction process of knowledge graphs requires close attention. If the method used to construct the knowledge graph is unreasonable, an inaccurate knowledge graph may be created, leading to business failures and ultimately financial losses.
[0003] Currently, knowledge graph construction methods in related technologies mainly involve inputting raw data multiple times into a large language model, which then performs multi-round entity judgments to build the knowledge graph. However, the context accumulation effect during these multi-round entity judgments leads to homogenization issues, resulting in decreased accuracy of the constructed knowledge graph. Therefore, these knowledge graph construction methods suffer from low accuracy. Summary of the Invention
[0004] This application provides a knowledge graph construction method, apparatus, and storage medium, which can improve the accuracy of knowledge graph construction.
[0005] To achieve the above objectives, this application adopts the following technical solution: In a first aspect, this application provides a knowledge graph construction method, which includes: acquiring original image data and original text data; determining image triples based on the original image data and a language recognition model; the image triples are used to indicate multiple entities in the original image data and the relationships between the multiple entities, and the language recognition model is used to determine the image triples; determining text triples based on the original text data and a large language model; the text triples are used to indicate multiple entities in the original text data and the relationships between the multiple entities, and the large language model is used to determine the text triples; and constructing a target knowledge graph by fusing the image triples and text triples based on a pre-trained model.
[0006] The above technical solution brings at least the following beneficial effects: This application processes the original image data and original text data separately to determine image triples and text triples, and then fuses the image triples and text triples. By independently extracting image triples and text triples, the homogenization problem caused by context accumulation effect is reduced. Constructing a knowledge graph using multimodal image triples and text triples reduces the homogenization results caused by data bias or ambiguity in a single modality, thereby constructing a more accurate knowledge graph. Therefore, this application can improve the accuracy of knowledge graph construction.
[0007] In one possible implementation, determining image triples based on original image data and a speech recognition model includes: inputting the original image data into a three-level speech recognition model to determine the descriptive information of the original image data; the three-level speech recognition model consists of three concatenated speech recognition models; extracting text information from the original image data based on an optical character recognition model; inputting the text information, descriptive information, original image data, and prompt information into the speech recognition model to determine the image triples; and using the prompt information to instruct the speech recognition model to determine the image triples.
[0008] In one possible implementation, the original image data is input into a three-level language recognition model to determine the descriptive information of the original image data, including: inputting the original image data into a first language recognition model in the three-level language recognition model to determine first descriptive information; inputting the first descriptive information and the original image data into a second language recognition model in the three-level language recognition model to determine second descriptive information; and inputting the second descriptive information and the original image data into a third language recognition model in the three-level language recognition model to determine the descriptive information of the original image data.
[0009] In one possible implementation, a target knowledge graph is constructed by fusing image triples and text triples based on a pre-trained model. This includes: inputting image triples into the pre-trained model to determine the embedding vector of each entity in the image triples; inputting text triples into the pre-trained model to determine the embedding vector of each entity in the text triples; determining the cosine similarity between the embedding vectors of each entity in the image triples and the embedding vectors of each entity in the text triples; connecting entities with a cosine similarity greater than a first threshold; fusing the image triples and text triples; and constructing the target knowledge graph.
[0010] In one possible implementation, the method further includes: obtaining preliminary descriptive information determined by a three-level language recognition model; the preliminary descriptive information includes multiple descriptive statements and the model confidence of each descriptive statement; inputting the descriptive information and the original image data into a long text contrastive language-image pre-trained model to determine the descriptive similarity of each descriptive statement; the descriptive similarity is used to indicate the similarity between each descriptive statement and the original image data; based on the model confidence of each descriptive statement and the descriptive similarity of each descriptive statement, determining a comprehensive score for each descriptive statement; and identifying the descriptive statements among the multiple descriptive statements whose comprehensive score is greater than a second threshold as the descriptive information of the original image data.
[0011] In one possible implementation, the composite score satisfies the following formula:
[0012] in, For the overall score, For model confidence, To describe similarity.
[0013] Secondly, this application provides a knowledge graph construction apparatus, which includes: an acquisition unit and a determination unit; the acquisition unit is used to acquire original image data and original text data; the determination unit is used to determine image triples based on the original image data and a language recognition model; the image triples are used to indicate multiple entities in the original image data and the relationships between the multiple entities, and the language recognition model is used to determine the image triples; the determination unit is also used to determine text triples based on the original text data and a large language model; the text triples are used to indicate multiple entities in the original text data and the relationships between the multiple entities, and the large language model is used to determine the text triples; the determination unit is also used to construct a target knowledge graph by fusing the image triples and text triples based on a pre-trained model.
[0014] In one possible implementation, the determining unit is specifically used for: inputting the original image data into a three-level speech recognition model to determine the descriptive information of the original image data; the three-level speech recognition model consists of three concatenated speech recognition models; extracting text information from the original image data based on an optical character recognition model; inputting the text information, descriptive information, original image data, and prompt information into the speech recognition model to determine image triples; and using the prompt information to instruct the speech recognition model to determine image triples.
[0015] In one possible implementation, the determining unit is specifically used for: inputting the original image data into the first language recognition model in the three-level language recognition model to determine the first descriptive information; inputting the first descriptive information and the original image data into the second language recognition model in the three-level language recognition model to determine the second descriptive information; and inputting the second descriptive information and the original image data into the third language recognition model in the three-level language recognition model to determine the descriptive information of the original image data.
[0016] In one possible implementation, the unit is specifically used for: inputting image triples into a pre-trained model to determine the embedding vector of each entity in the image triples; inputting text triples into a pre-trained model to determine the embedding vector of each entity in the text triples; determining the cosine similarity between the embedding vectors of each entity in the image triples and the embedding vectors of each entity in the text triples; connecting entities with a cosine similarity greater than a first threshold, fusing the image triples and the text triples, and constructing a target knowledge graph.
[0017] In one possible implementation, the acquisition unit is further configured to acquire preliminary descriptive information determined by the three-level language recognition model; the preliminary descriptive information includes multiple descriptive statements and the model confidence of each descriptive statement among the multiple descriptive statements; the determination unit is further configured to input the descriptive information and the original image data into the long text contrastive language-image pre-trained model to determine the descriptive similarity of each descriptive statement; the descriptive similarity is used to indicate the similarity between each descriptive statement and the original image data; the determination unit is further configured to determine the comprehensive score of each descriptive statement based on the model confidence of each descriptive statement and the descriptive similarity of each descriptive statement; the determination unit is further configured to determine the descriptive statements among the multiple descriptive statements whose comprehensive score is greater than a second threshold as the descriptive information of the original image data.
[0018] In one possible implementation, the composite score satisfies the following formula:
[0019] in, For the overall score, For model confidence, To describe similarity.
[0020] Thirdly, this application provides a knowledge graph construction apparatus, which includes: a processor and a communication interface; the communication interface and the processor are coupled, and the processor is used to run computer programs or instructions to implement the knowledge graph construction method as described in the first aspect and any possible implementation of the first aspect.
[0021] Fourthly, this application provides a computer-readable storage medium storing instructions that, when executed on a terminal, cause the terminal to perform the knowledge graph construction method as described in the first aspect and any possible implementation thereof.
[0022] Fifthly, this application provides a computer program product containing instructions that, when run on a knowledge graph construction apparatus, cause the knowledge graph construction apparatus to perform the knowledge graph construction method as described in the first aspect and any possible implementation thereof.
[0023] In a sixth aspect, this application provides a chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run computer programs or instructions to implement the knowledge graph construction method as described in the first aspect and any possible implementation thereof.
[0024] Specifically, the chip provided in this application also includes a memory for storing computer programs or instructions. Attached Figure Description
[0025] Figure 1 A schematic diagram of the composition of a knowledge graph construction device provided in an embodiment of this application; Figure 2 A flowchart illustrating a knowledge graph construction method provided in this application embodiment; Figure 3 This is a schematic diagram of raw image data provided in an embodiment of this application; Figure 4 A schematic diagram illustrating descriptive information provided in an embodiment of this application; Figure 5 This is a schematic diagram of an image triple provided in an embodiment of this application; Figure 6 A schematic diagram of a target knowledge graph provided in an embodiment of this application; Figure 7 This is a schematic diagram of a knowledge graph construction device provided in an embodiment of this application. Detailed Implementation
[0026] The knowledge graph construction method, apparatus, and storage medium provided in the embodiments of this application will be described in detail below with reference to the accompanying drawings.
[0027] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.
[0028] The terms "first" and "second," etc., used in the specification and drawings of this application are used to distinguish different objects or to distinguish different treatments of the same object, rather than to describe a specific order of objects.
[0029] Furthermore, the terms "comprising" and "having," and any variations thereof, used in the description of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the steps or units listed, but may optionally include other steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus.
[0030] It should be noted that in the embodiments of this application, the words "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of the words "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.
[0031] In the description of this application, unless otherwise stated, "a plurality of" means two or more.
[0032] With the rapid development of artificial intelligence and big data technologies, knowledge graphs, as structured semantic networks, are widely used in various industries. However, the construction process of knowledge graphs requires close attention. If the method used to construct the knowledge graph is unreasonable, an inaccurate knowledge graph may be created, leading to business failures and ultimately financial losses.
[0033] Currently, research on multimodal knowledge graph construction exhibits characteristics of cross-domain, multimodal fusion, and intelligent processing. Regarding data sources, research typically acquires open-source text knowledge graphs, target text data, and multimodal information such as images and videos from multiple disciplines or specific fields (e.g., education, military, healthcare, and cybersecurity), and ensures data quality through preprocessing steps such as data cleaning, annotation, and association matching.
[0034] In the semantic extraction stage, deep learning models are commonly used. For example, bidirectional encoder representations from transformers (BERT) or word embeddings from language models (ELMo) are used to encode text features. Residual networks (ResNet) or optimized real-time object detection algorithms (You Only Look Once, YOLO) are used to extract visual features, and graph neural networks are used to capture structural features. These methods comprehensively capture the semantic information of entities and relationships, thus enabling the description of image information. In addition, some methods fully utilize vision-language models (VLMs) to describe image information and then extract corresponding triples.
[0035] In the entity alignment stage, large language models (LLMs) are introduced in fields such as cybersecurity for multi-round alignment. Traditional pre-alignment algorithms are often used to narrow down the candidate pool, followed by multi-dimensional similarity analysis (such as node names, structured information, and external knowledge) and threshold determination based on LLM to improve alignment accuracy and interpretability. Furthermore, these methods typically integrate pre-trained language models or neural network models (such as decoders based on encoders and transformers) for dynamic interaction and reasoning, supporting applications such as knowledge retrieval, intelligent question answering, and relation completion. This enables efficient knowledge discovery and decision support in scenarios such as education, combat command, and medical diagnosis, demonstrating the advanced nature and adaptability of multimodal knowledge graphs in enhancing semantic understanding and improving analytical efficiency.
[0036] Currently, knowledge graph construction methods in related technologies mainly involve inputting raw data multiple times into a large language model, which then performs multi-round entity judgments to build the knowledge graph. However, the context accumulation effect during these multi-round entity judgments leads to homogenization issues, resulting in decreased accuracy of the constructed knowledge graph. Therefore, these knowledge graph construction methods suffer from low accuracy.
[0037] Therefore, this application processes the original image data and original text data separately to determine image triples and text triples. By fusing these image triples and extracting them independently, the homogenization problem caused by context accumulation effects is reduced. Constructing a knowledge graph using multimodal image and text triples reduces homogenization results caused by data bias or ambiguity in single-modal applications, thus constructing a more accurate knowledge graph. Therefore, this application can improve the accuracy of knowledge graph construction.
[0038] The application scenarios of the embodiments in this application are not limited. The system architecture and business scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.
[0039] Figure 1 This is a schematic diagram of the composition of a knowledge graph construction device 10 provided in an embodiment of this application, as shown below. Figure 1 As shown, the knowledge graph construction device 10 may include a processor 101, a bus 102, a communication interface 103, and a memory 104.
[0040] The processor 101, memory 104 and communication interface 103 can be connected via bus 102.
[0041] The processor 101 can be a central processing unit (CPU), a general-purpose processor, a network processor (NP), a digital signal processor (DSP), a microprocessor, a microcontroller, a programmable logic device (PLD), or any combination thereof. The processor 101 can also be other devices with processing capabilities, such as circuits, devices, or software modules, without limitation.
[0042] Bus 102 is used to transmit information between the components included in the knowledge graph construction device 10.
[0043] Communication interface 103 is used to communicate with other devices or other communication networks. These other communication networks can be Ethernet, radio access network (RAN), wireless local area networks (WLAN), etc. Communication interface 103 can be a module, circuit, communication interface, or any device capable of enabling communication.
[0044] Memory 104 is used to store instructions. These instructions can be computer programs.
[0045] The memory 104 can be a read-only memory (ROM) or other type of static storage device that can store static information and / or instructions; it can also be a random access memory (RAM) or other type of dynamic storage device that can store information and / or instructions; it can also be an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, etc., without limitation.
[0046] It should be noted that the memory 104 can exist independently of the processor 101 or can be integrated with the processor 101. The memory 104 can be used to store instructions, program code, or some data, etc. The memory 104 can be located inside or outside the knowledge graph construction device 10, without limitation.
[0047] In one example, processor 101 may include one or more CPUs.
[0048] As an optional implementation, the knowledge graph construction device 10 includes multiple processors.
[0049] As an optional implementation, the knowledge graph construction apparatus 10 may also include output devices and input devices. For example, input devices are devices such as keyboards, mice, microphones, or joysticks, and output devices are devices such as displays and speakers.
[0050] It should be noted that the knowledge graph construction device 10 can be a desktop computer, laptop computer, web server, mobile phone, tablet computer, wireless terminal, embedded device, or chip system. Furthermore, the structural composition shown in the figure does not constitute a representation of the device. Figure 1 The limitations of each device in the process, except Figure 1 In addition to the components shown, Figure 1 The various devices may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.
[0051] In this embodiment of the application, the chip system may be composed of chips or may include chips and other discrete devices.
[0052] Furthermore, the actions, terms, etc., involved in the various embodiments of this application can be referenced interchangeably without limitation. The message names or parameter names in the messages exchanged between the various devices in the embodiments of this application are merely examples, and other names may be used in specific implementations without limitation.
[0053] In order to solve the problems existing in the above-mentioned prior art, such as Figure 2 The diagram shown is a flowchart of a knowledge graph construction method provided in an embodiment of this application. This knowledge graph construction method can improve the accuracy of knowledge graph construction. The method includes: S201. Obtain the original image data and original text data.
[0054] The raw image data is an image file that is directly received by the knowledge graph construction device and has not been processed in any way. It can be a binary representation of a pixel array and is used as the raw input for knowledge graph construction.
[0055] For example, the raw image data can be screenshots and images uploaded by users, image data from dedicated imaging devices, or publicly available image datasets.
[0056] The raw text data is an initial collection of text in natural language form that has not undergone structured processing or has only undergone basic format conversion. Essentially, it is a human-readable sequence of symbols used as raw input for knowledge graph construction.
[0057] For example, the raw text data can be speech-to-text transcripts of customer service and user communications, social media conversation records, or business work order records.
[0058] In the process of constructing a knowledge graph, the acquisition methods for raw image and text data can be flexibly selected according to the actual scenario, supporting both passive reception and active acquisition. The core objective of both methods is to ensure the originality, integrity, and timeliness of the data, providing reliable input for subsequent processing.
[0059] When users or testers need to construct a knowledge graph from specific images and text, they can input the images and text into the knowledge graph construction device, which will then receive the images and text and construct the knowledge graph.
[0060] In one possible implementation, S201 above can be implemented as follows: in response to receiving raw image data and raw text data from the user, the acquisition of raw image data and raw text data is completed.
[0061] When users or testers need to construct a knowledge graph for processes or business operations that may contain images and text, they can send a knowledge graph construction instruction to the knowledge graph construction device. The corresponding knowledge graph construction device receives the knowledge graph construction instruction and collects raw image data and raw text data according to the relevant business operations or processes described in the knowledge graph construction instruction.
[0062] In another possible implementation, S201 above can be implemented as follows: in response to receiving a knowledge graph construction instruction from the user, collecting the original image data and original text data indicated in the knowledge graph construction instruction.
[0063] For example, when a user needs to build a knowledge graph for a subscriber identity module (SIM) card, the knowledge graph construction instruction could be "Please build the knowledge graph for card A". The knowledge graph construction device has a built-in data storage module used to deploy the database. The knowledge graph construction device retrieves raw image data and raw text data from the database based on the knowledge graph construction instruction. Figure 3 The image shown is a schematic diagram of raw image data provided in an embodiment of this application. The image depicts a commercial promotional poster for a SIM card. The corresponding raw text data could be feedback data from customers using the SIM card or work order records for the SIM card.
[0064] It should be noted that the embodiments of this application do not limit the method of obtaining the original image data and original text data. In actual application, it can be set according to needs to cover different application scenarios. For example, it can be to receive the original text data from user input through an interface, or to download the original image data after crawling the webpage, or to obtain the original image data and original text data from a database.
[0065] S202. Based on the original image data and the language recognition model, determine the image triples.
[0066] Image triples are triples extracted from raw image data. Triples are the smallest semantic units in a knowledge graph, used to represent semantic relationships between two entities or between an entity and an attribute.
[0067] For example, the triple can be represented as "subject-predicate-object". In the case that the original image data contains a red apple, the triple determined by the red apple can be "apple-color-red".
[0068] It should be noted that the embodiments of this application do not limit the representation of triples. In actual applications, the representation can be set according to requirements to cover different application scenarios. For example, the representation of a triple can be (subject, predicate, object).
[0069] Knowledge graphs are composed of entities and relationships between them, and triples are the smallest semantic units in a knowledge graph. Therefore, after acquiring the original image data, it is necessary to extract triples from it. Specifically, an artificial intelligence model can be used to determine the descriptive information of the original image data, and then an AI model with the ability to extract entities and relationships between them can be used to extract image triples from the descriptive information.
[0070] In one possible implementation, S202 above can be implemented as follows: The original image data is input into a three-level speech recognition model to determine the descriptive information of the original image data. The three-level speech recognition model consists of three cascaded speech recognition models. Based on an optical character recognition model, text information is extracted from the original image data. The text information, descriptive information, original image data, and prompt information are input into the speech recognition model to determine image triplets; the prompt information is used to instruct the speech recognition model to determine the image triplets.
[0071] In one possible implementation, inputting the original image data into a three-level language recognition model to determine the descriptive information of the original image data can be achieved as follows: The original image data is input into the first language recognition model within the three-level language recognition model to determine first descriptive information. The first descriptive information and the original image data are then input into the second language recognition model within the three-level language recognition model to determine second descriptive information. Finally, the second descriptive information and the original image data are input into the third language recognition model within the three-level language recognition model to determine the descriptive information of the original image data. The descriptive information of the original image data is used to describe the content, attributes, source, and relevant contextual information of the original image data using natural language.
[0072] The first and second language recognition models are used to extract as much information as possible about entities, attributes, actions, and contextual relationships from the original image data. The third language recognition model is used to proofread the descriptive information and output a structured description with a uniform format. Therefore, the first and second language recognition models are used to adjust the smoothness of the model output probability distribution, affecting the creativity, diversity, and determinism of the generated results. Their temperature values are greater than those of the third language recognition model. For example, the temperature T1 value of the first language recognition model is 0.8, the temperature T2 value of the second language recognition model is 0.8, and the temperature T3 value of the third language recognition model is 0.2.
[0073] For example, such as Figure 4 The diagram shown is a schematic representation of descriptive information provided in an embodiment of this application. Figure 3 Taking the original image data shown as an example, the first language recognition model VLM1 determines the first descriptive information as... The second language recognition model VLM2 determines the second descriptive information as... The third language recognition model VLM3 determines the third descriptive information as... The third descriptive information is... The format can be defined as "sentence <|> the sentence's self-confidence score". The third descriptive information is... The image is described as: "This image is a commercial promotional poster for 'A Card,' with red as the main color. It presents a clear hierarchical structure through information modules of different colors and shapes. Model confidence (0-10): 9," where model confidence is the confidence level of the third language recognition model in each descriptive statement in the third descriptive information.
[0074] In one possible implementation, to ensure the correctness of the descriptive information format of the original image data, the third descriptive information can be input into a small LLM of a large AI model distillation to further standardize the output format.
[0075] By employing a first-language recognition model and a second-language recognition model with a relatively large temperature parameter, more information such as entities, attributes, actions, and contextual relationships can be extracted from the original image data, ensuring the creativity of the output first and second-level descriptive information. A third-language recognition model with a smaller temperature parameter can output relatively accurate third-level descriptive information. This three-level language recognition model, to a certain extent, balances the creativity and accuracy of the generated content, improving the reliability of the descriptive information in the original image data.
[0076] To ensure the accuracy of the descriptive information in the original image data, it is necessary to calibrate the descriptive information to remove errors from the output of the three-level speech recognition model. Specifically, this can be achieved by quantifying the accuracy of each descriptive statement in the original image data and removing those with lower accuracy values, thus determining the most accurate descriptive information for the original image data.
[0077] In one possible implementation, the process of calibrating the descriptive information of the original image data can be achieved as follows: First, preliminary descriptive information determined by a three-level language recognition model is obtained. This preliminary descriptive information includes multiple descriptive statements and the model confidence score for each statement. The descriptive information and the original image data are then input into a long-text contrastive language-image pre-training (Long_CLP) model to determine the descriptive similarity of each statement; this similarity indicates the similarity between each statement and the original image data. Based on the model confidence score and the descriptive similarity of each statement, a comprehensive score is determined for each statement. Finally, descriptive statements with a comprehensive score greater than a second threshold are identified as the descriptive information of the original image data.
[0078] For example, by Figure 4 It can be seen that the preliminary descriptive information determined by the three-level language recognition model, i.e., the third descriptive information. The model confidence score for each descriptive statement is 9. By inputting the descriptive information and original image data into the Long_CLP model, the descriptive similarity for the statement "presents a clear hierarchical structure" is determined to be 0.05. Multiplying the descriptive similarity by the model confidence score, the overall score for this descriptive statement is determined to be 0.45. With a second threshold set at 0.5, the overall score for this descriptive statement is less than the second threshold. Therefore, this descriptive statement is removed. After removing descriptive statements with overall scores less than the second threshold, the following is determined: Figure 4 The description information shown is as follows.
[0079] In one possible implementation, the composite score satisfies the following formula 1: Formula 1.
[0080] in, For the overall score, For model confidence, To describe similarity.
[0081] By calculating the overall score of each descriptive statement and removing those with scores below a preset value, descriptions irrelevant to the original image data can be filtered out. This provides an accurate data foundation for subsequent extraction of image triples.
[0082] In one possible implementation, based on an optical character recognition model, extracting text information from the original image data can be achieved by: extracting the text from the original image data using a tool with optical character recognition capabilities, and determining the text information in the original image.
[0083] For example, using Paddle Optical Character Recognition (PaddleOCR) to... Figure 3 The original image data shown is processed to determine text information such as product name, pricing information, and product restrictions. .
[0084] For example, inputting text information, descriptive information, original image data, and prompt information into a language recognition model to determine image triples can be achieved by: inputting text information... Description information of the calibrated original image data and original image data Input the data into the VLM model, and instruct the VLM model to extract data through prompts. Figure 5 The image shown contains triplets. The prompt could be: "Please extract all possible triplets based on the input information." Figure 5 The image shown is a schematic diagram of an image triplet provided in an embodiment of this application.
[0085] It should be noted that this application does not limit the format or content of the prompt message. In practical applications, it can be set according to requirements to cover different application scenarios. For example, when there are requirements for the output format, a statement requiring the VLM model to output in a specific format can be added to the prompt message. To ensure the accuracy of the output statement format, the image triples output by the VLM model can be input into a small LLM to standardize the output format.
[0086] S203. Based on the original text data and the large language model, determine the text triples.
[0087] Text triples are triples extracted from the original text data. After obtaining the original text data, it is necessary to extract text triples from it. Specifically, this can be done using a large speech model with the ability to extract entities and relationships between entities, to extract text triples from the descriptive information of the original text data.
[0088] In one possible implementation, S203 above can be implemented as follows: inputting the original text data into a large language model, extracting entities and relationships between entities from the original text data through the large language model, and forming text triples.
[0089] For example, if the original text data is "a red apple", the triple determined by the original text data can be "apple-color-red".
[0090] S204. Based on a pre-trained model, image triples and text triples are fused to construct a target knowledge graph.
[0091] After obtaining image triples and text triples, since these are discrete triples, they need to be fused to construct a target knowledge graph. Specifically, this can be done by calculating the similarity between each entity in the image and text triples and connecting entities with high similarity to generate the target knowledge graph.
[0092] In one possible implementation, S204 above can be implemented as follows: Inputting image triples into a pre-trained model to determine the embedding vector of each entity in the image triples; inputting text triples into the pre-trained model to determine the embedding vector of each entity in the text triples; determining the cosine similarity between the embedding vectors of each entity in the image triples and the embedding vectors of each entity in the text triples; connecting entities with a cosine similarity greater than a first threshold; fusing the image triples and text triples to construct the target knowledge graph.
[0093] For example, text triples are input into a BERT series pre-trained model to determine the embedding vectors of entities in the text triples. Image triples are input into the same pre-trained model to determine the embedding vectors of entities in the image triples. The cosine similarity between the embedding vectors of entities from the image triples and text triples is calculated. If the similarity is higher than a preset threshold, it is determined that the two entities point to the same object in the real world and are linked, such as... Figure 6 The diagram shown is a schematic representation of a target knowledge graph provided in an embodiment of this application.
[0094] To improve the accuracy of determining whether entities belong to the same entity, business rules can be introduced for auxiliary verification. For identifiable attributes such as customer ID, mobile phone model, and package name, entity matching will be prioritized based on these fields, followed by further verification using cosine similarity. For matching conflicts, a conflict resolution strategy based on data source reliability, timestamp freshness, and model confidence is designed to ensure the consistency and reliability of the output results.
[0095] Therefore, this application processes the original image data and original text data separately to determine image triples and text triples. By fusing these image triples and extracting them independently, the homogenization problem caused by context accumulation effects is reduced. Constructing a knowledge graph using multimodal image and text triples reduces homogenization results caused by data bias or ambiguity in single-modal applications, thus constructing a more accurate knowledge graph. Therefore, this application can improve the accuracy of knowledge graph construction.
[0096] It is understood that the above-described knowledge graph construction method can be implemented by a knowledge graph construction device. To achieve the above functions, the knowledge graph construction device includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, based on the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, the embodiments disclosed in this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments disclosed in this application.
[0097] The embodiments disclosed in this application can divide the knowledge graph construction apparatus generated by the above method examples into functional modules. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing module. The integrated modules can be implemented in hardware or as software functional modules. It should be noted that the module division in the embodiments disclosed in this application is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.
[0098] Figure 7 This is a schematic diagram of a knowledge graph construction device provided in an embodiment of the present invention. Figure 7 As shown, the knowledge graph construction device 70 can be used to perform... Figure 2The illustrated knowledge graph construction method includes an acquisition unit 70 and a determination unit 702. The acquisition unit 701 acquires original image data and original text data. The determination unit 702 determines image triples based on the original image data and a language recognition model. Image triples indicate multiple entities and relationships between entities in the original image data, and the language recognition model is used to determine image triples. The determination unit 702 also determines text triples based on the original text data and a large language model. Text triples indicate multiple entities and relationships between entities in the original text data, and the large language model is used to determine text triples. The determination unit 702 further fuses image triples and text triples based on a pre-trained model to construct a target knowledge graph.
[0099] In one possible implementation, the determining unit 702 is specifically used for: inputting the original image data into a three-level language recognition model to determine the descriptive information of the original image data; the three-level language recognition model consists of three concatenated language recognition models; extracting text information from the original image data based on an optical character recognition model; inputting the text information, descriptive information, original image data, and prompt information into the language recognition model to determine image triples; the prompt information is used to instruct the language recognition model to determine image triples.
[0100] In one possible implementation, the determining unit 702 is specifically used to: input the original image data into the first language recognition model in the three-level language recognition model to determine the first description information; input the first description information and the original image data into the second language recognition model in the three-level language recognition model to determine the second description information; and input the second description information and the original image data into the third language recognition model in the three-level language recognition model to determine the description information of the original image data.
[0101] In one possible implementation, the determining unit 702 is specifically used for: inputting image triples into a pre-trained model to determine the embedding vector of each entity in the image triples; inputting text triples into a pre-trained model to determine the embedding vector of each entity in the text triples; determining the cosine similarity between the embedding vectors of each entity in the image triples and the embedding vectors of each entity in the text triples; connecting entities with a cosine similarity greater than a first threshold, fusing the image triples and the text triples, and constructing a target knowledge graph.
[0102] In one possible implementation, the acquisition unit 701 is further configured to acquire preliminary descriptive information determined by the three-level language recognition model; the preliminary descriptive information includes multiple descriptive statements and the model confidence of each descriptive statement among the multiple descriptive statements; the determination unit 702 is further configured to input the descriptive information and the original image data into the long text contrastive language-image pre-trained model to determine the descriptive similarity of each descriptive statement; the descriptive similarity is used to indicate the similarity between each descriptive statement and the original image data; the determination unit 702 is further configured to determine the comprehensive score of each descriptive statement based on the model confidence of each descriptive statement and the descriptive similarity of each descriptive statement; the determination unit 702 is further configured to determine the descriptive statements among the multiple descriptive statements whose comprehensive score is greater than a second threshold as the descriptive information of the original image data.
[0103] In one possible implementation, the composite score satisfies the following formula:
[0104] in, For the overall score, For model confidence, To describe similarity.
[0105] Through the above description of the embodiments, those skilled in the art will clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device, and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.
[0106] This disclosure also provides a computer-readable storage medium storing instructions that, when executed by a processor of an electronic device, enable the electronic device to perform the knowledge graph construction method provided in the embodiments of this disclosure described above.
[0107] This disclosure also provides a computer program product containing instructions that, when run on an electronic device, cause the electronic device to execute the knowledge graph construction method provided in the above-described embodiments of this disclosure.
[0108] The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires; a portable computer disk drive; a hard disk drive; a random access memory (RAM); a read-only memory (ROM); an erasable programmable read-only memory (EPROM); a register; a hard disk drive; an optical fiber; a portable compact disc read-only memory (CD-ROM); an optical storage device; a magnetic storage device; or any suitable combination thereof; or any other form of computer-readable storage medium known in the art. An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium may also be a component of the processor. The processor and the storage medium may reside in an application-specific integrated circuit (ASIC). In the embodiments of this application, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.
[0109] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A method for constructing a knowledge graph, characterized in that, include: Obtain the raw image data and raw text data; Based on the original image data and the language recognition model, image triples are determined; The image triples are used to indicate multiple entities in the original image data and the relationships between the multiple entities, and the language recognition model is used to determine the image triples; Based on the original text data and the large language model, text triples are determined; the text triples are used to indicate multiple entities in the original text data and the relationships between the multiple entities, and the large language model is used to determine the text triples. Based on the pre-trained model, the image triples and the text triples are fused to construct a target knowledge graph.
2. The method according to claim 1, characterized in that, The process of determining image triples based on raw image data and a language recognition model includes: The original image data is input into a three-level language recognition model to determine the descriptive information of the original image data; the three-level language recognition model consists of three cascaded language recognition models. Based on the optical character recognition model, text information is extracted from the original image data; The text information, the description information, the original image data, and the prompt information are input into the language recognition model to determine the image triplet; the prompt information is used to instruct the language recognition model to determine the image triplet.
3. The method according to claim 2, characterized in that, The step of inputting the original image data into a three-level language recognition model to determine the descriptive information of the original image data includes: The original image data is input into the first language recognition model in the three-level language recognition model to determine the first descriptive information; The first description information and the original image data are input into the second language recognition model in the third-level language recognition model to determine the second description information; The second description information and the original image data are input into the third language recognition model in the three-level language recognition model to determine the description information of the original image data.
4. The method according to claim 1, characterized in that, The construction of a target knowledge graph based on a pre-trained model, fusing the image triples and the text triples, includes: The image triples are input into the pre-trained model to determine the embedding vector of each entity in the image triples; the text triples are input into the pre-trained model to determine the embedding vector of each entity in the text triples. Determine the cosine similarity between the embedding vector of each entity in the image triplet and the embedding vector of each entity in the text triplet; Entities with a cosine similarity greater than a first threshold are connected, and the image triples and the text triples are fused to construct the target knowledge graph.
5. The method according to claim 2, characterized in that, The method further includes: Obtain preliminary description information determined by the three-level language recognition model; the preliminary description information includes multiple description statements and the model confidence of each description statement among the multiple description statements; The descriptive information and the original image data are input into a long text contrastive language-image pre-trained model to determine the descriptive similarity of each descriptive statement; the descriptive similarity is used to indicate the similarity between each descriptive statement and the original image data. Based on the model confidence and description similarity of each description statement, a comprehensive score is determined for each description statement. The description statements with a comprehensive score greater than the second threshold among the multiple description statements are determined as the description information of the original image data.
6. The method according to claim 3, characterized in that, The overall score satisfies the following formula: in, For the overall score, For model confidence, To describe similarity.
7. A knowledge graph construction device, characterized in that, The knowledge graph construction device includes: an acquisition unit and a determination unit; The acquisition unit is used to acquire the original image data and the original text data; The determining unit is used to determine image triples based on the original image data and the language recognition model; the image triples are used to indicate multiple entities in the original image data and the relationships between the multiple entities, and the language recognition model is used to determine the image triples; The determining unit is further configured to determine text triples based on the original text data and the large language model; the text triples are used to indicate multiple entities in the original text data and the relationships between the multiple entities, and the large language model is used to determine the text triples; The determining unit is also used to construct a target knowledge graph by fusing the image triples and the text triples based on a pre-trained model.
8. The apparatus according to claim 7, characterized in that, The determining unit is specifically used for: The original image data is input into a three-level language recognition model to determine the descriptive information of the original image data; the three-level language recognition model consists of three cascaded language recognition models. Based on the optical character recognition model, text information is extracted from the original image data; The text information, the description information, the original image data, and the prompt information are input into the language recognition model to determine the image triplet; The prompt information is used to instruct the language recognition model to determine the image triples.
9. A knowledge graph construction device, characterized in that, include: A processor and a communication interface; the communication interface is coupled to the processor, the processor being configured to run computer programs or instructions to implement the method as described in any one of claims 1-6.
10. A computer-readable storage medium storing instructions, characterized in that, When the computer executes the instruction, the computer performs the method described in any one of claims 1-6.