Knowledge graph construction method and device, equipment and storage medium

Abstract

The present disclosure provides a knowledge graph construction method and device, equipment and a storage medium, relates to the technical field of image processing, and in particular to the technical field of artificial intelligence, large models, knowledge extraction, knowledge graph construction and the like. The specific implementation scheme is: obtaining a target detection result of a to-be-processed text and a target image, the to-be-processed text being generated based on the target image and used to describe image content of the target image; based on the target detection result, screening information accurately describing the target image from the to-be-processed text to obtain a target text; extracting knowledge information from the target text; and constructing a knowledge graph based on the knowledge information. In the embodiment of the present disclosure, knowledge is extracted from an image, and a knowledge graph is constructed, thereby providing a reference. In the entire implementation process, the image does not need to be labeled, the labeling cost can be saved, and the method can be flexibly applied to various images.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to the fields of artificial intelligence, large models, knowledge extraction, and knowledge graph construction. Background Technology

[0002] Knowledge is the culmination of human understanding of the objective world through practice. Knowledge graphs (KG) are an important branch of knowledge engineering, which structurally describe concepts and their relationships in the physical world in symbolic form. Structured knowledge graphs have wide applications and can be used to conduct various business activities. Summary of the Invention

[0003] This disclosure provides a method, apparatus, device, and storage medium for constructing a knowledge graph.

[0004] According to one aspect of this disclosure, a method for constructing knowledge graphs is provided, including:

[0005] Obtain the target detection results of the text to be processed and the target image. The text to be processed is generated based on the target image and is used to describe the image content of the target image.

[0006] Based on the target detection results, information that accurately describes the target image is filtered out from the text to be processed to obtain the target text;

[0007] Extract knowledge information from the target text;

[0008] Knowledge graphs are constructed based on knowledge information.

[0009] According to another aspect of this disclosure, a knowledge graph construction apparatus is provided, comprising:

[0010] The acquisition module is used to acquire the target detection results of the text to be processed and the target image. The text to be processed is generated based on the target image and is used to describe the image content of the target image.

[0011] The processing module is used to filter out information that accurately describes the target image from the text to be processed based on the target detection results, and obtain the target text.

[0012] The extraction module is used to extract knowledge information from the target text;

[0013] The building module is used to construct knowledge graphs based on knowledge information.

[0014] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0015] At least one processor; and

[0016] The memory is communicatively connected to the at least one processor; wherein,

[0017] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the methods of any embodiment of the present disclosure.

[0018] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform a method according to any embodiment of this disclosure.

[0019] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements a method according to any embodiment of this disclosure.

[0020] This embodiment of the disclosure provides a reference for extracting knowledge from images and constructing a knowledge graph. Furthermore, the entire implementation process eliminates the need for image annotation, saving annotation costs and allowing for flexible application to various image types.

[0021] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0022] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0023] Figure 1 This is a flowchart illustrating a knowledge graph construction method according to an embodiment of the present disclosure;

[0024] Figure 2 This is a schematic diagram of the second prompting information according to another embodiment of this disclosure;

[0025] Figure 3 This is a schematic diagram of a large model according to another embodiment of the present disclosure;

[0026] Figure 4 This is a schematic diagram based on an image from another embodiment of this disclosure;

[0027] Figure 5 This is a schematic diagram of the structure of a knowledge graph construction apparatus according to another embodiment of the present disclosure;

[0028] Figure 6 This is a block diagram of an electronic device used to implement the knowledge graph construction method of the embodiments of this disclosure. Detailed Implementation

[0029] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0030] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this disclosure, "multiple" means two or more, unless otherwise explicitly specified.

[0031] Most related technologies can process text to extract knowledge for constructing knowledge graphs. However, while images contain less text, they can still convey a great deal of information. Therefore, how to extract knowledge from images to construct knowledge graphs is a problem worthy of attention.

[0032] In view of this, this disclosure proposes a knowledge graph construction method. This method first generates text based on an image. To minimize discrepancies between the generated text and the content expressed by the original image, this disclosure proposes a strategy to further optimize the generated text to obtain text that accurately describes the image content. Subsequently, knowledge can be extracted from the optimized text to achieve the goal of constructing a knowledge graph based on the image.

[0033] like Figure 1 The diagram shown is a flowchart illustrating a knowledge graph construction method provided in this embodiment of the present disclosure, including the following steps:

[0034] S101, Obtain the target detection results of the text to be processed and the target image. The text to be processed is generated based on the target image and is used to describe the image content of the target image.

[0035] In this embodiment of the disclosure, the process of generating text to be processed based on the target image achieves image-to-text conversion. However, hallucinations and secondary information may occur during this process, meaning that the converted text to be processed may contain incorrect or secondary descriptions of the image. Therefore, in order to identify character content generated by hallucinations from the text to be processed, this embodiment of the disclosure performs target detection on the target image to optimize the extracted text content by leveraging mature target detection results.

[0036] S102, Based on the target detection results, information that accurately describes the target image is filtered out from the text to be processed to obtain the target text.

[0037] S103, Extract knowledge information from the target text.

[0038] S104, Constructing a knowledge graph based on knowledge information.

[0039] In this embodiment of the disclosure, considering the possibility of image-generated text being misleading and thus failing to accurately describe the image content, a method is proposed to use object detection results to filter out effective descriptions of the target image from the text to be processed. This reduces the problem of misleading image descriptions in the text to be processed, improving the image-to-text conversion effect. By optimizing the target text obtained from the text to be processed, it can more accurately express the content to be conveyed by the target image. Then, the knowledge extracted based on the target text, using the image as a benchmark, can more accurately describe the knowledge expressed by the image. This embodiment of the disclosure provides a reference for extracting knowledge from images and constructing knowledge graphs. Moreover, the entire implementation process does not require image annotation, which can save annotation costs and can be flexibly applied to various types of images.

[0040] In this embodiment of the disclosure, the image-to-text conversion can be achieved based on the reasoning capabilities of a large model. Specifically, obtaining the text to be processed can be implemented as follows:

[0041] Step A1: Construct the first prompt information based on the target image.

[0042] In this embodiment, the first prompt information may include a task description, prompt content, and information output format requirements. The task description specifies that text needs to be generated based on the image, and the generated text must describe the information content conveyed by the image. To avoid a significant discrepancy between the model-generated text and the expected content, prompt content can be introduced to constrain the model's output text. For example, the prompt content may include a prompt word, which requires the model's output text to contain the prompt word or content related to the prompt word. In practice, the prompt word in the first prompt information may include one or more, and this embodiment does not limit this. The information output format requirements can be set according to actual needs. For example, it may require the output to include a triple, i.e., a subject, a predicate, and a description of their relationship. The information output format may also be output as a quintuple, i.e., a subject, a subject type, an object, an object type, and the relationship between the subject and the object. Of course, the information output format requirements may also require the output content to include categories, locations, places, and events. For example, a description of a group photo might be: "There are eight people in a classroom having a lesson."

[0043] Step A2: Input the first prompt information into the multimodal model so that the multimodal model can complete the picture description task and obtain the text to be processed from the target image.

[0044] In other words, this embodiment of the disclosure can employ a multimodal large model to implement the image description task, converting at least a portion of the information to be conveyed by the target image into text content. The input to the multimodal large model can include the target image or prompts related to the target image. In practice, if the target image contains characters, OCR (Optical Character Recognition) technology can be used to extract text content from the target image to construct the first prompt information. For characters recognized by OCR, important information such as subject, time, location, and event information can be extracted. For example, this important information can be triple or quintuple knowledge extracted from the OCR-recognized characters. The extracted important information is used as a prompt example to guide the multimodal model in completing the image description task.

[0045] In this embodiment, the reasoning capabilities of a large model are utilized to convert the content to be conveyed by the target image into text. This allows for the extraction of knowledge from the text using natural language processing to construct a knowledge graph. This completes the closed-loop operation from image to knowledge graph construction.

[0046] In this embodiment of the disclosure, the target detection result includes information about at least one target object identified from the target image and a confidence level for at least one target object. The target object information may include at least one of the following: target object category, target object location, target object expression, emotion, appearance, etc. The target detection result is not 100% reliable; therefore, the target detection result carries a confidence level for each target object, i.e., the probability of detecting the corresponding target object (which can be understood as credibility), to provide information for subsequent decision-making.

[0047] Correspondingly, the text to be processed also includes information about at least one candidate object. The specific expression of the candidate object's information is the same as or similar to the target object's information; that is, the candidate object's information may include at least one of the following: the candidate object's category, the candidate object's location, the candidate object's expression, emotion, appearance, etc.

[0048] Therefore, regarding the hallucination problem, based on the target detection results, information accurately describing the target image is filtered from the text to be processed to obtain the target text. Specifically, this can be implemented as follows:

[0049] Step B1 involves matching the information of candidate objects in the text to be processed with the information of at least one target object to obtain the information of the candidate objects that match the target objects, which serves as intermediate information.

[0050] In implementation, the information of the candidate object and the information of the target object can be matched according to the text information matching method. In one possible implementation, the information of the candidate object can be used as the first string, and the information of the target object can be used as the second string. Then, the similarity between the first string and the second string is calculated. If the similarity between the two is higher than the similarity threshold, the two are determined to be a match.

[0051] In another possible implementation, the target object's information includes multiple first elements, and the candidate object's information includes multiple second elements. After extracting the multiple first elements and multiple second elements, the multiple second elements are matched with the multiple first elements. The number of second elements that match the first elements is counted. Then, the ratio of the number of second elements that match the first elements to the total number of second elements in the candidate object's information is calculated. If this ratio is greater than a preset ratio, it is determined that the candidate object's information and the target object's information are matched. Otherwise, it is confirmed that they are not matched. In practice, which specific information from the target object's information and the candidate object's information is used as an element can be determined according to actual business needs, and this disclosure does not limit this.

[0052] After the processing in step B1, candidate objects not included in the target detection results will inevitably fail to match the corresponding target object and will be filtered out. Information about candidate objects included in the target detection results will be retained as intermediate information. Then, to further mitigate expression errors caused by the illusion problem, in step B2 of this disclosure example, information about candidate objects with confidence levels higher than a preset value is selected from the intermediate information to obtain the target text.

[0053] In other words, as described above, object detection results are not always 100% accurate and errors can occur. Therefore, the confidence score of each object in the object detection results can be used to measure the probability that the target image contains information about that object. Thus, using confidence scores can help optimize the effectiveness of image captioning.

[0054] Therefore, in this embodiment of the disclosure, by matching candidate objects and target objects, erroneous representations of the target image in the text to be processed can be filtered out, while retaining the information contained in the target detection results. Furthermore, by using confidence levels, secondary information is filtered out from the text to be processed, ensuring that the remaining target text contains the important information conveyed by the target image. Thus, by constraining and optimizing the representation of the text to be processed based on the target detection results, the final target text accurately describes the important content of the target image, improving the accuracy of knowledge extraction from the target image.

[0055] Accordingly, in order to accurately construct a knowledge graph, the following two methods can be provided to extract knowledge information from the target text.

[0056] 1. Knowledge extraction based on large language models

[0057] To ensure the generalization ability of knowledge graph construction for vertical domains, this embodiment employs a large model to extract knowledge from the target text. Specifically, it can be implemented as follows:

[0058] Step C1: Construct a second prompt message based on the target text.

[0059] In some possible implementations, the second prompt includes a task description that instructs the large language model to extract knowledge from a given target problem to obtain knowledge information. The second prompt may also include few-shot examples to facilitate the large language model's information extraction task based on these examples. Furthermore, to ensure the large model's output meets required structural requirements, the second prompt may also include output format requirements.

[0060] In this embodiment of the disclosure, a small amount of text within a vertical domain can be annotated to obtain an annotated dataset. Each text in this dataset has corresponding pre-annotated knowledge information. Therefore, few-shots can be constructed based on this dataset. Specifically, a second prompt message is constructed based on the target text, including:

[0061] Step C11: Select at least one reference text from the data set that has a similarity higher than the preset similarity with the target text, and the reference text has reference knowledge that is used to annotate the reference text.

[0062] In practice, the similarity between the target text and each candidate text in the dataset can be calculated. This can be done by calculating the similarity between the word embeddings of the target and candidate texts. Alternatively, a text feature extractor can be used to extract the first text vector of the target text, and then calculate the similarity with the second text vectors of each candidate text in the dataset.

[0063] Candidate texts with a similarity higher than a preset similarity can be selected as reference texts for the target text. During implementation, as many reference texts as possible should be selected so that the large language model can fully understand and learn from the few shots, thereby accurately extracting knowledge from the target text.

[0064] Step C12: Construct an example based on at least one reference text and the reference knowledge of at least one reference text.

[0065] Step C13: Construct a second prompt message based on the example, task description, output format requirements, and target text.

[0066] like Figure 2The image shown is a schematic diagram of the second prompt information provided in an embodiment of this disclosure. Figure 2 The first instruction contains task description information: Please extract information from the following sentence, with entity categories including ["alarm name", ...] and relationship categories including ["alarm relationship", ...], and output in JSON format. The first instruction also includes specific formatting requirements for the JSON format.

[0067] exist Figure 2 In the few-shots section, there are examples of the build, including example numbers such as "Example 1", reference text for the examples such as the content contained in the "Input" section, and annotation results of the reference text such as the content contained in the "Output" section.

[0068] exist Figure 2 The second instruction includes the target text to be processed. For example, "Input: 'The cause of the RDC unit unavailability alarm may be due to an unnecessary connection exception. Common handling methods include…'". The second instruction also includes the required output format.

[0069] In this embodiment, suitable reference text can be obtained based on the similarity between texts to construct the second prompt information, enabling the large language model to learn the knowledge from the reference text and extract knowledge information from the target text. The task description in the second prompt information allows the large language model to correctly understand the task to be completed and perform knowledge extraction based on prior knowledge and the reasoning ability of the large model, thereby improving the accuracy of knowledge extraction. The output format requirements in the second prompt information can constrain the output of the large model to meet the processing needs of downstream tasks.

[0070] Step C2: Input the second prompt information into the large language model so that the large language model can extract knowledge information from the target text.

[0071] In this embodiment of the disclosure, the second prompt information can guide the large language model to extract knowledge. The large language model has good learning and reasoning capabilities, can accurately extract knowledge information based on the second prompt information, and can generalize knowledge from different vertical domains.

[0072] In some embodiments, the text content generated by the picture description may be quite long. However, the total character length supported by the second cue information of the large language model is limited. Therefore, this disclosure also includes:

[0073] Step D1: If the length of the target text is greater than the preset length, the target text is divided into multiple subtexts; wherein, for each subtext, in the second prompt information of the subtext, the total length of the subtext, at least one example, task description and output format requirements is not greater than the upper limit of the total length allowed by the first prompt information.

[0074] One possible implementation is to statistically analyze the length of each candidate text in a dataset within the same vertical domain, obtaining a statistical distribution of the candidate text lengths in the dataset. This could include, for example, the mean length, minimum length, and maximum length. Therefore, when segmenting the target text, the target text can be split based on this statistical distribution of lengths.

[0075] In other possible implementations, the target text can also be segmented based on semantic coherence. The semantic coherence of each subtext can be evaluated after segmentation; if semantically coherent, the segmentation is successful. Otherwise, the segmentation is repeated until multiple semantically coherent subtexts are obtained. For example, for a single subtext, characters can be added or removed one by one, and then the semantic coherence is reassessed until a semantically coherent subtext is obtained.

[0076] In other possible implementations, the target text can also be segmented based on a user-defined length.

[0077] It should be noted that the present disclosure does not limit the method of segmentation.

[0078] The construction of second prompt information based on the target text can be implemented as step D2, which involves constructing corresponding second sub-prompt information for each sub-text; wherein, the second prompt information includes the second sub-prompt information of each sub-text, and the second sub-prompt information of each sub-text is used to input the large language model one by one;

[0079] Specifically, for each subtext, in the second sub-prompt information of the subtext, the total length of the subtext, at least one example of the subtext, the task description of the subtext, and the output format requirements of the subtext does not exceed the maximum allowed total length.

[0080] For each subtext, a corresponding second sub-hint is constructed so that the second sub-hint of each subtext can be input into the large language model to obtain the sub-knowledge information extracted by the large language model from each subtext, and finally obtain the knowledge information of the target text.

[0081] In this embodiment of the disclosure, for long texts containing multiple pieces of knowledge information, the long text can be segmented into different sub-texts. Then, based on the segmented sub-texts, the contained knowledge information can be accurately extracted.

[0082] The preceding text explained the use of large language models for knowledge extraction, highlighting their strong generalization and reasoning capabilities. During the pre-training phase, the large language model can be trained using labeled text from different domains to ensure it can accumulate knowledge across those domains. Alternatively, labeled text from different scenarios within the same domain can be used to train the model, enabling it to generalize across different scenarios within the same domain. Furthermore, large language models possess excellent learning capabilities, requiring only a small amount of labeled text and allowing them to learn from a limited set of annotations.

[0083] Since large language models have high hardware requirements, in this embodiment of the disclosure, smaller models can also be distilled from large language models to complete the knowledge extraction task.

[0084] 2. Knowledge extraction is performed using a distilled knowledge extraction model.

[0085] In this embodiment of the disclosure, the knowledge extraction model is obtained based on the distillation of a large language model, and the training samples of the knowledge extraction model are constructed based on the knowledge extracted by the large language model. Therefore, extracting knowledge information from the target text can also be implemented by: inputting the target text into the knowledge extraction model, so that the knowledge extraction model extracts knowledge information from the target text.

[0086] The knowledge extraction model in this embodiment is significantly smaller than that of a large language model, making deployment much easier. To address the needs of different vertical domains, a corresponding knowledge extraction model can be distilled for each domain. Furthermore, to address the needs of different scenarios within the same vertical domain, a corresponding knowledge extraction model can be distilled for each scenario. This provides generalization capability for different or the same vertical domain, accurately extracting knowledge information. Multiple knowledge extraction models can also work in parallel, improving the efficiency of knowledge extraction.

[0087] In this embodiment, the training samples for the knowledge extraction model are constructed using the knowledge information extracted from the aforementioned large language model. This allows for high-quality training samples with minimal manual annotation.

[0088] Using a large language model to extract a knowledge quintuple, including node name, node type, and node relationship as an example, the extracted dataset is shown in the following JSON.

[0089]

[0090]

[0091] In this embodiment, the knowledge extraction model is smaller in scale than the large language model, making it easier to deploy and implement. Since the knowledge extraction model is derived from the distillation of the large language model and uses the knowledge extracted from the large language model as training samples, the distilled small model can fully learn the knowledge extraction capabilities of the large language model, thus obtaining accurate knowledge information.

[0092] In some embodiments, a large language model can be optimized through multiple rounds of fine-tuning training to distill a knowledge extraction model. To preserve the prior knowledge of the large language model, each training round may include pre-training the large language model. Furthermore, to enable the large language model to improve its ability to extract information within a specific domain, each training round may also use knowledge extraction task data to optimize the large language model.

[0093] Therefore, during a single training round, pre-training data and knowledge extraction task data are used to optimize the large language model. Fine-tuning is employed during training, such as using the LoRA (Low-Rank Adaptation) method. Both pre-training data and knowledge extraction tasks can be fine-tuned to optimize the large language model.

[0094] From the perspective of the entire training process, in the single-round training phase, the large language model used for knowledge extraction is fine-tuned using pre-training data and knowledge extraction task data, thereby achieving alternating optimization of the large language model. Then, the knowledge extraction model is extracted based on the optimized large language model.

[0095] The pre-training data can be any sample, and the pre-training task can be the task used when the large language model was pre-trained. This allows the large language model to retain the experiential knowledge it has already learned.

[0096] When fine-tuning based on knowledge extraction tasks, knowledge extraction tasks in vertical domains are used to optimize the large language model.

[0097] In the distillation stage, a large language model is used to extract knowledge from the training samples of the knowledge extraction task data, resulting in labeled training samples to complete the distillation of the knowledge extraction model. The training samples extracted using the large language model can optimize the knowledge extraction model in a supervised manner, improving the accuracy of information extraction.

[0098] Furthermore, since there are differences between the training tasks of the pre-training data and the knowledge extraction task, in order to better optimize the large language model to distill a high-quality knowledge extraction model, in this embodiment of the disclosure, the ratio between the first sample size of the pre-training data and the second sample size of the knowledge extraction task data is determined based on the accuracy of knowledge extraction and the correctness of the output knowledge format.

[0099] For example, when optimizing a large language model, after completing one round of training, determine the accuracy of knowledge extraction and the correctness of the output knowledge format of the large language model; based on the accuracy and correctness, determine the ratio between the first and second sample sizes used in the pre-training in this round of training; after multiple rounds of training, select the optimal ratio; and continue to optimize the large language model based on the optimal ratio.

[0100] For example, after the first round of training, accuracy (m1) and accuracy (n1) both improve. After the second round, both accuracy (m2) and accuracy (n2) continue to increase. After the third, fourth, and qth rounds of training, both accuracy and accuracy begin to decline. Therefore, the optimal ratio can be chosen based on the ratio between accuracy and accuracy at the qth round of training. In short, the optimal ratio is the one that maximizes both accuracy and accuracy. Alternatively, the optimal ratio is the one where accuracy exceeds an accuracy threshold and accuracy exceeds an accuracy threshold.

[0101] In this embodiment, the large language model is optimized by alternately using pre-training data and knowledge extraction task data. This allows the knowledge extraction model to retain the pre-trained knowledge of the large model and learn new knowledge from the corresponding vertical domain through fine-tuning. Furthermore, training and optimizing the large language model using the optimal ratio accelerates training and improves the knowledge extraction accuracy of the distilled knowledge extraction model.

[0102] Furthermore, in this embodiment, a span-based UIE (span-based Unified Structure Generation for Universal Information Extraction) can be selected to build the knowledge extraction model. UIE is suitable for extracting general information, possesses excellent few-sample fine-tuning capabilities, and can quickly adapt to specific extraction targets. Experiments demonstrate that UIE achieves state-of-the-art (SOTA) performance on four IE tasks, 13 datasets, and a wide range of entity, relation, event, and sentiment extraction tasks, along with unified supervision, low-resource, and few-sample settings.

[0103] Regardless of whether a large language model is used for knowledge extraction or a smaller knowledge extraction model is used, in order to prevent the extracted knowledge information from being unrelated to the schema or not generated in accordance with the output format requirements, the output process of the model is filtered and controlled in this embodiment of the disclosure.

[0104] When the model outputs text word by word, the output results can be controlled by adding filtering rules through engineering. Therefore, extracting knowledge information from the target text also includes the structural requirements based on knowledge graphs and the output format requirements of the knowledge information.

[0105] In this embodiment of the disclosure, the structural requirement of the knowledge graph is a predefined knowledge graph schema. The output format requirements, for example... Figure 2 The JSON format requirements.

[0106] In this embodiment of the disclosure, knowledge information is extracted by referring to the structural requirements of knowledge graphs and the output format requirements of knowledge information, which is beneficial to obtaining effective knowledge information within the vertical domain, so as to facilitate the construction of knowledge graphs.

[0107] In some embodiments, in order to meet the structural requirements of a knowledge graph, extracting knowledge information based on the structural requirements of the knowledge graph may include the following aspects:

[0108] 1) For each first character to be predicted in the node name in the knowledge information, select the first candidate character with the highest probability of being contained in the target text from multiple first candidate characters of the first character to be predicted as the first character to be predicted;

[0109] 2) For each second character to be predicted in the node type of the knowledge information, select the second candidate character with the highest probability of being contained in the preset node type of the knowledge graph from multiple second candidate characters of the second character to be predicted.

[0110] 3) For each third character to be predicted in the node relationship in the knowledge information, select the third candidate character with the highest probability of being included in the preset node relationship of the knowledge graph from multiple third candidate characters of the third character to be predicted.

[0111] For example, Figure 3 As shown, the predicted "RRU link failure" by the subject name is correct. However, when predicting the "alarm name," the first and second characters "high alert" are correctly predicted by the subject type `subject_type`. When predicting the third character, the character with the highest probability is "category" with a value of 0.52. However, "high alert category" does not belong to the preset node type. Therefore, the character "name" with a probability of 0.39 is chosen, resulting in the "alarm name," which perfectly matches the preset node type. Thus, node type generation based on schema constraints is achieved.

[0112] The prediction of other node names and alarm relationships follows the same constraint method as the prediction of node types, and will not be elaborated here.

[0113] Therefore, by controlling the output of the model character by character, the final predicted knowledge information can meet the requirements of the schema, so as to construct a knowledge graph that meets the schema requirements.

[0114] To make it easier to understand, the following example is provided. For example... Figure 4 As shown, the image is input into the large model. The group photo is taken in a classroom and mainly depicts two students tidying up their desks, one student in the aisle, and another student wiping the blackboard. After inputting the image into the multimodal large model and completing the picture description task, the resulting text is: The image shows a classroom with midday sunlight; five students (positions of which are [x1, y1, x2, y2], ...) and a classroom; two students tidying up their desks, one student in the aisle, one student wiping the blackboard, two trees and a vase outside the window, and one student standing under a tree...

[0115] To filter out secondary and hallucinatory content, object detection was performed on the image, and the results showed that there were five students, a classroom, and two trees.

[0116] Therefore, after filtering the text to be processed based on the target detection results, the target text is obtained: There is a classroom, the midday sun, there are five students and a classroom in the picture, two students are tidying up the desks, one student is in the aisle, one student is wiping the blackboard, there are two trees outside the window, and one student is standing under a tree.

[0117] Then, a large language model or knowledge extraction model is used to extract knowledge information from the target text according to the schema requirements and the output JSON format requirements. Finally, a knowledge graph is constructed based on the extracted knowledge information.

[0118] Based on the same technical concept, this disclosure provides a knowledge graph construction apparatus 500, such as... Figure 5 As shown, the device includes:

[0119] The acquisition module 501 is used to acquire the target detection results of the text to be processed and the target image. The text to be processed is generated based on the target image and is used to describe the image content of the target image.

[0120] Processing module 502 is used to filter out information that accurately describes the target image from the text to be processed based on the target detection results, and obtain the target text;

[0121] Extraction module 503 is used to extract knowledge information from the target text;

[0122] Module 504 is used to build knowledge graphs based on knowledge information.

[0123] In some embodiments, the acquisition module is configured to:

[0124] Based on the target image, construct the first prompt information;

[0125] The first prompt information is input into the multimodal model so that the multimodal model can complete the picture description task and obtain the text to be processed from the target image.

[0126] In some embodiments, the target detection result includes information about at least one target object identified from the target image and the confidence level of at least one target object; the text to be processed includes information about at least one candidate object; the processing module is configured to:

[0127] The information of candidate objects in the text to be processed is matched with the information of at least one target object to obtain the information of the candidate objects that match the target objects, which is used as intermediate information.

[0128] From the intermediate information, information on candidate objects with a confidence level higher than a preset value is selected to obtain the target text.

[0129] In some embodiments, the extraction module includes:

[0130] Construction unit, used to construct a second prompt message based on the target text;

[0131] The extraction unit is used to input the second prompt information into the large language model so that the large language model can extract knowledge information from the target text.

[0132] In some embodiments, the building unit is used for:

[0133] Select at least one reference text from the data set that has a similarity higher than a preset similarity with the target text, and the reference text has reference knowledge that is used to annotate the reference text;

[0134] An example is constructed based on at least one reference text and the reference knowledge of at least one reference text;

[0135] The second prompt message is constructed based on the example, task description, output format requirements, and target text.

[0136] In some embodiments, when the length of the target text is greater than a preset length, the target text is segmented into multiple sub-texts; the construction unit is further configured to:

[0137] For each subtext, a corresponding second sub-cue information is constructed; wherein, the second cue information includes the second sub-cue information of each subtext, and the second sub-cue information of each subtext is used to input the large language model one by one;

[0138] Specifically, for each subtext, in the second sub-prompt information of the subtext, the total length of the subtext, at least one example of the subtext, the task description of the subtext, and the output format requirements of the subtext does not exceed the maximum allowed total length.

[0139] In some embodiments, the extraction module is configured to:

[0140] The target text is input into the knowledge extraction model so that the knowledge extraction model can extract knowledge information from the target text;

[0141] The knowledge extraction model is derived from the distillation of the large language model, and the training samples of the knowledge extraction model are constructed based on the knowledge extracted from the large language model.

[0142] In some embodiments,

[0143] The training process of distilling a knowledge extraction model based on a large language model includes a single-round training phase, in which the large language model used for knowledge extraction is fine-tuned using pre-training data and knowledge extraction task data.

[0144] The ratio between the first sample size of the pre-training data and the second sample size of the knowledge extraction task data is determined based on the accuracy of knowledge extraction and the correctness of the output knowledge format.

[0145] In some embodiments, the extraction module is configured to:

[0146] Based on the structural requirements of knowledge graphs and the output format requirements of knowledge information, knowledge information is extracted from target text.

[0147] In some embodiments, the extraction module is configured to:

[0148] For each first character to be predicted in the node name of the knowledge information, select the first candidate character with the highest probability of being contained in the target text from multiple first candidate characters of the first character to be predicted; and,

[0149] For each second character to be predicted in the node types of the knowledge information, from multiple second candidate characters of the second character to be predicted, select the second candidate character with the highest probability that it is contained in the preset node types of the knowledge graph as the second character to be predicted; and,

[0150] For each third character to be predicted in the node relationships in the knowledge information, select the third candidate character with the highest probability of being included in the preset node relationships of the knowledge graph from multiple third candidate characters of the third character to be predicted.

[0151] The specific functions and examples of each module and submodule of the apparatus in this disclosure can be found in the relevant descriptions of the corresponding steps in the above method embodiments, and will not be repeated here.

[0152] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0153] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0154] like Figure 6 As shown, device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 602 or a computer program loaded from storage unit 608 into random access memory (RAM) 603. RAM 603 may also store various programs and data required for the operation of device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0155] Multiple components in device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of monitors, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0156] The computing unit 601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as knowledge graph construction methods. For example, in some embodiments, the knowledge graph construction method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the knowledge graph construction method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform knowledge graph construction methods by any other suitable means (e.g., by means of firmware).

[0157] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0158] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0159] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0160] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0161] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0162] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0163] According to embodiments of this disclosure, the electronic device can be integrated with the communication component, display screen, and information acquisition device, or it can be separately configured with the communication component, display screen, and information acquisition device.

[0164] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0165] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A knowledge graph construction method, comprising: The process involves obtaining target detection results for both the text to be processed and the target image. The text to be processed is obtained by using a multimodal large model to perform an image description task, converting at least a portion of the information to be conveyed by the target image into text content to describe the image content of the target image. The target detection results include information about at least one target object identified from the target image and the confidence level of the at least one target object. The text to be processed includes information about at least one candidate object. Based on the target detection results, information accurately describing the target image is filtered out from the text to be processed to obtain target text, including: matching the information of candidate objects in the text to be processed with the information of at least one target object to obtain the information of candidate objects that match the target objects, as intermediate information; filtering out the information of candidate objects with a confidence level higher than a preset value from the intermediate information to obtain the target text; Extract knowledge information from the target text; A knowledge graph is constructed based on the aforementioned knowledge information.

2. The method according to claim 1, wherein, The process of obtaining the text to be processed includes: Based on the target image, a first prompt message is constructed; The first prompt information is input into the multimodal model so that the multimodal model can complete the image description task and obtain the text to be processed from the target image.

3. The method according to claim 1, wherein, The extraction of knowledge information from the target text includes: A second prompt message is constructed based on the target text; The second prompt information is input into the large language model so that the large language model can extract the knowledge information from the target text.

4. The method according to claim 3, wherein, The construction of the second prompt information based on the target text includes: At least one reference text with a similarity higher than a preset similarity to the target text is selected from the data set, and the reference text has reference knowledge that is annotated for the reference text; An example is constructed based on the at least one reference text and the reference knowledge of the at least one reference text; The second prompt message is constructed based on the example, task description, output format requirements, and target text.

5. The method according to claim 4, further comprising: If the length of the target text is greater than a preset length, the target text is divided into multiple sub-texts; The construction of the second prompt information based on the target text includes: For each subtext, a corresponding second sub-hint is constructed; wherein, the second hint includes the second sub-hint of each subtext, and the second sub-hint of each subtext is used to input the large language model one by one; Specifically, for each subtext, in the second sub-prompt information of the subtext, the total length of the subtext, at least one example of the subtext, the task description of the subtext, and the output format requirements of the subtext does not exceed the maximum allowed total length.

6. The method according to claim 1, wherein, The extraction of knowledge information from the target text includes: The target text is input into the knowledge extraction model so that the knowledge extraction model can extract the knowledge information from the target text; The knowledge extraction model is obtained based on the distillation of a large language model, and the training samples of the knowledge extraction model are constructed based on the knowledge extracted by the large language model.

7. The method according to claim 6, wherein, The training process of distilling the knowledge extraction model based on the large language model includes a single-round training phase, in which the large language model used for knowledge extraction is fine-tuned using pre-training data and knowledge extraction task data. The ratio between the first sample size of the pre-training data and the second sample size of the knowledge extraction task data is determined based on the accuracy of knowledge extraction and the correctness of the output knowledge format.

8. The method according to any one of claims 1-7, wherein, The extraction of knowledge information from the target text includes: Based on the structural requirements of the knowledge graph and the output format requirements of the knowledge information, the knowledge information is extracted from the target text.

9. The method according to claim 8, wherein, Extracting the knowledge information based on the structural requirements of the knowledge graph includes: For each first character to be predicted in the node name of the knowledge information, from a plurality of first candidate characters of the first character to be predicted, the first candidate character with the highest probability of being contained in the target text is selected as the first character to be predicted; and, For each second character to be predicted in the node types of the knowledge information, from a plurality of second candidate characters of the second character to be predicted, the second candidate character with the highest probability of being contained in a preset node type of the knowledge graph is selected as the second character to be predicted; and, For each third character to be predicted in the node relationships of the knowledge information, the third candidate character with the highest probability of being included in the preset node relationships of the knowledge graph is selected from multiple third candidate characters of the third character to be predicted as the third character to be predicted.

10. A knowledge graph construction apparatus, comprising: The acquisition module is used to acquire the target detection results of the text to be processed and the target image. The text to be processed is obtained by using a multimodal large model to implement the image description task, converting at least part of the information to be conveyed by the target image into text content, which is used to describe the image content of the target image. The target detection results include information on at least one target object identified from the target image and the confidence level of the at least one target object. The text to be processed includes information on at least one candidate object. The processing module is used to filter out information that accurately describes the target image from the text to be processed based on the target detection result, obtain the target text, and perform a matching operation between the information of the candidate objects in the text to be processed and the information of the at least one target object to obtain the information of the candidate objects that match the target objects, as intermediate information. From the intermediate information, information on candidate objects with a confidence level higher than a preset value is selected to obtain the target text; The extraction module is used to extract knowledge information from the target text; A construction module is used to construct a knowledge graph based on the knowledge information.

11. The apparatus according to claim 10, wherein, The acquisition module is used for: Based on the target image, a first prompt message is constructed; The first prompt information is input into the multimodal model so that the multimodal model can complete the image description task and obtain the text to be processed from the target image.

12. The apparatus according to claim 10, wherein, The extraction module includes: A construction unit is used to construct a second prompt message based on the target text; An extraction unit is used to input the second prompt information into a large language model so that the large language model can extract the knowledge information from the target text.

13. The apparatus according to claim 12, wherein, The building unit is used for: At least one reference text with a similarity higher than a preset similarity to the target text is selected from the data set, and the reference text has reference knowledge that is annotated for the reference text; An example is constructed based on the at least one reference text and the reference knowledge of the at least one reference text; The second prompt message is constructed based on the example, task description, output format requirements, and target text.

14. The apparatus according to claim 13, wherein when the length of the target text is greater than a preset length, the target text is segmented into multiple sub-texts; the construction unit is further configured to: For each subtext, a corresponding second sub-hint message is constructed; where, The second prompt information includes the second sub-prompt information of each sub-text, and the second sub-prompt information of each sub-text is used to input the large language model one by one; Specifically, for each subtext, in the second sub-prompt information of the subtext, the total length of the subtext, at least one example of the subtext, the task description of the subtext, and the output format requirements of the subtext does not exceed the maximum allowed total length.

15. The apparatus according to claim 10, wherein, The extraction module is used for: The target text is input into the knowledge extraction model so that the knowledge extraction model can extract the knowledge information from the target text; The knowledge extraction model is obtained based on the distillation of a large language model, and the training samples of the knowledge extraction model are constructed based on the knowledge extracted by the large language model.

16. The apparatus according to claim 15, wherein, The training process of distilling the knowledge extraction model based on the large language model includes a single-round training phase, in which the large language model used for knowledge extraction is fine-tuned using pre-training data and knowledge extraction task data. The ratio between the first sample size of the pre-training data and the second sample size of the knowledge extraction task data is determined based on the accuracy of knowledge extraction and the correctness of the output knowledge format.

17. The apparatus according to any one of claims 10-16, wherein, The extraction module is used for: Based on the structural requirements of the knowledge graph and the output format requirements of the knowledge information, the knowledge information is extracted from the target text.

18. The apparatus according to claim 17, wherein, The extraction module is used for: For each first character to be predicted in the node name in the knowledge information, the first candidate character with the highest probability of being contained in the target text is selected from a plurality of first candidate characters of the first character to be predicted as the first character to be predicted. as well as, For each second character to be predicted in the node type of the knowledge information, the second candidate character with the highest probability of being included in the preset node type of the knowledge graph is selected from multiple second candidate characters of the second character to be predicted as the second character to be predicted. as well as, For each third character to be predicted in the node relationships of the knowledge information, the third candidate character with the highest probability of being included in the preset node relationships of the knowledge graph is selected from multiple third candidate characters of the third character to be predicted as the third character to be predicted.

19. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-9.

20. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-9.

21. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-9.

Images