A knowledge graph-based multi-modal data extraction method and system

Abstract

The application relates to the technical field of data processing, and discloses a multi-modal data extraction method and system based on a knowledge graph, which comprises the following steps: judging whether an extraction condition is met based on a source data format, determining a corresponding processing strategy based on a target source data format, determining the association similarity of each data and a target source data feature, determining a first extraction result based on the association similarity, determining non-associated data based on the association similarity, determining the association result between the non-associated data based on a non-associated data and an association rule algorithm, determining a model prediction result of the non-associated data based on a Bayesian model, determining the coincidence relationship of a target extraction result and the first extraction result, reducing the target extraction result based on the coincidence relationship, and determining the summary result as the extraction result of the target source data format. The stability of multi-modal data extraction is ensured by determining the corresponding processing strategy and the association similarity.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and more specifically, to a method and system for multimodal data extraction based on knowledge graphs. Background Technology

[0002] Knowledge graphs, as a structured semantic network, store domain knowledge in the form of triple entity-relation-entity pairs, and have demonstrated significant value in fields such as intelligent question answering, recommendation systems, and decision support. Data extraction involves extracting elements such as entities, relations, and attributes from multi-source heterogeneous data in knowledge graphs and storing them in a structured manner. With the development of internet technology, data formats have gradually evolved from single text to multimodal data including text, images, and videos, placing higher demands on data extraction.

[0003] However, traditional data extraction methods are mainly designed for extracting text data and have difficulty processing unstructured data such as images and videos, resulting in the ineffective use of knowledge graphs. Furthermore, existing data extraction methods lack in-depth analysis of the correlations between multi-source data, leading to the omission of some related data. At the same time, there is a certain amount of redundant data during extraction, resulting in insufficient reliability of the extraction results.

[0004] Therefore, it is necessary to design a multimodal data extraction method and system based on knowledge graphs to solve the problems existing in the current technology. Summary of the Invention

[0005] In view of this, the present invention proposes a multimodal data extraction method and system based on knowledge graphs, which aims to solve the problems of difficulty in processing unstructured data such as images and videos, resulting in the ineffective use of knowledge graphs, lack of in-depth analysis of the correlation between multi-source data, leading to the omission of some related data, and the existence of certain redundant data during extraction, resulting in insufficient reliability of extraction results.

[0006] In one aspect, this invention proposes a multimodal data extraction method based on knowledge graphs, comprising:

[0007] Determine the source data format of the source data, determine whether the extraction conditions are met based on the source data format, and determine the target source data format based on the judgment result;

[0008] Based on the target source data format, a corresponding processing strategy is determined. The processing strategy includes a text strategy and an image strategy. Based on the text strategy or the image strategy, the target source data features are determined. Based on the target source data features, a traversal is performed to determine the association similarity between each data and the target source data features. Based on the association similarity, a first extraction result is determined.

[0009] Based on the association similarity, non-associative data is determined, and based on the non-associative data and the association rule algorithm, the association result between the non-associative data is determined. Based on the Bayesian model, the model prediction result of the non-associative data is determined, and the target extraction result is determined according to the relationship between the model prediction result and the association result.

[0010] Determine the overlap relationship between the target extraction result and the first extraction result, reduce the target extraction result based on the overlap relationship, determine the second extraction result based on the reduction result, summarize the first extraction result and the second extraction result, and determine the summarized result as the extraction result of the target source data format.

[0011] Furthermore, when determining whether the extraction conditions are met based on the source data format, and determining the target source data format based on the determination result, the process includes:

[0012] The source data format includes the data format and file header;

[0013] Determine the standard data format and standard file header corresponding to the source data format, and compare the data format and file header with the standard data format and standard file header respectively;

[0014] When the data format is consistent with the standard data format and the file header is consistent with the standard file header, it is determined that the source data format meets the extraction conditions, and the source data format is determined as the target source data format.

[0015] If the data format is inconsistent with the standard data format, and / or the file header is inconsistent with the standard file header, then the source data format is determined not to meet the extraction conditions.

[0016] Furthermore, when determining a corresponding processing strategy based on the target source data format, the processing strategy includes a text strategy and an image strategy. When determining the target source data features based on the text strategy or the image strategy, the process includes:

[0017] Determine the target data format of the target source data format and the corresponding data content of the source data;

[0018] When the target data format is text data, the processing strategy is determined to be the text strategy;

[0019] When the target data format is video data or image data, the processing strategy is determined to be the image strategy;

[0020] The text strategy is based on NLP analysis of the data content, which involves word segmentation, stop word removal, and stemming. Dependency parsing is used to analyze the relationships between words and extract noun phrases. Noun phrases that conform to the syntactic structure are used as the target source data features.

[0021] The image strategy preprocesses the data content, including denoising, contrast adjustment, and edge sharpening, extracts feature vectors and content phrases from the preprocessed data content, and uses the feature vectors and content phrases as the target source data features.

[0022] Furthermore, when traversing the target source data features to determine the association similarity between each data point and the target source data features, and determining the first extraction result based on the association similarity, the process includes:

[0023] The data features of each data in the knowledge graph are determined, and the data features are compared with the target source data features. The association similarity is determined based on the comparison results.

[0024] Extract all data from the knowledge graph whose association similarity is greater than or equal to the association similarity threshold;

[0025] Cluster all data in the knowledge graph whose association similarity is less than the association similarity threshold;

[0026] The data from the extraction result and the data from the clustering result are determined as the first extraction result.

[0027] Furthermore, when clustering all data in the knowledge graph whose association similarity is less than the association similarity threshold, this includes:

[0028] A clustering dataset is established by combining all data in the knowledge graph with association similarity less than the association similarity threshold and the features of the target source data. The expected number of clusters k is determined to be 2. The parameters of the Gaussian distribution are initialized, the probability of each data in the clustering dataset belonging to each Gaussian distribution is determined, the responsibility value is determined, and the cluster with the largest responsibility value is selected as the clustering result.

[0029] Furthermore, when determining unrelated data based on the association similarity, determining the association result between the unrelated data and the association rule algorithm, and determining the model prediction result of the unrelated data based on the Bayesian model, the process includes:

[0030] All data in the knowledge graph with a similarity less than the similarity threshold are identified as non-associated data. Frequent combination terms are established for the non-associated data according to the Apriori algorithm. The association results between the non-associated data are determined based on the frequent combination terms.

[0031] The model building parameters are determined by random search. An initial Bayesian model is built based on the model building parameters, and a historical association set is obtained. The historical association set is divided into a model training set and a model validation set. The initial Bayesian model is trained based on the model training set and the model validation set to determine the target Bayesian model. The model prediction result of the non-associative data is determined based on the target Bayesian model.

[0032] Furthermore, when training the initial Bayesian model based on the model training set and model validation set, determining the target Bayesian model, and determining the model prediction result for the unrelated data based on the target Bayesian model, the process includes:

[0033] The initial Bayesian model is trained using the model training set, and the model validation set is substituted into the trained initial Bayesian model for validation.

[0034] If the model validation value of the current trained initial Bayesian model is less than the model validation value of the previous trained initial Bayesian model, then the L1 / L2 regularization term is used to adjust the current trained initial Bayesian model, and training continues until the model validation value of the trained initial Bayesian model is greater than or equal to the model validation value of the previous trained initial Bayesian model.

[0035] If the model validation value of the currently trained initial Bayesian model is greater than or equal to the model validation value of the previously trained initial Bayesian model, then training is stopped, and the currently trained initial Bayesian model is determined as the target Bayesian model.

[0036] The non-correlated data is substituted into the target Bayesian model to determine the model prediction result.

[0037] Furthermore, when determining the target extraction result based on the relationship between the model prediction result and the association result, the process includes:

[0038] When the model prediction result is consistent with the association result, the model prediction result or the association result is determined as the target extraction result;

[0039] When the model prediction result is inconsistent with the association result, the model prediction result and the association result are determined as the target extraction result.

[0040] Furthermore, when determining the overlap between the target extraction result and the first extraction result, reducing the target extraction result based on the overlap, and determining the second extraction result based on the reduction result, the process includes:

[0041] The data in the target extraction result is compared with the data in the first extraction result to determine whether there is any overlap;

[0042] When the data in the target extraction result does not overlap with the data in the first extraction result, the target extraction result is determined as the second extraction result.

[0043] When there is overlap between the data in the target extraction result and the data in the first extraction result, the data in the target extraction result that overlaps with the first extraction result is deleted, and the result of deletion is determined as the second extraction result.

[0044] Compared with existing technologies, the beneficial effects of this invention are as follows: By determining the source data format of the source data and judging whether the extraction conditions are met based on the source data format, extraction failures caused by data corruption, format spoofing, or missing metadata are avoided. For determining the corresponding processing strategy based on the target source data format, unified access and structured processing of multi-source heterogeneous data are achieved. This not only expands the data source of the knowledge graph from single text data to multimodal data, but also improves the stability of multimodal data extraction by traversing according to the characteristics of the target source data. Based on the characteristics of the target source data, the similarity between data is determined, forming the first extraction result of explicit associations. Then, for non-associative data, implicit association patterns are mined by combining association rule algorithms, and probabilistic inference is performed through a Bayesian model to generate model prediction results of implicit associations. The dual mining mechanism of explicit and implicit associations not only covers easily overlooked indirect relationships, but also ensures the complete capture of potential relationships between multimodal data through cross-validation of association results and model prediction results. This effectively avoids the limitations of incomplete data extraction. Furthermore, after obtaining the first extraction result and the target extraction result, duplicate data is identified through overlap relationship analysis, and the target extraction result is reduced to finally form the deduplicated second extraction result. The hierarchical data processing not only retains the effective information in the multimodal data, but also ensures the stability and reliability of multimodal data extraction through the automatic removal of redundant data.

[0045] On the other hand, this application also provides a knowledge graph-based multimodal data extraction system for applying the above-mentioned knowledge graph-based multimodal data extraction method, including:

[0046] The format confirmation unit is configured to determine the source data format of the source data, determine whether the extraction conditions are met based on the source data format, and determine the target source data format based on the judgment result.

[0047] The first extraction unit is configured to determine a corresponding processing strategy based on the target source data format, the processing strategy including a text strategy and an image strategy, determine the target source data features based on the text strategy or the image strategy, traverse based on the target source data features, determine the association similarity between each data and the target source data features, and determine the first extraction result based on the association similarity.

[0048] The second extraction unit is configured to determine unrelated data based on the association similarity, determine the association result between the unrelated data and the association rule algorithm, determine the model prediction result of the unrelated data based on the Bayesian model, and determine the target extraction result according to the relationship between the model prediction result and the association result.

[0049] The extraction and summarization unit is configured to determine the overlap relationship between the target extraction result and the first extraction result, reduce the target extraction result based on the overlap relationship, determine the second extraction result based on the reduction result, summarize the first extraction result and the second extraction result, and determine the summarized result as the extraction result of the target source data format.

[0050] It is understandable that the aforementioned knowledge graph-based multimodal data extraction method and system have the same beneficial effects, and will not be elaborated further here. Attached Figure Description

[0051] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0052] Figure 1 A flowchart of a multimodal data extraction method based on knowledge graphs provided in an embodiment of the present invention;

[0053] Figure 2 This is a functional block diagram of a knowledge graph-based multimodal data extraction system provided in an embodiment of the present invention. Detailed Implementation

[0054] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the disclosure to those skilled in the art. It should be noted that, unless otherwise specified, embodiments and features in the embodiments of the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0055] In some embodiments of this application, see Figure 1 As shown, a multimodal data extraction method based on knowledge graphs includes:

[0056] S100: Determine the source data format of the source data, judge whether the extraction conditions are met based on the source data format, and determine the target source data format based on the judgment result.

[0057] S200: Determine the corresponding processing strategy based on the target source data format. The processing strategy includes text strategy and image strategy. Determine the target source data features based on the text strategy or image strategy. Traverse the target source data based on the target source data features. Determine the association similarity between each data and the target source data features. Determine the first extraction result based on the association similarity.

[0058] S300: Identify unrelated data based on association similarity, determine the association results between unrelated data and association rule algorithms based on unrelated data, determine the model prediction results of unrelated data based on Bayesian model, and determine the target extraction results based on the relationship between model prediction results and association results.

[0059] S400: Determine the overlap between the target extraction result and the first extraction result, reduce the target extraction result based on the overlap, determine the second extraction result based on the reduction result, summarize the first extraction result and the second extraction result, and determine the summarized result as the extraction result in the target source data format.

[0060] Specifically, the focus is on the structured processing and correlation analysis of multimodal data, employing various processing strategies to achieve the entire process from data processing to intelligent extraction. By identifying the source data formats (text, images, and videos), extraction conditions are determined based on matching rules (such as format consistency and information consistency), ensuring the accuracy and reliability of the target source data format and guaranteeing inclusiveness for multi-source heterogeneous data extraction. Determining corresponding processing strategies based on the target source data format enhances the flexibility and compatibility of multimodal data extraction. These strategies include text and image strategies. Text strategies can determine target source data features through semantic features such as dependency syntax, while image strategies can utilize Faster... R-CNN detects and extracts visual entities, and combines it with ResNet to generate visual feature vectors to determine the features of the target source data. Knowledge graphs typically contain multimodal data such as text, images, and videos. Matching the data in the knowledge graph with the features of the target source data improves the stability of multimodal data extraction. First, it traverses the target source data features to determine the association similarity between each data point and the target source data features, thus determining the first extraction result. For non-associative data with low association similarity, although their association is poor during traversal, the relationships between the data may still have some connection with the features of the target source data. For example... For example, data A, B, and C may have low similarity to the target source data features. However, when data A, B, and C appear simultaneously, they will have a strong association with the target source data features. Therefore, identifying unrelated data based on association similarity and determining the association results between unrelated data using association rule algorithms can uncover implicit connections and potential patterns in the data. Then, a Bayesian model can be used to determine the model prediction results for unrelated data. The Bayesian model constructs a probabilistic inference model using a Bayesian network, leveraging prior knowledge (such as the distribution of historical entity associations) to predict the potential connections between unrelated data. For instance, when an unlabeled image of a "first aid kit" appears in the data, a Bayesian model infers its association probability with the nearby text "medical assistance" to generate a model prediction result. By comparing the model prediction result with the association result, the target extraction results with higher confidence are selected, thus achieving the explicit extraction of implicit knowledge. Finally, based on the overlap between the target extraction result and the first extraction result, it is determined whether there is redundancy between the two, and then the target extraction result is reduced. The second extraction result is determined based on the reduction result, and the first extraction result and the second extraction result are summarized, thus realizing the semantic parsing of unstructured data such as images and videos.

[0061] Understandably, in terms of deep data association analysis, the synergistic effect of association rule algorithms and Bayesian models has constructed a dual mechanism of feature matching and implicit association inference, realizing the leap from single-modal text data to multimodal data extraction, effectively reducing redundant data. At the same time, the introduction of Bayesian models provides a probability confidence assessment for the extraction results, thus forming a closed-loop extraction process of modality adaptation, dual association mining, and redundancy processing, thereby ensuring the reliability of the extraction results.

[0062] In some embodiments of this application, when determining whether the extraction conditions are met based on the source data format and determining the target source data format based on the determination result, the process includes: the source data format includes a data format and a file header; determining the standard data format and standard file header corresponding to the source data format; comparing the data format and file header with the standard data format and standard file header respectively; when the data format and the standard data format are consistent, and the file header and the standard file header are consistent, the source data format is determined to meet the extraction conditions, and the source data format is determined to be the target source data format; when the data format and the standard data format are inconsistent, and / or the file header and the standard file header are inconsistent, the source data format is determined not to meet the extraction conditions.

[0063] Specifically, the source data format includes the data format and the file header. The data format includes image formats, video formats, and text formats, etc. The file header refers to the beginning of the source data and usually contains information about the source data. For image formats, the standard data formats are represented by JPG, PNG, and BMP, etc. For video formats, the standard data formats are represented by MP4, MKV, and AVI, etc. For text formats, the standard data formats are represented by JSON, XML, and CSV, etc. The standard file header for the source data format of image is FF D8 FF E0, 89 50 4E 47 0D 0A 1A 0A, and 47 49 4638 37 61, etc. The standard file header for the source data format of video is 1A 45 DF A3 and ASCII (52 4946 46), etc. The standard file header for the source data format of text is 0xEF 0xBB 0xBF, 0xFF 0xFE, and 0xFE 0xFF, etc. By verifying the principle of double consistency, the integrity and compliance of the source data format are ensured. Only when both meet the consistency requirements can the source data format be determined to be effective, avoiding extraction failures caused by data corruption, format spoofing, or missing metadata, and providing a reliable data foundation for subsequent data processing.

[0064] In some embodiments of the present application, when determining the corresponding processing strategy based on the target source data format, the processing strategy includes a text strategy and an image strategy. When determining the target source data features based on the text strategy or the image strategy, it includes: determining the target data format of the target source data format and the data content of the corresponding source data. When the target data format is text data, the processing strategy is determined as the text strategy. When the target data format is video data or image data, the processing strategy is determined as the image strategy. The text strategy analyzes the data content based on NLP, performs word segmentation, stop word removal, and stemming on the data content, and uses dependency syntactic analysis to parse the relationships between words and extract noun phrases. The noun phrases that conform to the syntactic structure are used as the target source data features. The image strategy preprocesses the data content, and the preprocessing includes denoising, contrast adjustment, and edge sharpening. The feature vectors and content phrases of the preprocessed data content are extracted and used as the target source data features.

[0065] Specifically, although the source data format is determined as the target source data format, the entity content of the source data is not changed. Therefore, the target data format of the target source data format and the data content of the corresponding source data are determined. The data content reflects the specific content information of each format of data. When the target data format is text data, the processing strategy is determined as the text strategy. The text strategy analyzes the data content of the text data through natural language processing technology, including word segmentation, stop word removal, and stemming. Word segmentation divides the continuous text in the data content of the text data into independent words or phrases to facilitate subsequent processing. Stop word removal removes common words (such as "of") that do not affect the actual semantics in the data content to improve the reliability of the analysis. Stemming reduces semantic redundancy by reducing words to their root forms (e.g., "form of" and "form" are unified as "form"). Dependency syntactic analysis is used to parse the grammatical and semantic relationships between words and extract noun phrases from the data content. Dependency syntactic analysis can accurately identify noun phrases (such as "objective view" and "subjective condition"), which reflect the theme of the data content and are the key basis for extracting the target source data features. Through NLP technology and dependency syntactic analysis, the accurate extraction and effective identification of the target source data features are achieved. The extraction of the target source data features is based on the syntactic structure, ensuring the semantic value of the extracted content.

[0066] Understandably, when the target data format is video or image data, the processing strategy is determined to be an image strategy. The image strategy involves preprocessing, filtering random noise through denoising, enhancing image quality (each frame of the video data) through contrast adjustment, and sharpening edges to highlight the contours of the image data (each frame of the video data), further improving image quality and laying the foundation for feature extraction from the target source data. Feature vectors and content phrases are extracted from the preprocessed data content. Content phrases are the corresponding subtitles and text information in the video or image data, determined using NLP techniques and dependency parsing, consistent with the target source data feature extraction process of the text strategy, and will not be repeated here. Through the progressive application of NLP techniques, unstructured text is transformed into structured target source data features. Preprocessing enhances the quality of image and video data, ensuring the stability and reliability of target source data feature extraction and providing a semantic bridge for multimodal data extraction.

[0067] In some embodiments of this application, when traversing based on the features of the target source data, determining the association similarity between each data and the features of the target source data, and determining the first extraction result based on the association similarity, the process includes: determining the data features of each data in the knowledge graph, comparing the data features with the features of the target source data, determining the association similarity based on the comparison result, extracting all data in the knowledge graph with an association similarity greater than or equal to the association similarity threshold, clustering all data in the knowledge graph with an association similarity less than the association similarity threshold, and determining the extracted data and the clustered data as the first extraction result.

[0068] In some embodiments of this application, when clustering all data in the knowledge graph whose association similarity is less than the association similarity threshold, the process includes: establishing a clustering dataset with all data in the knowledge graph whose association similarity is less than the association similarity threshold and the features of the target source data; determining the expected number of clusters k as 2; initializing the parameters of the Gaussian distribution; determining the probability that each data in the clustering dataset belongs to each Gaussian distribution; determining the responsibility value; and selecting the cluster with the largest responsibility value as the clustering result.

[0069] Specifically, a two-layer screening system combining association similarity and cluster analysis is constructed to achieve refined processing of knowledge graph data. The features of the target source data (such as noun phrases extracted using text strategies and feature vectors generated using image strategies) are compared with each data point in the knowledge graph. This comparison focuses on the similarity between the data features of the target source data and the features of the target source data. Association similarity is determined using cosine similarity and Euclidean distance, and the association similarity threshold can be dynamically determined based on the actual amount of data in the knowledge graph. In this example, the preferred association similarity threshold is 0.7. Since association similarity calculations (such as cosine similarity and Euclidean distance) often remove some truly relevant but weakly performing data, relying solely on the association similarity threshold for extraction can easily lead to missing some truly useful associations due to an excessively high threshold. Cluster analysis identifies the cluster with the highest responsibility value as the clustering result, which can adaptively cope with the impact of different association similarity thresholds. This allows for the identification of clusters that are relatively closer to the feature distribution of the target source data, thus supplementing the extracted data accordingly. As source data is continuously input, new entity relationship patterns can be discovered by monitoring changes in cluster distribution (such as a continuous increase in the density of a certain cluster), thereby ensuring the stability and reliability of multimodal data extraction.

[0070] In some embodiments of this application, when determining unrelated data based on association similarity, determining the association results between unrelated data based on the unrelated data and the association rule algorithm, and determining the model prediction results of unrelated data based on the Bayesian model, the process includes: identifying all data in the knowledge graph with association similarity less than the association similarity threshold as unrelated data; establishing frequent combination items for the unrelated data according to the Apriori algorithm; determining the association results between the unrelated data based on the frequent combination items; determining model building parameters using random search; building an initial Bayesian model based on the model building parameters; obtaining a historical association set; dividing the historical association set into a model training set and a model validation set; training the initial Bayesian model based on the model training set and the model validation set; determining the target Bayesian model; and determining the model prediction results of the unrelated data based on the target Bayesian model.

[0071] Specifically, through a two-layer analysis combining association rule mining and Bayesian probabilistic inference, deep mining of non-associative data in the knowledge graph was achieved. The Apriori algorithm was used to mine frequent itemsets for non-associative data with similarity below a threshold, generating candidate itemsets layer by layer (e.g., the relationship between data A, B, and C: {A,B}, {B,C}), and determining support (itemset frequency). Generally, itemsets with support greater than or equal to the minimum support threshold are identified as frequent combinations, thus determining the association between non-associative data. Subsequently, a Bayesian model was constructed, and a random search algorithm was used to explore the parameter space (e.g., empirical probability distribution and conditional probability table structure). Heuristic search is performed to find the optimal parameter combination to construct the initial Bayesian model. The historical association set contains various association results determined by the association rule algorithm. The Bayesian model is trained using the historical association set, which is divided into a model training set and a model validation set. Typically, 70% of the data is used as the model training set and the remaining 30% as the model validation set to ensure the model's adaptability and generalization to different data. The model prediction results for non-associative data are determined based on the target Bayesian model, effectively avoiding the one-sidedness and limitations of relying solely on the association rule algorithm. The Bayesian model can further determine the potential connections between association results through probabilistic inference, thereby ensuring the stability and reliability of multimodal data extraction.

[0072] In some embodiments of this application, when training an initial Bayesian model based on a model training set and a model validation set, determining a target Bayesian model, and determining the model prediction result for unrelated data based on the target Bayesian model, the process includes: training the initial Bayesian model based on the model training set, and validating the initial Bayesian model by substituting the model validation set into the trained model; if the model validation value of the current trained initial Bayesian model is less than the model validation value of the previous trained initial Bayesian model, then adjusting the current trained initial Bayesian model using L1 / L2 regularization terms, and continuing training until the model validation value of the current trained initial Bayesian model is greater than or equal to the model validation value of the previous trained initial Bayesian model; if the model validation value of the current trained initial Bayesian model is greater than or equal to the model validation value of the previous trained initial Bayesian model, then stopping training, determining the current trained initial Bayesian model as the target Bayesian model, and substituting the unrelated data into the target Bayesian model to determine the model prediction result.

[0073] Specifically, the training set is used to train the Bayesian model, while the validation set is used to test the performance of the trained model. During each training iteration, the Bayesian model attempts to learn patterns and relationships in the data to improve its prediction or classification abilities. After each training iteration, the model is tested using data from the validation set, with metrics including recall, accuracy, and F1 score. If the model validation value is less than the validation value of the initial Bayesian model after the previous training, it indicates that the model's performance is at risk of decline, and it has not learned the true distribution and general patterns of the data. In this case, the trained model will have poor generalization and prediction capabilities when facing new and unseen data. Therefore, L1 / L2 regularization terms are used to adjust the current trained initial Bayesian model to improve its fit, and training continues until the validation value of the current trained initial Bayesian model is greater than or equal to the validation value of the previous trained initial Bayesian model. If the validation value is greater than or equal to the validation value of the previous trained initial Bayesian model, it indicates that the model's performance is showing an upward and stable trend, and it has reached a satisfactory performance level. At this point, training stops, and the current trained initial Bayesian model is designated as the target Bayesian model. The target Bayesian model can stably approach the global optimum, ensuring the stability and reliability of the model's prediction results.

[0074] In some embodiments of this application, when determining the target extraction result based on the relationship between the model prediction result and the association result, the method includes: when the model prediction result and the association result are consistent, determining the model prediction result or the association result as the target extraction result; when the model prediction result and the association result are inconsistent, determining the model prediction result and the association result as the target extraction result.

[0075] Specifically, by comparing the prediction results of the Bayesian model (potential associations derived from probabilistic inference) with the association results mined by the association rule algorithm (explicit associations generated by frequent itemsets), when the two are consistent, it proves that there is a correlation between the non-associative data and the characteristics of the target source data, and it is directly determined as the target extraction result. When the two are inconsistent, both results are retained and the target extraction result is determined, so as to improve the stability and reliability of data complementarity. In essence, it is the unification of probabilistic inference and statistical laws. It uses the predictive ability of the Bayesian model to capture implicit relationships, while ensuring the reliability of the results through the association rule algorithm, thus ensuring the stability and reliability of multimodal data extraction.

[0076] In some embodiments of this application, when determining the overlap between the target extraction result and the first extraction result, reducing the target extraction result based on the overlap, and determining the second extraction result based on the reduction result, the process includes: comparing the data in the target extraction result with the data in the first extraction result to determine whether there is overlap; when there is no overlap between the data in the target extraction result and the data in the first extraction result, the target extraction result is determined as the second extraction result; when there is overlap between the data in the target extraction result and the data in the first extraction result, the data in the target extraction result that overlaps with the first extraction result is reduced, and the data in the target extraction result that overlaps with the first extraction result is determined as the second extraction result based on the reduction result.

[0077] Specifically, by identifying the overlapping portion between the target extraction results (potential associations derived from probabilistic inference and explicit associations generated from frequent itemsets) and the first extraction results (data directly filtered based on association similarity), when there is overlap, it indicates that this part of the data has already been extracted in the first extraction result and belongs to the duplicate extraction data, which needs to be removed from the target extraction results to avoid redundant extraction data. When there is no overlap, it indicates that the target extraction results supplement the potential association data not covered by the first extraction results, and the target extraction results are determined as the second extraction results, directly included as effective increments in the final extraction results. By removing overlapping data, the duplicate extraction of the same entity or relationship is avoided, ensuring the non-repetitiveness and no omissions in the extraction of multimodal data, and ensuring the stability and reliability of the extraction of multimodal data.

[0078] In summary, the beneficial effects of this invention are as follows: By determining the source data format of the source data and judging whether the extraction conditions are met based on the source data format, extraction failures caused by data corruption, format spoofing, or missing metadata are avoided. For determining the corresponding processing strategy based on the target source data format, unified access and structured processing of multi-source heterogeneous data are achieved. This not only expands the data source of the knowledge graph from single text data to multimodal data, but also improves the stability of multimodal data extraction by traversing according to the characteristics of the target source data. Based on the characteristics of the target source data, the similarity between data is determined, forming the first extraction result of explicit associations. Then, for non-associative data, implicit association patterns are mined by combining association rule algorithms, and probabilistic inference is performed through a Bayesian model to generate model prediction results of implicit associations. The dual mining mechanism of explicit and implicit associations not only covers easily overlooked indirect relationships, but also ensures the complete capture of potential relationships between multimodal data through cross-validation of association results and model prediction results. This effectively avoids the limitations of incomplete data extraction. Furthermore, after obtaining the first extraction result and the target extraction result, duplicate data is identified through overlap relationship analysis, and the target extraction result is reduced to finally form the deduplicated second extraction result. The hierarchical data processing not only retains the effective information in the multimodal data, but also ensures the stability and reliability of multimodal data extraction through the automatic removal of redundant data.

[0079] In another preferred embodiment based on the above embodiments, see [reference] Figure 2 As shown, this embodiment provides a knowledge graph-based multimodal data extraction system for applying the above-described knowledge graph-based multimodal data extraction method, including:

[0080] The format confirmation unit is configured to determine the source data format of the source data, judge whether the extraction conditions are met based on the source data format, and determine the target source data format based on the judgment result.

[0081] The first extraction unit is configured to determine the corresponding processing strategy based on the target source data format. The processing strategy includes text strategy and image strategy. The target source data features are determined based on the text strategy or image strategy. The target source data features are traversed to determine the association similarity between each data and the target source data features. The first extraction result is determined based on the association similarity.

[0082] The second extraction unit is configured to determine unrelated data based on association similarity, determine the association results between unrelated data based on association rule algorithms, determine the model prediction results of unrelated data based on Bayesian models, and determine the target extraction results based on the relationship between model prediction results and association results.

[0083] The extraction and summarization unit is configured to determine the overlap relationship between the target extraction result and the first extraction result, reduce the target extraction result based on the overlap relationship, determine the second extraction result based on the reduction result, summarize the first extraction result and the second extraction result, and determine the summarized result as the extraction result in the target source data format.

[0084] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program goods. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program goods embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0085] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program goods according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0086] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0087] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0088] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A multimodal data extraction method based on knowledge graphs, characterized in that, include: Determine the source data format, which may be text, image, or video. Based on the source data format, determine whether the extraction conditions are met, and determine the target source data format based on the determination result. Based on the target source data format, a corresponding processing strategy is determined. The processing strategy includes a text strategy and an image strategy. Based on the text strategy or the image strategy, the target source data features are determined. Based on the target source data features, a traversal is performed to determine the association similarity between each data and the target source data features. Based on the association similarity, a first extraction result is determined. Based on the association similarity, non-associative data is determined, and based on the non-associative data and the association rule algorithm, the association result between the non-associative data is determined. Based on the Bayesian model, the model prediction result of the non-associative data is determined, and the target extraction result is determined according to the relationship between the model prediction result and the association result. The overlap relationship between the target extraction result and the first extraction result is determined. Based on the overlap relationship, the target extraction result is reduced. Based on the reduction result, a second extraction result is determined. The first extraction result and the second extraction result are summarized, and the summarized result is determined as the extraction result in the target source data format. When traversing the target source data features to determine the association similarity between each data point and the target source data features, and determining the first extraction result based on the association similarity, the process includes: The data features of each data in the knowledge graph are determined, and the data features are compared with the target source data features. The association similarity is determined based on the comparison results. Extract all data from the knowledge graph whose association similarity is greater than or equal to the association similarity threshold; Cluster all data in the knowledge graph whose association similarity is less than the association similarity threshold; The data from the extraction result and the data from the clustering result are determined as the first extraction result; When clustering all data in a knowledge graph whose association similarity is less than the association similarity threshold, the following applies: A clustering dataset is established by combining all data in the knowledge graph with association similarity less than the association similarity threshold and the features of the target source data. The expected number of clusters k is determined to be 2. The parameters of the Gaussian distribution are initialized, the probability of each data in the clustering dataset belonging to each Gaussian distribution is determined, the responsibility value is determined, and the cluster with the largest responsibility value is selected as the clustering result.

2. The multimodal data extraction method based on knowledge graphs according to claim 1, characterized in that, When determining whether the extraction conditions are met based on the source data format, and determining the target source data format based on the determination result, the process includes: The source data format includes the data format and file header; Determine the standard data format and standard file header corresponding to the source data format, and compare the data format and file header with the standard data format and standard file header respectively; When the data format is consistent with the standard data format and the file header is consistent with the standard file header, it is determined that the source data format meets the extraction conditions, and the source data format is determined as the target source data format. If the data format is inconsistent with the standard data format, and / or the file header is inconsistent with the standard file header, then the source data format is determined not to meet the extraction conditions.

3. The multimodal data extraction method based on knowledge graphs according to claim 2, characterized in that, When determining a corresponding processing strategy based on the target source data format, the processing strategy includes a text strategy and an image strategy. When determining the target source data features based on the text strategy or the image strategy, the process includes: Determine the target data format of the target source data format and the corresponding data content of the source data; When the target data format is text data, the processing strategy is determined to be the text strategy; When the target data format is video data or image data, the processing strategy is determined to be the image strategy; The text strategy is based on NLP analysis of the data content, which involves word segmentation, stop word removal, and stemming. Dependency parsing is used to analyze the relationships between words and extract noun phrases. Noun phrases that conform to the syntactic structure are used as the target source data features. The image strategy preprocesses the data content, including denoising, contrast adjustment, and edge sharpening, extracts feature vectors and content phrases from the preprocessed data content, and uses the feature vectors and content phrases as the target source data features.

4. The multimodal data extraction method based on knowledge graphs according to claim 3, characterized in that, When determining unrelated data based on the association similarity, determining the association result between the unrelated data and the association rule algorithm, and determining the model prediction result of the unrelated data based on the Bayesian model, the process includes: All data in the knowledge graph with a similarity less than the similarity threshold are identified as non-associated data. Frequent combination terms are established for the non-associated data according to the Apriori algorithm. The association results between the non-associated data are determined based on the frequent combination terms. The model building parameters are determined by random search. An initial Bayesian model is built based on the model building parameters, and a historical association set is obtained. The historical association set is divided into a model training set and a model validation set. The initial Bayesian model is trained based on the model training set and the model validation set to determine the target Bayesian model. The model prediction result of the non-associative data is determined based on the target Bayesian model.

5. The multimodal data extraction method based on knowledge graphs according to claim 4, characterized in that, Training the initial Bayesian model based on the model training set and model validation set, determining the target Bayesian model, and determining the model prediction result for the unrelated data based on the target Bayesian model include: The initial Bayesian model is trained using the model training set, and the model validation set is substituted into the trained initial Bayesian model for evaluation. If the model validation value of the current trained initial Bayesian model is less than the model validation value of the previous trained initial Bayesian model, then the L1 / L2 regularization term is used to adjust the current trained initial Bayesian model, and training continues until the model validation value of the trained initial Bayesian model is greater than or equal to the model validation value of the previous trained initial Bayesian model. If the model validation value of the currently trained initial Bayesian model is greater than or equal to the model validation value of the previously trained initial Bayesian model, then training is stopped, and the currently trained initial Bayesian model is determined as the target Bayesian model. The non-correlated data is substituted into the target Bayesian model to determine the model prediction result.

6. The multimodal data extraction method based on knowledge graphs according to claim 5, characterized in that, When determining the target extraction result based on the relationship between the model prediction result and the association result, the following are included: When the model prediction result is consistent with the association result, the model prediction result or the association result is determined as the target extraction result; When the model prediction result is inconsistent with the association result, the model prediction result and the association result are determined as the target extraction result.

7. The multimodal data extraction method based on knowledge graphs according to claim 6, characterized in that, When determining the overlap between the target extraction result and the first extraction result, reducing the target extraction result based on the overlap, and determining the second extraction result based on the reduction result, the process includes: The data in the target extraction result is compared with the data in the first extraction result to determine whether there is any overlap; When the data in the target extraction result does not overlap with the data in the first extraction result, the target extraction result is determined as the second extraction result. When there is overlap between the data in the target extraction result and the data in the first extraction result, the data in the target extraction result that overlaps with the first extraction result is deleted, and the result of deletion is determined as the second extraction result.

8. A knowledge graph-based multimodal data extraction system, used to apply the knowledge graph-based multimodal data extraction method as described in any one of claims 1-7, characterized in that, include: The format confirmation unit is configured to determine the source data format of the source data, wherein the source data format is text, image or video, and to determine whether the extraction conditions are met based on the source data format, and to determine the target source data format based on the judgment result. The first extraction unit is configured to determine a corresponding processing strategy based on the target source data format, the processing strategy including a text strategy and an image strategy, determine the target source data features based on the text strategy or the image strategy, traverse based on the target source data features, determine the association similarity between each data and the target source data features, and determine the first extraction result based on the association similarity. The second extraction unit is configured to determine unrelated data based on the association similarity, determine the association result between the unrelated data and the association rule algorithm, determine the model prediction result of the unrelated data based on the Bayesian model, and determine the target extraction result according to the relationship between the model prediction result and the association result. The extraction and summarization unit is configured to determine the overlap relationship between the target extraction result and the first extraction result, reduce the target extraction result based on the overlap relationship, determine the second extraction result based on the reduction result, summarize the first extraction result and the second extraction result, and determine the summarized result as the extraction result of the target source data format.

Images