Method for constructing traditional Chinese and western medicine multi-modal knowledge graph based on multi-level game weight strategy

Abstract

The present application relates to the technical field of knowledge graph construction, and particularly relates to a Chinese and Western medicine multi-modal knowledge graph construction method based on a multi-level game weight strategy. The present application represents the structural difference between modalities and disease categories by obtaining the modality-disease correlation strength, obtains candidate triples and their confidence under each modality, and provides a data basis for graph construction; fuses the confidence within the modality to obtain an evidence support coefficient to reflect the overall authenticity of the modality; quantifies the modality abnormal risk by analyzing the deviation of the evidence support coefficient of each modality, and calculates the modality consistency; generates a reference weight based on the correlation strength and the modality consistency, realizes a dynamic game that high correlation modalities can make up for low consistency and high consistency can improve the weight proportion of low correlation modalities; finally, the reference weight is used to correct the confidence to obtain the game weight, and then the candidate triples of the knowledge graph are screened and constructed, and the present application significantly improves the integrity and reliability of the knowledge graph construction.

Description

Technical Field

[0001] This invention relates to the field of knowledge graph construction technology, specifically to a method for constructing a multimodal knowledge graph of traditional Chinese and Western medicine based on a multi-level game weight strategy. Background Technology

[0002] Traditional Chinese medicine data (such as tongue features, pulse waveforms, and chief complaint text) has the characteristics of being holistic, dynamic, and non-linear; Western medicine data (such as laboratory indicators and imaging features) has the characteristics of being highly accurate, standardized, and localized. These two types of data together constitute a multimodal information source for the same knowledge object.

[0003] Currently, when constructing multimodal knowledge graphs spanning billions of data points across different systems, equal weights are commonly used to weight and prune candidate triples extracted from different modalities. However, in the collaborative construction of knowledge graphs from multimodal heterogeneous data, there are structural differences in the strength of association between different modalities and specific disease categories: the original data features of some modalities are highly co-occurring with specific disease categories, exhibiting extremely high and stable association strength; while other modalities, although weakly associated with the same disease category, still show non-zero weak dependencies in large-scale data statistics, possessing significant potential information. Using equal weights for fusion and pruning easily overlooks the structural differences in the strength of association between modalities and disease categories. On the one hand, this leads to the high-deterministic structured evidence provided by highly associated modalities being diluted by the high-noise information of low-associated modalities; on the other hand, a large number of weakly associated edges that still have potential statistical value are incorrectly pruned, causing edge weight distortion or knowledge gaps in the knowledge graph during the construction stage, reducing the coverage completeness and data fidelity of the knowledge base. Therefore, existing technologies that use equal weights to fuse multimodal data ignore the structural differences between different modalities, resulting in low completeness and accuracy of knowledge graph construction. Summary of the Invention

[0004] To address the problem that existing technologies using equal weights to fuse multimodal data neglect structural differences between different modalities, resulting in low completeness and accuracy of knowledge graph construction, this invention aims to provide a method for constructing a multimodal knowledge graph of traditional Chinese and Western medicine based on a multi-level game-theoretic weighting strategy. The specific technical solution adopted is as follows: This invention first proposes a method for constructing a multimodal knowledge graph of traditional Chinese and Western medicine based on a multi-level game weighting strategy. The method includes: In a multimodal medical dataset, obtain all candidate triples for each disease category under each modality of data and the confidence level of each candidate triple; obtain the association strength between each modality of data and each disease category; The confidence scores of each modality are fused to determine the evidence support coefficient of each modality. The modality consistency of each modality is determined based on the mutual deviation between its evidence support coefficient and those of all other modalities. Based on the association strength and modality consistency, a reference weight between each modality and each disease category is analyzed and determined. The confidence scores are then adjusted using these reference weights to obtain the game weights for each candidate triple. Based on the numerical value of the game weights, it is determined whether different candidate triples can construct a multimodal knowledge graph.

[0005] Furthermore, the process of obtaining all candidate triples for each disease category under each modality of data and the confidence level corresponding to each candidate triple includes: Separate plain text data from non-text data in all modal data; Input plain text data into the BERT entity relation extraction model and output each plain text triple and its corresponding confidence score; Non-text data is input into a pre-trained entity relation extraction convolutional neural network, which outputs each non-text triple and its corresponding confidence score. Both the plain text triples and the non-text triples are considered as candidate triples.

[0006] Furthermore, the method for obtaining the correlation strength includes: For each modality of data, the number of candidate triples included in each disease category is normalized to obtain the association strength between each modality of data and each disease category.

[0007] Furthermore, the method for obtaining the evidence support coefficient includes: The evidence support coefficient for each modality is determined by the average confidence level of all candidate triples for each modality.

[0008] Furthermore, the method for obtaining the modal consistency includes: Choose one modality of data as the target modality; determine the comparison coefficient based on the overall magnitude of the evidence support coefficients of all other modalities besides the target modality; and perform a negative correlation mapping between the difference between the evidence support coefficient of the target modality and the comparison coefficient to obtain the modality consistency of the target modality.

[0009] Furthermore, the method for obtaining the reference weights includes: The sum of the association strength and the modality consistency is used as a reference weight between each modality data and each disease category.

[0010] Furthermore, the method for obtaining the game weights includes: Each disease category under the target modality is sequentially taken as the target category; the game weight of each candidate triple is determined based on the reference weight between the target modality and the target category and the confidence level of each candidate triple in the target category; the reference weight and the confidence level are both positively correlated with the game weight.

[0011] Furthermore, the determination of whether different candidate triples can construct a multimodal knowledge graph includes: If the game weight is greater than a preset threshold, a multimodal knowledge graph is constructed using the corresponding candidate triples; If the game weight is less than or equal to a preset threshold, then the corresponding candidate triples will not be used to construct a multimodal knowledge graph.

[0012] Furthermore, the normalization method is maximum value normalization.

[0013] This invention also proposes a multimodal knowledge graph construction system for traditional Chinese and Western medicine based on a multi-level game weighting strategy. The system includes: The data acquisition module is used to acquire all candidate triples for each disease category under each modality of data and the confidence level of each candidate triple in a multimodal medical dataset; and to acquire the association strength between each modality of data and each disease category. The multimodal game analysis module is used to fuse all confidence scores for each modality of data to determine the evidence support coefficient for each modality; determine the modality consistency of each modality of data based on the mutual deviation between the evidence support coefficients of each modality of data and the evidence support coefficients of all other modalities; analyze and determine the reference weight between each modality of data and each disease category based on the association strength and the modality consistency; and use the reference weight to correct the confidence scores to obtain the game weight for each candidate triple. The graph construction module is used to determine whether different candidate triples can be used to construct a multimodal knowledge graph based on the numerical magnitude of the game weights.

[0014] The present invention has the following beneficial effects: This invention first obtains the association strength between modal data and disease categories to objectively quantify the reliability of diagnostic methods for different diseases under different modal data, and can characterize the structural differences between different modal data and specific disease categories. By obtaining candidate triples for each disease category under each modal data, it can provide data support for the subsequent construction of a knowledge graph. Simultaneously, it obtains the confidence level of the candidate triples, providing reliable basic data for analyzing whether different candidate triples can be used to construct a knowledge graph. By fusing all confidence levels under each modal data, it characterizes the overall authenticity of different modal data, and the resulting evidence support coefficient reflects the overall low-quality noise level within the modal data. Furthermore, it analyzes the evidence support coefficients of each modal data with other modal data. The deviation between them can quantify the anomaly risk of each modality's data. Furthermore, analyzing the correlation strength and modality consistency yields reference weights, realizing a dynamic game effect between correlation strength differences and modality consistency. This allows highly correlated modalities to supplement low-modality consistency, reducing the dilution effect of correlation noise. Simultaneously, high modality consistency increases the weight proportion of low-correlation modalities, preventing the erroneous removal of weakly correlated edges with potential statistical value. The reference weights are then used to correct the confidence of each candidate triplet, and the resulting game weights effectively improve the accuracy of candidate triplet selection. Finally, triples for constructing the knowledge graph are selected based on the magnitude of the game weights, significantly improving the completeness and accuracy of knowledge graph construction. Attached Figure Description

[0015] Figure 1 A flowchart illustrating a method for constructing a multimodal knowledge graph of traditional Chinese and Western medicine based on a multi-level game weighting strategy, as provided in an embodiment of the present invention; Figure 2 This is a structural block diagram of a multimodal knowledge graph construction system for traditional Chinese and Western medicine based on a multi-level game weight strategy, provided as an embodiment of the present invention. Detailed Implementation

[0016] The following description, in conjunction with the accompanying drawings, details a specific scheme for constructing a multimodal knowledge graph of traditional Chinese and Western medicine based on a multi-level game weight strategy, as provided by this invention.

[0017] Please see Figure 1 The diagram illustrates a flowchart of a method for constructing a multimodal knowledge graph of traditional Chinese and Western medicine based on a multi-level game weight strategy, according to an embodiment of the present invention. The method includes: Step S101: In the multimodal medical dataset, obtain all candidate triples for each disease category under each modality of data and the confidence level corresponding to each candidate triple; obtain the association strength between each modality of data and each disease category.

[0018] Different modal data can be viewed as different dimensions of information obtained from the same disease, collectively forming multi-perspective evidence supporting the disease. Examples include multiple modal data from Traditional Chinese Medicine (TCM) such as medical history taking, tongue diagnosis, and pulse diagnosis, and multiple modal data from Western medicine such as X-rays, ultrasound images, and biochemical test indicators. Since different modal data reflect different disease categories—for example, biochemical test indicators provide richer information while medical history taking is less comprehensive—this embodiment first obtains each disease category under each modal data before extracting candidate triples to avoid cross-disease category modal statistics interfering with each other. As an example, in this embodiment, the types of modal data include medical history taking data, tongue diagnosis data, pulse diagnosis, X-rays, ultrasound images, and biochemical test indicators. Implementers can adjust other types of modal data according to specific implementation scenarios; this is not limited here.

[0019] Specifically, in this embodiment of the invention, the disease category can be a specific name of each disease, such as type 1 diabetes, type 2 diabetes, rheumatoid arthritis, and chronic gastritis; or it can be a superordinate concept of similar diseases, for example, classifying closed fractures, comminuted fractures, and compression fractures as fracture diseases, and classifying gastric ulcers and duodenal ulcers as peptic ulcer diseases. In actual use cases, implementers can adjust according to specific situations. The specific classification process is a technical means well-known to those skilled in the art, and will not be elaborated or limited here. As an example, this embodiment of the invention uses superordinate concepts of similar diseases as each disease category.

[0020] It should be noted that each disease category under each modality of data corresponds to a portion of the data in that modality. For example, the fracture disease category corresponds to all X-ray images belonging to fracture diseases in the modality of data. In this embodiment of the invention, each candidate triple is extracted from the portion of data corresponding to each disease category under each modality of data to provide basic data support for the subsequent construction of a knowledge graph. Considering that there is noise caused by misjudgment in the modality data, which leads to different probabilities of the head entity-relationship-tail entity combination in each candidate triple, this embodiment further obtains the confidence level of each candidate triple, which can preliminarily characterize the reliability of each candidate triple.

[0021] Preferably, the process of obtaining all candidate triples for each disease category under each modality of data and the confidence level corresponding to each candidate triple includes: Since all modal data includes plain text, image, and video data, using the same extraction model cannot adapt to different types of modal data, making it difficult to obtain reliable entity and relation extraction results. Therefore, this embodiment of the invention distinguishes between plain text and non-text data within all modal data; that is, all text-based data is treated as plain text data, while image and video data are treated as non-text data.

[0022] Considering that the BERT entity relation extraction model can effectively extract semantic information from text-type data and has high accuracy in entity recognition and relation classification, this embodiment inputs plain text data into the BERT entity relation extraction model, outputting each plain text triple and the confidence probability values ​​between the head entity, relation, and tail entity in the plain text triple, ranging from... The confidence level of each plain text triple is then determined, and this confidence probability value is used as the confidence level of each plain text triple.

[0023] Non-text data cannot be processed by the BERT entity relation extraction model. Therefore, this embodiment trains an entity relation extraction convolutional neural network (CNN) in advance and inputs the non-text data into the entity relation extraction convolutional neural network to output each non-text triple and its corresponding confidence score.

[0024] Both the plain text triples and the non-text triples are considered as candidate triples.

[0025] Specifically, the training process of the entity relation extraction convolutional neural network includes: First, historical images and video frames from non-text data are selected as training samples. Head entities and tail entities in the training samples are manually labeled for location and category, and relationship labels between head and tail entities are also defined. Second, a convolutional neural network (CNN) is used for training. This CNN includes a feature extraction network, an entity detection branch, and a relationship classification branch. The training samples are input into the CNN, and the feature extraction network extracts visual feature maps. The entity detection branch then extracts the predicted location of each head entity and the predicted location of each tail entity from the visual feature maps. The predicted position of the entity is obtained, and the head and tail entities are concatenated and input into the relation classification branch. The output is the predicted confidence probability of the inter-entity relationship, as well as the non-text triples composed of entities and relations. The model parameters of the convolutional neural network are updated using a cross-entropy loss function and gradient descent algorithm through backpropagation until the model converges, for example, by reaching the maximum number of iterations (e.g., 100 times) or the rate of change of the loss function value in two adjacent iterations is less than the minimum loss change value (e.g., 2%), thus completing the training. The annotation process and the specific training process are well-known to those skilled in the art and are not specifically described or limited. It should be noted that in this embodiment of the invention, the accurate confidence probability of the inter-entity relationship output by the entity relation extraction convolutional neural network obtained after training is used as the confidence level of each non-text triple, with a range of... .

[0026] Each disease category corresponds to a portion of the data under each modality. That is, there is no modality without a disease category. Therefore, each candidate triple extracted from the modality data also corresponds to a disease category. Each disease category includes multiple candidate triples and the confidence of the candidate triples.

[0027] Since not all modal data can effectively describe the same disease category—for example, X-ray images can comprehensively and clearly represent fracture-related information with a strong correlation, while medical history cannot accurately assess fracture-related information with a weak correlation—there are differences in the strength of the correlation between different modal data and different disease categories. Based on the above analytical approach, this embodiment characterizes the differences in the descriptive ability of different modal data for different disease categories by obtaining the correlation strength between each modal data and each disease category.

[0028] Preferably, the method for obtaining the correlation strength includes: For each modality of data, the number of candidate triples included in each disease category is normalized to obtain the association strength between each modality of data and each disease category.

[0029] The more candidate triples extracted from the partial modal data corresponding to each disease category, the more accurately and comprehensively the modal data can describe the various pathological information of the disease category, and the stronger the association between the modal data and the disease category. Therefore, this embodiment counts the number of candidate triples included in each disease category under each modal data, and normalizes the number of candidate triples to use as the association strength between each modal data and each disease category.

[0030] Preferably, in this embodiment of the invention, since the number of candidate triples will not be negative, this embodiment uses a maximum value normalization algorithm for normalization. The process of normalizing the number of candidate triples included in each disease category under any modality of data is described in detail below: The number of candidate triples included in each disease category under any modality of data is used as the reference strength. The reference strength is normalized using a maximum value normalization algorithm, and the normalized result is used as the association strength between each modality of data and each disease category, within the range of... The maximum value used in the maximum value normalization algorithm can be obtained by statistically analyzing a large amount of historical data. That is, the maximum reference intensity extracted in history is used as the maximum value to achieve maximum value normalization. If the extracted maximum value is 0, it means that no record has been generated between the modality data and the disease category within the current statistical range. The normalization result is directly set to 0, indicating that there is no correlation. It should be noted that if the actual calculated reference intensity is greater than the maximum value, the normalization result is forcibly truncated to a positive integer of 1.

[0031] Step S102: Fuse all confidence scores for each modality to determine the evidence support coefficient for each modality; determine the modality consistency for each modality based on the mutual deviation between the evidence support coefficients of each modality and the evidence support coefficients of all other modalities; analyze and determine the reference weight between each modality and each disease category based on the association strength and the modality consistency; use the reference weight to correct the confidence scores to obtain the game weight for each candidate triple.

[0032] Considering that the same disease category manifests differently across different objects—for example, fever may coexist with weakness in one object, while only fever may be present in another—not all candidate triples for the same disease category within each modality can represent the typical features of the disease category. To accurately evaluate all typical data features within each modality, this embodiment fuses all confidence scores for each modality to determine the evidence support coefficient. A higher evidence support coefficient indicates less outlier noise in the modality data and more typical data features that can be used to construct a knowledge graph; therefore, it should be given greater weight in the subsequent multimodal data fusion process for constructing the knowledge graph.

[0033] Preferably, the method for obtaining the evidence support coefficient includes: The evidence support coefficient for each modality is determined by the average confidence level of all candidate triples for each modality.

[0034] The confidence score of each candidate triple characterizes the credible probability of the relationship between the head and tail entities included in the candidate triple. A higher confidence probability indicates that the candidate triple is more likely to represent a typical data feature. Therefore, in this embodiment of the invention, the average confidence score of all candidate triples for each modality of data is used as the evidence support coefficient for each modality of data, with a range of [range missing]. .

[0035] The strength of association between each modality and each disease category does not always represent the situation in every specific instance. When multi-source data is subsequently fused into a knowledge graph, there may be situations where the candidate triples given by modality data that is highly associated with a certain disease category are completely opposite to those of other modality groups. That is, the confidence of the candidate triples of a certain modality data deviates significantly from the average confidence of the same candidate triples of other modality groups. This often means that there is a high risk of heterogeneity in the modality data, which may be due to collection errors, entity disambiguation errors, or model mis-extraction.

[0036] Based on the above analysis, it is necessary to quantify the consistency between different modal data. Considering that the evidence support coefficient integrates the confidence of all candidate triples in each modal data, this embodiment of the invention determines the modal consistency of each modal data based on the mutual deviation between the evidence support coefficient of each modal data and the evidence support coefficients of all other modal data. The higher the modal consistency, the lower the risk of heterogeneity such as acquisition error in the corresponding modal data and the greater the overall reliability of the candidate triples provided by the modal data.

[0037] Preferably, the method for obtaining the modal consistency includes: Choose one modality of data as the target modality; determine a comparison coefficient based on the overall magnitude of the evidence support coefficients of all other modalities besides the target modality. The meaning of this comparison coefficient is: the evaluation benchmark for the consistency performance of the target modality when the other modalities besides the target modality are used as the group consensus; since the smaller the difference between the evidence support coefficient of the target modality and the comparison coefficient, the higher the consistency of the target modality, and vice versa, the larger the difference, the lower the consistency of the target modality, further negatively correlate the difference between the evidence support coefficient of the target modality and the comparison coefficient to obtain the modal consistency of the target modality.

[0038] In one implementation of this invention, the average value of the evidence support coefficients of all modalities other than the target modality is used to represent the overall magnitude of the evidence support coefficients of all modalities other than the target modality, and this average value is used as a comparison coefficient, with a value range of [missing value]. The absolute value of the difference between the evidence support coefficient of the target modality and the comparison coefficient is expressed as the difference between the evidence support coefficient of the target modality and the comparison coefficient, and the range is [missing value]. The absolute value of the difference is negatively correlated by subtracting the positive integer 1 from it, and the resulting difference is used as the modal consistency of the target mode, ranging from [value missing]. .

[0039] In summary, modality consistency can characterize the degree of fit between each modality's data and the consensus of other modal groups. High modality consistency indicates that the evidence (candidate triples) currently provided by the modality's data is consistent with the overall direction of other modal data, with low internal noise levels and reliable data quality. Association strength can characterize the degree of prior diagnostic association between each modality and a specific disease category at the historical statistical level. High association strength indicates that the source of the modality's data has a high structural descriptive advantage for this type of disease category.

[0040] Therefore, this embodiment of the invention integrates the correlation strength and the modal consistency to analyze and determine the reference weight between each modal data and each disease category. This allows for the comprehensive consideration of highly correlated modal data and highly consistent modal data, ensuring that a lower reference weight is only obtained when the correlation strength between the modal data and the disease category is low and the modal consistency between the modal data and all other modal data is low. This embodiment of the invention implements a game theory based on the relationship between correlation strength and modal consistency: when the modal consistency is low but the correlation strength is high, the obtained reference weight is compensated by the higher correlation strength, thereby preventing the high deterministic structured evidence provided by the high correlation modality from being diluted by the high noise information of the low correlation modality; when the correlation strength is low but the modal consistency is high, the obtained reference weight is increased by the higher modal consistency, effectively reducing the possibility of a large number of weakly correlated edges that still have potential statistical value being erroneously pruned.

[0041] Furthermore, the method for obtaining the reference weights includes: The sum of the association strength and the modal consistency is used as the reference weight between each modality and each disease category. The range of values ​​for this reference weight is... .

[0042] Considering that the reference weights are a measure between each modality of data and each disease category, while each specific candidate triplet is used when constructing the knowledge graph, after obtaining the above reference weights, it is necessary to use the reference weights to correct the confidence of each candidate triplet and obtain the game weight of each candidate triplet, so as to improve the accuracy of subsequent candidate triplet screening and the reliability of knowledge graph construction.

[0043] Preferably, the method for obtaining the game weights includes: Each disease category under the target modality is sequentially taken as the target category; the game weight of each candidate triple is determined based on the reference weight between the target modality and the target category and the confidence level of each candidate triple in the target category; the reference weight and the confidence level are both positively correlated with the game weight.

[0044] It should be noted that each modality of data includes each disease category, and each disease category includes each candidate triple. In order to ensure that the reference weights and confidence levels used when calculating game weights are under the same modality of data, this embodiment uses the reference weight between the target modality and the target category, and each confidence level under the target category to determine the game weight of each candidate triple. This ensures that the calculation of game weights is accurately aligned within the modality of data and the disease category, and avoids misuse of weights due to index mismatch.

[0045] As an example, the game weight of each candidate triple can be represented by the following formula: ,in, Let be the game weight of the i-th candidate triple in the k-th disease category under the m-th modality of data, and take the value of . ; The reference weight between the m-th modality and the k-th disease category is set to a value of [value]. ; Let be the confidence score of the i-th candidate triplet in the k-th disease category under the m-th modality of data, and take the value of . .

[0046] Step S103: Based on the numerical value of the game weights, determine whether different candidate triples can construct a multimodal knowledge graph. The larger the game weight of a candidate triple, the more reliable the head entity-relationship-tail entity combination of the candidate triple is in cross-modal evidence fusion, and thus more likely it can be used to construct a multimodal knowledge graph.

[0047] Preferably, determining whether different candidate triples can construct a multimodal knowledge graph includes: If the game weight is greater than a preset threshold, it indicates that the reliability of the candidate triple is higher and it can be entered into the knowledge graph. Then, a multimodal knowledge graph is constructed using the corresponding candidate triple. If the game weight is less than or equal to the preset threshold, it means that the candidate triple has not yet met the inclusion criteria under the current fusion conditions. There may be intermodal evidence conflicts, low self-extraction confidence, or insufficient structural description ability of the modal data to the disease category to which the candidate triple belongs. In this case, the corresponding candidate triple will not be used to construct a multimodal knowledge graph.

[0048] Specifically, in this embodiment of the invention, the preset threshold can be set at... In this embodiment, the preset threshold is preferably set to 0.9. Obviously, the implementer can also flexibly configure the preset threshold according to the specific implementation scenario. For example, when the accuracy requirement of the knowledge graph is high in the scenario, the value of the preset threshold can be appropriately increased, such as setting it to 1.3, to reduce the possibility of introducing erroneous edges. When the scenario pays more attention to the coverage integrity of the knowledge graph or has a higher tolerance for noise, the value of the preset threshold can be reduced, such as setting it to 0.6, to improve the knowledge density and coverage of the knowledge graph.

[0049] Using candidate triples with game weights greater than a preset threshold obtained by this invention to construct knowledge graphs can significantly improve the completeness and accuracy of knowledge graph construction. The specific knowledge graph construction methods are well known to those skilled in the art and will not be further elaborated or limited here.

[0050] In summary: This invention first obtains the correlation strength between modal data and disease categories to objectively quantify the reliability of diagnostic methods for different diseases under different modal data, and can characterize the structural differences between different modal data and specific disease categories; by obtaining candidate triples for each disease category under each modal data, it can provide data support for the subsequent construction of knowledge graphs, and at the same time obtain the confidence of candidate triples, providing reliable basic data for analyzing whether different candidate triples can be used to construct knowledge graphs; by fusing all confidence scores under modal data, it characterizes the overall authenticity of different modal data, and the obtained evidence support coefficient can reflect the overall low-quality noise level within the modal data; further, by analyzing the evidence support of each modal data with other modal data... The deviation between the support coefficients can quantify the anomaly risk of each modality's data. Further analysis of the association strength and modality consistency yields reference weights, realizing a dynamic game effect between the differences in association strength and the degree of consistency between modalities. This allows highly correlated modalities to supplement the consistency of low-modal modalities, reducing the dilution effect of association noise. Simultaneously, high modality consistency increases the weight proportion of low-correlation modalities, preventing the erroneous pruning of weakly correlated edges with potential statistical value. The reference weights are then used to correct the confidence of each candidate triplet, and the resulting game weights effectively improve the accuracy of candidate triplet selection. Finally, triples for constructing the knowledge graph are selected based on the magnitude of the game weights, significantly improving the completeness and accuracy of the knowledge graph construction.

[0051] Based on the same inventive concept, this invention also proposes a multimodal knowledge graph construction system for traditional Chinese and Western medicine based on a multi-level game weighting strategy. Please refer to [link / reference]. Figure 2 The diagram illustrates a structural block diagram of a multimodal knowledge graph construction system for traditional Chinese and Western medicine based on a multi-level game weight strategy, provided by an embodiment of the present invention. The system includes: a data acquisition module 201, a multimodal game analysis module 202, and a graph construction module 203.

[0052] The data acquisition module 201 is used to acquire, in the multimodal medical dataset, all candidate triples for each disease category under each modality of data and the confidence level corresponding to each candidate triple; and to acquire the association strength between each modality of data and each disease category. The multimodal game analysis module 202 is used to fuse all confidence scores of each modality data to determine the evidence support coefficient of each modality data; determine the modality consistency of each modality data based on the mutual deviation between the evidence support coefficient of each modality data and the evidence support coefficients of all other modality data; analyze and determine the reference weight between each modality data and each disease category based on the association strength and the modality consistency; and use the reference weight to correct the confidence scores to obtain the game weight of each candidate triple. The graph construction module 203 is used to determine whether different candidate triples can construct a multimodal knowledge graph based on the numerical magnitude of the game weights.

Claims

1. A method for constructing a multimodal knowledge graph of traditional Chinese and Western medicine based on a multi-level game weighting strategy, characterized in that, The method includes: In a multimodal medical dataset, obtain all candidate triples for each disease category under each modality of data and the confidence level of each candidate triple; obtain the association strength between each modality of data and each disease category; The confidence scores of each modality are fused to determine the evidence support coefficient of each modality. The modality consistency of each modality is determined based on the mutual deviation between its evidence support coefficient and those of all other modalities. Based on the association strength and modality consistency, a reference weight between each modality and each disease category is analyzed and determined. The confidence scores are then adjusted using these reference weights to obtain the game weights for each candidate triple. Based on the numerical value of the game weights, it is determined whether different candidate triples can construct a multimodal knowledge graph.

2. The method for constructing a multimodal knowledge graph of traditional Chinese and Western medicine based on a multi-level game weighting strategy according to claim 1, characterized in that, The process of obtaining all candidate triples for each disease category under each modality of data and the confidence level corresponding to each candidate triple includes: Separate plain text data from non-text data in all modal data; Input plain text data into the BERT entity relation extraction model and output each plain text triple and its corresponding confidence score; Non-text data is input into a pre-trained entity relation extraction convolutional neural network, which outputs each non-text triple and its corresponding confidence score. Both the plain text triples and the non-text triples are considered as candidate triples.

3. The method for constructing a multimodal knowledge graph of traditional Chinese and Western medicine based on a multi-level game weighting strategy according to claim 1, characterized in that, The method for obtaining the correlation strength includes: For each modality of data, the number of candidate triples included in each disease category is normalized to obtain the association strength between each modality of data and each disease category.

4. The method for constructing a multimodal knowledge graph of traditional Chinese and Western medicine based on a multi-level game weighting strategy according to claim 1, characterized in that, The method for obtaining the evidence support coefficient includes: The evidence support coefficient for each modality is determined by the average confidence level of all candidate triples for each modality.

5. The method for constructing a multimodal knowledge graph of traditional Chinese and Western medicine based on a multi-level game weighting strategy according to claim 1, characterized in that, The method for obtaining modal consistency includes: Choose one modality of data as the target modality; determine the comparison coefficient based on the overall magnitude of the evidence support coefficients of all other modalities besides the target modality; and perform a negative correlation mapping between the difference between the evidence support coefficient of the target modality and the comparison coefficient to obtain the modality consistency of the target modality.

6. The method for constructing a multimodal knowledge graph of traditional Chinese and Western medicine based on a multi-level game weighting strategy according to claim 1, characterized in that, The method for obtaining the reference weights includes: The sum of the association strength and the modality consistency is used as a reference weight between each modality data and each disease category.

7. The method for constructing a multimodal knowledge graph of traditional Chinese and Western medicine based on a multi-level game weighting strategy according to claim 5, characterized in that, The methods for obtaining the game weights include: Each disease category under the target modality is sequentially taken as the target category; the game weight of each candidate triple is determined based on the reference weight between the target modality and the target category and the confidence level of each candidate triple in the target category; the reference weight and the confidence level are both positively correlated with the game weight.

8. The method for constructing a multimodal knowledge graph of traditional Chinese and Western medicine based on a multi-level game weighting strategy according to claim 1, characterized in that, The determination of whether different candidate triples can construct a multimodal knowledge graph includes: If the game weight is greater than a preset threshold, a multimodal knowledge graph is constructed using the corresponding candidate triples; If the game weight is less than or equal to a preset threshold, then the corresponding candidate triples will not be used to construct a multimodal knowledge graph.

9. The method for constructing a multimodal knowledge graph of traditional Chinese and Western medicine based on a multi-level game weighting strategy according to claim 3, characterized in that, The normalization method is maximum value normalization.

10. A system for constructing a multimodal knowledge graph of traditional Chinese and Western medicine based on a multi-level game weighting strategy, characterized in that, The system includes: The data acquisition module is used to acquire all candidate triples for each disease category under each modality of data and the confidence level of each candidate triple in a multimodal medical dataset; and to acquire the association strength between each modality of data and each disease category. The multimodal game analysis module is used to fuse all confidence scores for each modality of data to determine the evidence support coefficient for each modality; determine the modality consistency of each modality of data based on the mutual deviation between the evidence support coefficients of each modality of data and the evidence support coefficients of all other modalities; analyze and determine the reference weight between each modality of data and each disease category based on the association strength and the modality consistency; and use the reference weight to correct the confidence scores to obtain the game weight for each candidate triple. The graph construction module is used to determine whether different candidate triples can be used to construct a multimodal knowledge graph based on the numerical magnitude of the game weights.

Images