Hysteroscopy lesion auxiliary diagnosis and treatment information processing method based on multi-modal graph contrast learning

Abstract

The application discloses a hysteroscopy lesion auxiliary diagnosis and treatment information processing method based on multi-modal graph contrast learning, and relates to the technical field of medical information processing; the method acquires hysteroscopy images, traditional Chinese medicine four diagnostic information and clinical phenotype data of reference patient samples and patients to be evaluated to construct original graphs, comparison graphs and test graphs, and the original graphs, the comparison graphs and the test graphs form first double-layer graphs and second double-layer graphs. A double-layer random walk algorithm is used to generate multi-view subgraphs of the comparison graph branches and the test graph branches. A cross-contrast learning model is used to extract multi-view embeddings to calculate and fuse the consistency scores of the two branches. In the reasoning stage, the multi-view subgraphs of the patient to be evaluated are compared with the instances of the low-risk and high-risk reference instance sets respectively, the disease score is generated according to the difference between the high-risk and low-risk reference similarity scores, and endometrial lesion risk prompt information is output. The application realizes the correlation representation and auxiliary risk prompt of multi-source heterogeneous medical data of traditional Chinese medicine and western medicine.

Description

Technical Field

[0001] This invention relates to the field of medical information processing and artificial intelligence, and in particular to a method for processing hysteroscopic lesion auxiliary diagnosis and treatment information based on multimodal graph contrastive learning. Background Technology

[0002] Endometrial lesions are common diseases of the female reproductive system. However, current clinical practice heavily relies on physician experience to identify lesions from relevant medical data, leading to difficulties in differentiating early-stage cancer from dysplasia. Furthermore, the information from the four diagnostic methods of traditional Chinese medicine (inspection, auscultation, inquiry, and palpation) is fragmented with Western hysteroscopic imaging data, failing to effectively integrate relevant medical data to form auxiliary diagnostic information. With the rapid development of artificial intelligence and multimodal medical imaging technology, they have shown great potential in the clinical management of gynecological diseases. In addition, existing AI-based auxiliary models often rely on single-modal data, exhibiting poor generalization ability and lacking the ability to mine deep correlations between multi-source heterogeneous data.

[0003] Graph neural networks, as a powerful tool for processing relational data, have demonstrated their potential for large-scale processing of medical data in the medical field. However, there are still the following technical challenges in directly applying them to the processing of medical data on multimodal endometrial lesions in both traditional Chinese and Western medicine: (1) How to effectively construct a graph structure that can simultaneously express the complex relationships between Western medicine images, traditional Chinese medicine syndrome elements, and clinical phenotypes; (2) How to design a learning mechanism that enables the model to autonomously mine cross-modal feature representations that are beneficial to diagnosis and treatment without relying on a large amount of labeled data; (3) How to improve the generalization performance and robustness of the model on data collected from different medical institutions and different equipment.

[0004] In summary, current practices in endometrial disease treatment face problems such as "subjectivity in Western medicine diagnosis and treatment, insufficient objectivity in traditional Chinese medicine diagnosis and treatment, poor multimodal fusion and interpretability of AI models, and insufficient connection with clinical application scenarios." There is an urgent need to develop an auxiliary diagnostic and treatment information processing method that can integrate multimodal data from both Western and traditional Chinese medicine and be used throughout the entire process of outpatient screening, preoperative or intraoperative auxiliary judgment, and postoperative follow-up management. Summary of the Invention

[0005] The purpose of this invention is to provide a method for processing auxiliary diagnostic and treatment information of hysteroscopic lesions based on multimodal graph contrast learning, which combines multimodal features of traditional Chinese and Western medicine with two-layer multi-view contrast learning technology to achieve auxiliary diagnostic and treatment information processing of endometrial lesions.

[0006] The technical solution adopted in this invention is:

[0007] A method for processing hysteroscopic lesion auxiliary diagnostic information based on multimodal graph contrastive learning, executed by a computing device, includes the following:

[0008] Hysteroscopic images, TCM diagnostic information, and clinical phenotypic data of reference patient samples are obtained, as well as hysteroscopic images, TCM diagnostic information, and clinical phenotypic data of patients to be evaluated. All data are preprocessed to form a standardized multimodal dataset. The reference patient samples are used for model training, model validation, or to construct a reference instance set, and the patients to be evaluated are those for whom lesion risk warning information is to be generated during the model inference stage.

[0009] The original image was constructed based on the comprehensive similarity of hysteroscopic image features and clinical phenotype features between the reference patient sample and the patients to be evaluated. A comparison chart was constructed based on the matching relationship between TCM syndrome features and hysteroscopic image features. A test graph was constructed based on the interaction between clinical phenotypic characteristics and hysteroscopic lesion risk signals. ; the original image Comparison Charts Test chart Construct the first two-layer graph by connecting cross-layer edges. Second double-layer diagram This leads to the formation of a multimodal graphical network model for traditional Chinese and Western medicine.

[0010] A two-layer random walk algorithm is used to perform two separate random walks in the first two-layer graph. Second double-layer diagram In this process, local sub-image sampling is performed with the node of the patient to be evaluated as the center to generate multi-view sub-images; the multi-view sub-images include the original image sub-image, the comparison image branch enhancement sub-image, the test image branch enhancement sub-image, the cross-layer sub-image between the original image and the comparison image, and the cross-layer sub-image between the original image and the test image.

[0011] The multi-view subgraph is input into the trained cross-contrast learning model; the cross-contrast learning model obtains node embeddings and subgraph global embeddings through a shared graph neural network encoder, and calculates the multi-view subgraphs corresponding to the comparison graph branch and the test graph branch, respectively. Figure 1 Consistency scores are calculated, and the consistency scores corresponding to the two branches are merged.

[0012] During the model inference phase, a low-risk reference instance set and a high-risk reference instance set are pre-established based on reference patient samples. The multi-view subgraph corresponding to the patient to be evaluated is compared with the reference instances in the low-risk reference instance set and the high-risk reference instance set in multiple rounds to obtain the low-risk reference similarity score and the high-risk reference similarity score. The lesion score of the patient to be evaluated is generated based on the difference between the high-risk reference similarity score and the low-risk reference similarity score.

[0013] Endometrial lesion risk warning information is generated based on the lesion score; in one embodiment, a TCM syndrome tendency warning can be generated based on a preset syndrome element matching rule, or a follow-up review suggestion warning can be generated based on a preset follow-up review suggestion mapping rule; the warning information is used to provide doctors with auxiliary decision-making reference, and is not directly used as a final clinical diagnosis conclusion or treatment prescription.

[0014] Furthermore, the implementation of preprocessing to form a standardized multimodal dataset includes:

[0015] Hysteroscopic images are standardized and enhanced, TCM diagnostic information is quantified and encoded to generate TCM syndrome features, and clinical phenotypic data are cleaned and normalized to form a standardized multimodal dataset.

[0016] Furthermore, the construction steps of the multimodal graphical network model of traditional Chinese and Western medicine include:

[0017] The original image was constructed based on the comprehensive similarity of hysteroscopic image features and clinical phenotype features between the reference patient sample and the patients to be evaluated. .in, This represents the set of nodes in the original graph. Represents the set of edges of the original graph. The set of edge weights represents the original graph; each patient corresponds to a patient node in the original graph; the features of the patient nodes in the original graph are composed of hysteroscopic image features, TCM syndrome features, and clinical phenotype features.

[0018] The similarity of hysteroscopic image features between patient nodes is represented as follows:

[0019] ;

[0020] The clinical phenotype similarity between patient nodes is represented as follows:

[0021] ;

[0022] The edge weights of the original graph are represented as follows:

[0023] ;

[0024] in, This is a scaling parameter for clinical phenotype distance; 0、 ,and + =1. When When the connection threshold is not lower than the preset threshold, at the patient node With patient nodes Establish the original graph edges between them.

[0025] A comparison graph was constructed based on the matching relationship between TCM syndrome features and hysteroscopic image features between patient nodes. .in, This represents the set of nodes in the comparison graph. Denotes the set of edges in the comparison graph. The comparison graph represents the set of edge weights; the comparison graph and the original graph are constructed based on the same set of patients; the patient node features in the comparison graph are the same as those in the original graph. Figure 1 Multimodal fusion node representation.

[0026] For any two patient nodes and Based on preset comparison rules, binary matching results corresponding to multiple comparison indicators are obtained. The edge weights of the comparison graph are determined based on the mean of the binary matching results.

[0027] ;

[0028] Where M represents the total number of comparison indicators; Represents patient node With patient nodes The binary matching result on the k-th comparison index.

[0029] A test graph was constructed based on the clinical phenotypic similarity between patient nodes and the interaction of hysteroscopic lesion risk signals. .in, This represents the set of nodes in the test graph. This represents the set of edges in the test graph. The test graph represents the set of edge weights; the test graph and the original graph are constructed based on the same set of patients; the patient node features in the test graph are the same as those in the original graph. Figure 1 A multimodal fusion node representation. For the patient node... With patient nodes The signal matching degree is calculated using preset signal matching rules. The edge weights of the test graph are determined based on the weighted results of the clinical phenotype similarity item and the signal matching item.

[0030] ;

[0031] in, This is a scaling parameter for clinical phenotype distance; , ,and .

[0032] Original image Comparison Chart The first double-layer diagram is formed by connecting the cross-layer edges. This is used to characterize the matching relationship between hysteroscopic image features and TCM syndrome feature features; the original image... With test chart A second double-layer diagram is formed by connecting the cross-layer edges. It is used to characterize the interaction between clinical phenotypic features, hysteroscopic lesion features and lesion risk signals; the cross-layer edges are used to connect corresponding nodes of the same patient in different layers; the similarity between different patients is characterized by the intra-layer edges in the corresponding layers, thereby forming a multimodal graph network model of traditional Chinese and Western medicine.

[0033] Furthermore, the implementation of generating multi-view subgraphs using a two-layer random walk algorithm includes:

[0034] In the first double-layer diagram Second double-layer diagram Random walk sampling is performed. Let... This indicates the patient to be evaluated in the original image. The corresponding central node, This represents the current patient node during the random walk. Indicates the patient node to be transferred next; let... Indicates the layer where the current node is located. This indicates the target layer for the next step.

[0035] Random walk from state Transition to state The probability is expressed as:

[0036] ;

[0037] Where r is the restart probability, used to control the random walk to return to the central patient node in the original graph. The probability of; This is the inter-layer switching coefficient, used to control the ratio between intra-layer expansion and inter-layer switching during random walks; Represents patient node In the current layer The set of first-order neighborhood nodes in the; Represents patient node With patient nodes Edge weights in the current layer. Represents patient node The corresponding node for the same patient in another layer;

[0038] Based on the aforementioned transition probabilities, in the original graph Medium sampling Generate a subgraph of the original graph from each of the first-order neighbor nodes. In the comparison chart and test chart Sampled separately Generate a comparison graph branch enhancement subgraph from each of the first-order neighbor nodes. and test graph branch enhancement subgraph Cross-layer sampling is performed between the original and comparison maps, and between the original and test maps, to generate cross-layer sub-maps. and The original image subgraph, the comparison image branch enhancement subgraph, the test image branch enhancement subgraph, and the two types of cross-layer subgraphs are collectively used as multi-view subgraphs.

[0039] Furthermore, the training process of the cross-comparison learning model includes:

[0040] During the model training phase, the shared graph neural network encoder of the learning model is cross-comparison to obtain the node embeddings in the multi-view subgraphs corresponding to the patients in the training samples; the node embeddings are multimodal representations obtained by joint encoding of hysteroscopic image features, TCM syndrome features and clinical phenotypic features.

[0041] The global representation of the subgraph is obtained by aggregating the hysteroscopic image features, TCM syndrome features, and clinical phenotype features in the subgraph through the readout module of the cross-comparison learning model.

[0042] The similarity calculation module of the cross-comparison learning model is used to calculate the similarity of the first two-layer graph. The generated comparison graph branch multi-view subgraph, and the second double-layer graph The generated test graph branches into multiple view subgraphs, calculates various consistency scores between node embeddings, between node embeddings and subgraph global embeddings, and between different subgraph global embeddings, and performs weighted fusion of the consistency scores corresponding to the two two-layer graph branches;

[0043] Based on the fused consistency scores, a contrastive learning objective function is constructed, which includes node-node-node contrastive loss, node-subgraph-subgraph contrastive loss, and subgraph-subgraph-subgraph contrastive loss. The parameters of the cross-contrast learning model are then optimized and trained.

[0044] Furthermore, for any branch Calculate the node-node-node consistency score and the node-subgraph-subgraph consistency score respectively. Figure 1 Consistency score and subgraph-subgraph-subgraph Figure 1 Consistency score.

[0045] The node-to-node consistency score for any branch d is represented as:

[0046] ;

[0047] Node-Subgraph-Sub Figure 1 Consistency score is expressed as:

[0048] ;

[0049] Subgraph-Subgraph-Sub Figure 1 Consistency score is expressed as:

[0050] ;

[0051] The consistency scores of the two bi-level graph branches are merged using a weighted average:

[0052] ,

[0053] in, , , These are the learnable weight matrices calculated for node-level, node-subgraph-level, and subgraph-level similarity, respectively. It is a logical function; , , These are the node embeddings corresponding to the original subgraph, the enhanced subgraph, and the cross-layer subgraph, respectively. , , These are global embeddings of subgraphs in the original subgraph, the enhanced subgraph, and the cross-layer subgraph, respectively. Where d=m represents the graph comparison branch and d=t represents the test graph branch; Used to measure whether the individual lesion characteristics of the same patient are consistent in different views. Used to measure whether there is a match between the patient node being evaluated and its lesion-related neighborhood. This is used to measure the consistency of local case population patterns expressed by different lesion-related subgraphs. The consistency score is used to characterize the degree of matching between patient multi-view features and between patient nodes and local case neighborhoods. During the model training phase, multi-view features under different risk states are learned through positive and negative instance pairs. Figure 1 Consistency representation: During the model inference phase, the risk of lesions is not determined directly based on a single consistency score, but rather based on the similarity differences between the patient to be evaluated and the high-risk reference instance set and the low-risk reference instance set to generate a lesion score.

[0054] Furthermore, the lesion score calculation includes:

[0055] During the model training phase, training instance pairs are constructed based on known pathological diagnoses or risk levels in the training samples. During the model inference phase, low-risk reference instance sets and high-risk reference instance sets are pre-established based on the training set or validation set. The patient to be evaluated is only compared with samples in the reference instance sets in multiple rounds, without using the pathological diagnoses of the patient to be evaluated as input.

[0056] Let Q low-risk reference instances and Q high-risk reference instances be sampled in each round. The average similarity score between the patient to be evaluated and the low-risk reference instances in the r-th round is expressed as:

[0057] ;

[0058] The average similarity score between the patient to be evaluated and the high-risk reference instance in round r is represented as:

[0059] ;

[0060] in, This represents the q-th low-risk reference instance obtained from the r-th round of sampling; This represents the q-th high-risk reference instance obtained from the r-th round of sampling; This represents the predicted similarity output by the cross-comparison learning model.

[0061] Patient nodes to be evaluated The lesion score is expressed as:

[0062] ;

[0063] Where R represents the number of sampling rounds for the reference instance; This represents the average similarity score between the patient to be evaluated and the high-risk reference instance in round r; The lesion score represents the average similarity score between the patient to be evaluated and the low-risk reference instance in the r-th round; the lesion score increases when the average similarity between the patient to be evaluated and the high-risk reference instance is higher than the average similarity between the patient and the low-risk reference instance.

[0064] Furthermore, the method also includes comparing the lesion score with a preset threshold to generate lesion risk warning information.

[0065] when ≥ At that time, it generates a higher risk warning message for endometrial lesions; when < At that time, a low risk of lesion is generated; among them, The prompt information is used to provide doctors with auxiliary decision-making references, either as a preset threshold or an auxiliary judgment threshold determined based on a validation set.

[0066] The present invention adopts the above technical solution and has the following beneficial effects compared with the prior art:

[0067] (1) This invention constructs a two-layer network model of “original graph-comparison graph-test graph”, which encodes the features of hysteroscopic images, the quantified TCM syndrome features and the standardized clinical phenotype features into graph nodes and edge weights. This realizes the deep integration and representation of multimodal data in a unified topological structure, enabling the system to comprehensively depict the patient’s condition from multiple dimensions of “image-syndrome-phenotype”, and significantly improves the comprehensive utilization value of medical data.

[0068] (2) This invention introduces a graph contrastive learning framework. By designing three layers of contrastive loss—"node-node-node," "node-subgraph-subgraph," and "subgraph-subgraph-subgraph"—it can learn the consistency relationships between different views under conditions of limited annotation or weak supervision. During the inference stage, it outputs lesion risk prompts based on a pre-constructed set of reference instances, and mines cross-modal feature representations that are beneficial for lesion identification. This invention reduces the dependence on a large amount of accurate pathological annotation data, making the model more adaptable to data from different sources and standards, and improving its robustness and universality in cross-institutional applications.

[0069] (3) This invention generates patient-centered original image sub-images, comparison image branch enhancement sub-images, test image branch enhancement sub-images, and two types of cross-layer sub-images through a two-layer random walk, which can reveal a group of cases that are similar to the patient to be evaluated in terms of imaging manifestations, TCM syndrome elements, or clinical risks; and forms a lesion score by comparing the similarity between the multi-view representation of the patient to be evaluated and high-risk and low-risk reference instances, thereby forming an auxiliary decision-making basis that is easy for doctors to understand.

[0070] (4) This invention quantifies and encodes the four diagnostic methods of traditional Chinese medicine (TCM) – observation, auscultation, inquiry, and palpation – to generate computable “syndrome element features”, and performs graph-structure association and comparative analysis with Western medical imaging and clinical indicators to achieve information processing that allows TCM and Western medical information to be mutually referenced, enabling TCM diagnostic experience to be deeply integrated with modern medical diagnosis and treatment in a data-driven manner.

[0071] This invention further integrates with the actual diagnosis and treatment process of endometrial lesions, and can be applied to scenarios such as outpatient screening, preoperative or intraoperative auxiliary judgment, and postoperative follow-up management, providing doctors with auxiliary decision-making information such as risk warning information for endometrial lesions, TCM syndrome tendency indications, and follow-up recommendations. Attached Figure Description

[0072] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments;

[0073] Figure 1 This is a flowchart illustrating the hysteroscopic lesion auxiliary diagnosis and treatment information processing method based on multimodal graph contrast learning according to the present invention.

[0074] Figure 2This is a schematic diagram illustrating the principle architecture of the hysteroscopic lesion auxiliary diagnosis and treatment information processing method based on multimodal graph contrastive learning according to the present invention. Detailed Implementation

[0075] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0076] With the continuous advancement of modern medical diagnostic and treatment technologies, hysteroscopy combined with the four diagnostic methods of traditional Chinese medicine has become a key means of diagnosing and treating endometrial lesions. However, due to the multi-source nature of data, including different individual patient characteristics, differences in hysteroscopic equipment, and the subjectivity of data collection in the four diagnostic methods of traditional Chinese medicine, traditional intelligent diagnostic and treatment models often encounter performance degradation in new clinical scenarios, a problem known as multimodal data heterogeneity or cross-institutional adaptation.

[0077] This invention represents the patient's endometrial lesions and TCM syndrome information as a multimodal two-layer topological graph of TCM and Western medicine (e.g., Figure 2 As shown in the figure, each patient is considered as a node, and its feature vector is generated from multimodal data. Visual features such as lesion color, shape, and boundary are extracted from hysteroscopic images, combined with the syndrome element scores (such as blood stasis syndrome and qi stagnation syndrome scores) obtained by the four diagnostic methods of traditional Chinese medicine (inspection, auscultation, inquiry, and palpation) through the "Traditional Chinese Medicine Health Status Identification System", and then integrated with clinical phenotypic data such as age and medical history. The edges in the topological graph are used to represent the multimodal association between patient nodes. The edge weights in the original graph are used to represent the comprehensive similarity between hysteroscopic image features and clinical phenotypic features. The edge weights in the comparison graph are used to represent the matching relationship between TCM syndrome element features and hysteroscopic image features. The edge weights in the test graph are used to represent the comprehensive correlation between clinical phenotypic similarity and the interaction relationship of hysteroscopic lesion risk signals. Building upon this, a cross-modal association representation in the multi-view subgraphs is extracted using a cross-contrast learning model: First, a two-layer random walk algorithm is used to generate the original subgraph centered on the node of the patient to be evaluated, the comparison graph branch-enhanced subgraph, the test graph branch-enhanced subgraph, and two types of cross-layer subgraphs; then, a shared graph neural network encoder is used to obtain the node embeddings and the global embeddings of the subgraphs, and the multi-view representations corresponding to the comparison graph branch and the test graph branch are calculated respectively. Figure 1Consistency scores are further fused between the two branches to enable the model to learn cross-modal relationships between hysteroscopic image features, TCM syndrome features, and clinical phenotypic features. During the model inference phase, the multi-view sub-image corresponding to the patient to be evaluated is compared with reference instances in both the low-risk and high-risk reference instance sets. A lesion score is generated based on the difference between the high-risk and low-risk reference similarity scores, which is used to output endometrial lesion risk warning information. This method transforms a direct classification problem into a similarity comparison process based on reference instances, helping to reduce reliance on large amounts of precise pathological label data and improving the interpretability of auxiliary risk warnings in scenarios involving the fusion of TCM and Western medicine multimodal data.

[0078] like Figure 1 As shown in Figure 2, this invention specifically discloses a method for processing hysteroscopic lesion auxiliary diagnostic and treatment information based on multimodal graph contrastive learning, which includes the following steps:

[0079] Step 1, Multimodal medical data acquisition and preprocessing: Collect hysteroscopic image data, TCM four diagnostic information and clinical phenotype data from different institutions, and perform standardization, enhancement, quantitative coding, cleaning and normalization on all data to form a standardized TCM and Western medicine multimodal dataset.

[0080] Step 2, Construction of the Multimodal Graph Model of Traditional Chinese and Western Medicine: Construct the original graph based on the multimodal dataset. ,in This represents the set of nodes in the original graph, corresponding to individual patients. The set of edges in the original graph represents the comprehensive similarity between patients based on hysteroscopic image features and clinical phenotypic features. The set of edge weights in the original graph represents the comprehensive connectivity strength obtained by weighting the similarity terms of hysteroscopic image features and clinical phenotypes. A comparison graph is constructed based on the matching relationship between TCM syndrome features and hysteroscopic image features. A test map was constructed based on the signal interaction relationship between clinical phenotype and hysteroscopic lesion characteristics. ; the original image Comparison Chart Construct the first two-layer graph by connecting cross-layer edges. , the original image With test chart Construct a second two-layer graph by connecting cross-layer edges. ;

[0081] Step 3, sampling of multi-view lesion-related sub-images for the patient to be evaluated: A two-layer random walk algorithm is used to sample the lesion-related sub-images in the first two-layer image. Second double-layer diagram In the process, local sub-map sampling is performed centered on the patient node to be evaluated to generate multi-view lesion-related sub-maps, including those from the original map. Original image, sub-image, from comparison image Or test chart Enhanced subgraphs, and across the original graph Comparison Chart Or test chart Cross-layer subgraph;

[0082] Step 4, Training of a graph contrastive learning model for hysteroscopic lesion risk representation: Construct a cross-contrast learning model based on graph neural networks. Input multi-view lesion-related subgraphs into a shared graph neural network encoder to generate patient node embeddings and lesion-related subgraph embeddings. Aggregate global features of subgraphs through the readout module, calculate consistency scores of the comparison graph branch and the test graph branch through the similarity module, and perform weighted fusion of the consistency scores of the two branches. Train the model by combining three contrastive loss functions: node-node-node, node-subgraph-subgraph, and subgraph-subgraph-subgraph.

[0083] Step 5: Calculate the lesion score of the patient node to be evaluated based on the trained model: During the model inference stage, a low-risk reference instance set and a high-risk reference instance set are established in advance based on the training set or validation set; For the patient node to be evaluated, multiple rounds of reference instance sampling are performed from the low-risk reference instance set and the high-risk reference instance set respectively. The multi-view subgraph corresponding to the patient to be evaluated and the sampled reference instances are input into the trained cross-comparison learning model to obtain the predicted similarity score. The lesion score is generated by the mean difference between the high-risk reference similarity score and the low-risk reference similarity score.

[0084] Step 6: Input the hysteroscopic images, TCM four diagnostic methods information, and clinical phenotype data of the patient to be evaluated into the trained model. According to the edge weight calculation rules of the original graph, comparison graph, and test graph, establish a connection between the node of the patient to be evaluated and the node of the reference patient. Perform multi-view subgraph sampling and lesion score calculation with the node of the patient to be evaluated as the center. Generate endometrial lesion risk warning information based on the lesion score to complete the processing of multimodal medical data of TCM and Western medicine.

[0085] Furthermore, the specific steps of step 1 are as follows:

[0086] Step 1-1, Multimodal Medical Data Acquisition and Preprocessing: Hysteroscopic Images: Images of patients undergoing hysteroscopy were acquired using a high-definition hysteroscopic system (resolution ≥1920×1080). Samples included non-lesion or low-risk patients, as well as core lesion types such as endometrial polyps, endometrial hyperplasia, and endometrial cancer; multiple hysteroscopic images from different angles were acquired for each patient; Traditional Chinese Medicine Four Diagnostic Methods Information: Observation (facial complexion, tongue coating) and auscultation (breath) were recorded using the "Four Diagnostic Methods Information Collection Form" (referring to the four diagnostic methods listed in the 10th edition of "Traditional Chinese Medicine Diagnostics"). The data includes four categories: discharge odor, menstrual cycle, dysmenorrhea severity, and pulse type (wiry / slippery / thready pulse). Each category records 3-5 key indicators. Clinical phenotype data includes patient age, medical history (duration of endometrial disease: 0-20 years), weight (calculated body mass index: weight / height², unit kg / m²), color Doppler ultrasound results (endometrial thickness, lesion blood flow signal), and pathological diagnosis results. These serve as supervision labels during the model training phase or as the gold standard during the validation phase, but are not used as input data for the inference phase of the patients to be evaluated.

[0087] Steps 1-2, Data Cleaning and Standardization: Median imputation was used to handle missing values ​​in clinical phenotypes (e.g., missing body mass index was imputed using the overall median of 23.5) and the four diagnostic methods of Traditional Chinese Medicine (TCM) (e.g., missing pulse values ​​were imputed using "wiry pulse," which had the highest proportion). Samples with a missing value rate >20% were directly removed. Outliers were identified and removed through box plots (IQR=1.5) and manual review, such as abnormal hysteroscopic image quality (massive bleeding causing lesions to be invisible, or blank images without lesions). TCM diagnostic information was entered into the TCM syndrome element quantification module to generate 8-dimensional syndrome element features (qi stagnation, blood stasis, etc., each syndrome element scored from 0-100 points) and 3 syndrome types (liver qi stagnation transforming into fire, qi deficiency, phlegm dampness). Data enhancement was performed on the hysteroscopic images (rotation angle ±15°, histogram equalization (grayscale range 0-255), random cropping (cropping ratio 0.8-1.0)), increasing the number of images by 3 times after enhancement. The final standardized multi-source dataset consists of "hysteroscopic images (224×224) + TCM syndrome elements (8 dimensions) + clinical phenotype (5 dimensions)," which is divided into training set, validation set and test set in a 7:2:1 ratio.

[0088] Furthermore, the specific steps of step 2 are as follows:

[0089] Step 2-1, Define the original graph Each patient is defined as a patient node in the original graph. The feature vector of each patient node is formed by fusing the TCM syndrome element quantification vector, the clinical phenotype standardized vector, and the hysteroscopic image feature vector. For any two patient nodes... and Calculate the feature similarity of hysteroscopic images respectively. Similarity to clinical phenotype And calculate the edge weights of the original graph according to the weighting method. .when When the connection threshold is not lower than the preset threshold, at the patient node and Original graph edges are established between patients to form an original graph topology that characterizes the comprehensive similarity between hysteroscopic image features and clinical phenotypic features among patients.

[0090] Step 2-2, Construct a comparison chart A comparison graph was constructed based on the matching relationship between TCM syndrome features and hysteroscopic image features between patient nodes. The comparison graph and the original graph are constructed based on the same set of patients, with each patient corresponding to a patient node in the comparison graph.

[0091] In one implementation, four TCM syndrome element comparison indicators and four hysteroscopic image comparison indicators are selected, for a total of M=8 comparison indicators. For any two patient nodes... and The binary matching result is obtained through preset comparison rules:

[0092] ;

[0093] Comparison of patient nodes in the graph With patients The edge weights between them are represented as follows:

[0094] ;

[0095] Specifically, for the TCM syndrome element comparison index, when the difference in scores of corresponding syndrome elements does not exceed the first preset threshold, the binary matching result is 1; for the hysteroscopic image comparison index, when the similarity of quantitative features of corresponding images is not lower than the second preset threshold, the binary matching result is 1; otherwise, the binary matching result is 0.

[0096] The comparison index is only used to calculate the edge weights of the comparison graph and determine its topology. To ensure that the subgraphs corresponding to the original graph, comparison graph, and test graph can be input into the same shared graph neural network encoder, the node features in the comparison graph adopt the same characteristics as those in the original graph. Figure 1 Multimodal fusion node representation:

[0097] ;

[0098] in, Indicates the characteristics of hysteroscopic images. Indicates the characteristics of TCM syndrome elements. Indicates clinical phenotypic characteristics.

[0099] Steps 2-3: Construct the test graph A test graph was constructed based on the clinical phenotypic similarity among patients and the interaction of hysteroscopic lesion risk signals. Each patient corresponds to a patient node in the test graph; the features of the patient nodes in the test graph are compared with those in the original graph. Figure 1 Multimodal fusion node representation:

[0100] ;

[0101] For patient nodes With patient nodes The candidate lesion risk representation generated by the hysteroscopic image feature extraction module is used as the input signal, and the corresponding TCM syndrome element status and clinical phenotype risk status of the patient are used as the response signal. The signal matching degree is calculated through preset signal matching rules. When the input signal and the response signal meet the preset strong matching condition, When partial matching conditions are met, When the matching conditions are not met, .

[0102] Patient node in the test graph With patient nodes The edge weights between them are represented as follows:

[0103] ;

[0104] in, >0 represents the clinical phenotype distance scaling parameter; , ,and + =1.

[0105] During the model training phase, pathological diagnosis results or risk levels are only used to construct training instance pairs, establish low-risk reference instance sets and high-risk reference instance sets, and perform model validation; during the model inference phase, the pathological diagnosis results of the patients to be evaluated do not participate in the calculation of test graph node features or edge weights.

[0106] Steps 2-4: Construct a two-layer diagnostic and treatment map for hysteroscopic lesion auxiliary diagnosis and treatment: [The original map is then...] Comparison Chart The first two-layer graph is formed by connecting the layers with cross-layer edges. This is used to characterize the matching relationship between hysteroscopic lesion image features and TCM syndrome element features of the patient to be evaluated; the original image... With test chart A second two-layer graph is formed by connecting the layers with cross-layer edges. This is used to characterize the interaction between the clinical phenotype, hysteroscopic lesion characteristics, and lesion risk signals of the patient to be evaluated. The cross-layer edges are used to connect corresponding nodes of the same patient in the original and comparison images, as well as connecting corresponding nodes of the same patient in the original and test images. The similarity between different patients is characterized by the intra-layer edges in the original, comparison, and test images, thereby forming a multimodal diagnostic and treatment image structure that can simultaneously reflect "hysteroscopic image characteristics - TCM syndrome characteristics - clinical lesion risk" in both Chinese and Western medicine.

[0107] Furthermore, the specific steps of step 3 are as follows:

[0108] Step 3-1, define the two-layer random walk probability for hysteroscopic lesion-assisted diagnosis and treatment:

[0109] In the first double-layer diagram Second double-layer diagram Random walk sampling is performed; let... This indicates the patient to be evaluated in the original image. The corresponding central node, This represents the current patient node during the random walk. Indicates the patient node to be transferred next; let... Indicates the layer where the current node is located. Indicates the next target layer;

[0110] Random walk from state ( )Transfer to state( The probability of ) is expressed as:

[0111] ;

[0112] Where r is the restart probability, used to control the random walk to return to the central patient node in the original graph. The probability of; This is the inter-layer switching coefficient, used to control the ratio between intra-layer expansion and inter-layer switching during random walks; Represents patient node In the current layer The set of first-order neighborhood nodes in the; Represents patient node With patient nodes Edge weights in the current layer; Represents patient node On another layer The same patient corresponding node in the data;

[0113] Step 3-2, the method for generating multi-view subgraphs is as follows:

[0114] Sub-image of the original image: within the original image In the middle, the corresponding nodes of the patients to be evaluated Centered on the original graph, sample according to the one-step transition probability. Given a set of first-order neighbor patient nodes, the node set of the original graph subgraph is represented as:

[0115] ;

[0116] The corresponding subgraph of the original image is represented as follows: ;in, In the original diagram From the patient node The one-step transition probability to its first-order neighbor node.

[0117] Enhanced subgraph: in comparison graph and test chart In the middle, the corresponding nodes of the patients to be evaluated are respectively and Centered on the data, sample according to the one-step transition probability in the corresponding layer. There are one first-order neighbor patient nodes. The node sets of the comparison graph branch enhancement subgraph and the test graph branch enhancement subgraph are represented as follows:

[0118] ;

[0119] The corresponding enhancement subgraph is represented as:

[0120] ;

[0121] in, Comparison chart The one-step transition probability in; Represents test graph The one-step transition probability in the equation.

[0122] Cross-level subgraph: in the original graph Comparison Chart Between, and the original diagram With test chart Cross-layer sampling is performed between each layer. The sets of nodes in the cross-layer subgraphs are represented as follows:

[0123] ;

[0124] ;

[0125] The corresponding cross-level subgraph is represented as follows:

[0126] , ;

[0127] in, Represents patient node In the comparison graph, the second-order neighborhood set; Represents patient node The second-order neighborhood set in the test graph; and Let represent the two-step transition probabilities in the comparison graph and the test graph, respectively.

[0128] In one implementation, the following can be set:

[0129] ;

[0130] Furthermore, the specific steps of step 4 are as follows:

[0131] Step 4-1, Lesion-related feature encoding module:

[0132] For the first double-layer diagram Sub-images of the original image Comparison graph branch enhancement subgraph Cross-layer subgraphs between the original graph and the comparison graph The set of sub-graphs comprising the branch three-view diagrams of the comparison diagram:

[0133] ;

[0134] For the second double-layer diagram Sub-images of the original image Test graph branch enhancement subgraph Cross-layer subgraphs between the original graph and the test graph The test graph branch three-view subgraph set:

[0135] ;

[0136] For any branch For any given subgraph, the initial node feature matrix adopts a unified multimodal fusion representation:

[0137] ;

[0138] in, This represents the feature matrix of the hysteroscopic image. Represents the TCM syndrome feature matrix. This represents the clinical phenotypic feature matrix.

[0139] Shared graph neural network encoder The node embedding update formula for the layer is:

[0140] ;

[0141] in, ; This is the adjacency matrix of the subgraph; It is the identity matrix; The degree matrix after adding self-loops; Here is the learnable weight matrix for the l-th layer; ( ) is a non-linear activation function.

[0142] After encoding by an L-layer graph neural network, the node embeddings of the patient node to be evaluated in the original graph subgraph, the enhanced graph subgraph, and the cross-layer subgraph are respectively represented as follows:

[0143] ;

[0144] Step 4-2, Lesion-related sub-map reading module: For any branch The node embeddings from the original subgraph, the enhanced subgraph, and the cross-layer subgraph are aggregated respectively to obtain the corresponding subgraph global embedding:

[0145] ;

[0146] For any patient node in a subgraph Its relative to the patient node to be evaluated The lesion-related aggregate weights are represented as follows:

[0147] + + ;

[0148] in, , , ,and:

[0149] ;

[0150] The global embedding of a subgraph is represented as:

[0151] ;

[0152] in, Represents patient node Node embeddings encoded by a shared graph neural network.

[0153] Step 4-3, Multi-view Figure 1 Consistency similarity calculation module: for any branch Calculate the node-node-node consistency score and the node-subgraph-subgraph consistency score respectively. Figure 1 Consistency score and subgraph-subgraph-subgraph Figure 1 Consistency score:

[0154] ;

[0155] ;

[0156] ;

[0157] The consistency scores of the two bi-level graph branches are merged using a weighted average:

[0158] ;

[0159] in, , ,and .

[0160] Step 4-4, Contrastive Learning Objective Function for Lesion Risk Differentiation: During the model training phase, positive and negative instance pairs are constructed. If two multi-view instances come from the same patient, or from patients with similar lesion risk levels or similar TCM syndrome types, they are labeled as a positive instance pair, denoted as... If two multi-view instances come from patients with significant differences in disease risk level or TCM syndrome type, they are marked as a negative instance pair, denoted as... .

[0161] For any fusion consistency score ,in Its binary contrastive loss function is expressed as:

[0162] ;

[0163] The total loss function is expressed as:

[0164] ;

[0165] in, , , These are the weighting coefficients for the three contrast losses. By minimizing the total loss function, the multi-view representations corresponding to similar lesion risk states are made closer to each other, while multi-view representations with significant differences in lesion risk states are made distinct from each other, thereby enhancing the model's ability to indicate lesion risk.

[0166] Furthermore, the specific steps of step 5 are as follows:

[0167] Step 5-1, Reference Instance Sampling: During the model inference phase, a low-risk reference instance set is pre-established based on the training set or validation set. and a collection of high-risk reference examples Among them, the low-risk reference instance set Includes multi-view sub-images of patients with pathological diagnoses of non-lesion or low risk, and a collection of high-risk reference examples. This includes multi-view subplots of patients with high-risk lesions diagnosed by pathology. For nodes of patients to be evaluated... R rounds of reference instance sampling are performed; in one implementation, R=10 is set, and Q=256 reference instances are sampled from the low-risk reference instance set and the high-risk reference instance set in each round.

[0168] Step 5-2, Similarity score prediction: Predict the similarity score of the patient node to be evaluated. The corresponding multi-view subgraphs are input into the cross-comparison learning model trained with the low-risk and high-risk reference instances obtained from the r-th round of sampling. The average similarity score between the patient to be evaluated and the low-risk reference instance in the r-th round is expressed as:

[0169] ;

[0170] The average similarity score between the patient to be evaluated and the high-risk reference instance in round r is represented as:

[0171] ;

[0172] Step 5-3, Lesion Score Calculation: Calculate the lesion score based on the average similarity difference between the patient to be evaluated and high-risk and low-risk reference instances.

[0173] ;

[0174] in, Indicates the node of the patient to be evaluated lesion score; Indicates the number of sampling rounds for the reference instance; This represents the average similarity score between the patient to be evaluated and the high-risk reference instance in round r; The score represents the average similarity score between the patient to be evaluated and the low-risk reference instance in round r; the lesion score increases when the average similarity between the patient to be evaluated and the high-risk reference instance is higher than the average similarity between the patient and the low-risk reference instance.

[0175] Furthermore, the specific steps of step 6 are as follows:

[0176] Step 6-1, Input patient multimodal diagnosis and treatment data: Obtain hysteroscopic images, TCM four diagnostic information and clinical phenotype data of the patient to be evaluated, and form the node features of the patient to be evaluated; Calculate the connection relationship between the node of the patient to be evaluated and the reference patient node according to the edge weight calculation rules of the original graph, comparison graph and test graph, and add the node of the patient to be evaluated to the corresponding graph structure; Perform multi-view subgraph sampling and lesion score calculation with the node of the patient to be evaluated as the center.

[0177] Step 6-2, Multimodal Integration Results: Output endometrial lesion risk warning information based on lesion scores. In one implementation, TCM syndrome tendency warnings can be further generated based on the TCM syndrome element scores of the patient to be evaluated and preset syndrome matching rules, and follow-up examination suggestions and intervention information for doctors' reference can be generated based on the lesion risk level and preset follow-up examination suggestion mapping rules. The warning information is used to provide doctors with auxiliary decision-making references and is not directly used as a final clinical diagnosis or treatment prescription.

[0178] Furthermore, as a feasible application, based on the stage of diagnosis and treatment of the patient to be evaluated, the lesion score and auxiliary diagnosis and treatment information can be applied to outpatient initial screening, preoperative or intraoperative auxiliary judgment, and postoperative follow-up management scenarios. In the outpatient initial screening scenario, lesion risk warning information is output to assist doctors in assessing whether further examination is needed. In the preoperative or intraoperative auxiliary judgment scenario, lesion risk warning information is output to assist doctors in determining key observation areas and sampling locations. In the postoperative follow-up management scenario, re-examination suggestions and follow-up intervention information for doctors' reference can be output based on the lesion risk level, TCM syndrome element score, and preset rules.

[0179] This invention employs the above technical solution, combining hysteroscopic image data, information from the four diagnostic methods of Traditional Chinese Medicine (TCM), and clinical phenotypic data to construct a multimodal, two-layer topological map of TCM and Western medicine that accurately reflects the characteristics of endometrial lesions. Based on this, the invention uses a cross-contrast learning model to extract cross-modal association representations from the multi-view subgraphs. During the model training phase, positive and negative instance pairs are used to train the model to learn multi-view representations under different risk states. Figure 1 In the model inference phase, the multi-view subgraph corresponding to the patient to be evaluated is compared with reference instances in pre-established low-risk and high-risk reference instance sets. A lesion score is generated based on the mean difference between the high-risk and low-risk reference similarity scores. This lesion score is used to generate lesion risk warning information, providing auxiliary decision-making references for doctors to further determine whether pathological examination, focused observation, or follow-up management is necessary. When the lesion score is not lower than a preset threshold, a high-risk endometrial lesion warning is generated; when the lesion score is lower than the preset threshold, a low-risk endometrial lesion warning is generated, thus providing auxiliary decision-making references for doctors to further determine whether pathological examination, focused observation, or follow-up management is necessary. This invention effectively transforms a direct classification problem into a lesion score comparison process based on reference instances, facilitating the generation of endometrial lesion risk warning information from multi-source medical data. This provides auxiliary decision-making references for preoperative assessment, intraoperative assistance, and postoperative follow-up, thereby improving the processing efficiency of endometrial lesion-related medical data.

[0180] The present invention adopts the above technical solution and has the following beneficial effects compared with the prior art:

[0181] (1) This invention constructs a two-layer network model of “original graph-comparison graph-test graph”, which encodes the features of hysteroscopic images, the quantified TCM syndrome features and the standardized clinical phenotype features into graph nodes and edge weights. This realizes the deep integration and representation of multimodal data in a unified topological structure, enabling the system to comprehensively depict the patient’s condition from multiple dimensions of “image-syndrome-phenotype”, and significantly improves the comprehensive utilization value of medical data.

[0182] (2) This invention introduces a graph contrastive learning framework. By designing three layers of contrastive loss—"node-node-node," "node-subgraph-subgraph," and "subgraph-subgraph-subgraph"—it can learn the consistency relationships between different views under conditions of limited annotation or weak supervision. During the inference stage, it outputs lesion risk prompts based on a pre-constructed set of reference instances, and mines cross-modal feature representations that are beneficial for lesion identification. This invention reduces the dependence on a large amount of accurate pathological annotation data, making the model more adaptable to data from different sources and standards, and improving its robustness and universality in cross-institutional applications.

[0183] (3) This invention generates patient-centered original image sub-images, comparison image branch enhancement sub-images, test image branch enhancement sub-images, and two types of cross-layer sub-images through a two-layer random walk, which can reveal a group of cases that are similar to the patient to be evaluated in terms of imaging manifestations, TCM syndrome elements, or clinical risks; and forms a lesion score by comparing the similarity between the multi-view representation of the patient to be evaluated and high-risk and low-risk reference instances, thereby forming an auxiliary decision-making basis that is easy for doctors to understand.

[0184] (4) This invention quantifies and encodes the four diagnostic methods of traditional Chinese medicine (TCM) – observation, auscultation, inquiry, and palpation – to generate computable “syndrome element features”, and performs graph-structure association and comparative analysis with Western medical imaging and clinical indicators to achieve information processing that allows TCM and Western medical information to be mutually referenced, enabling TCM diagnostic experience to be deeply integrated with modern medical diagnosis and treatment in a data-driven manner.

[0185] This invention further integrates with the actual diagnosis and treatment process of endometrial lesions, and can be applied to scenarios such as outpatient screening, preoperative or intraoperative auxiliary judgment, and postoperative follow-up management, providing doctors with auxiliary decision-making information such as risk warning information for endometrial lesions, TCM syndrome tendency indications, and follow-up recommendations.

[0186] Obviously, the described embodiments are only a portion, not all, of the embodiments of this application. Without conflict, the embodiments and features described and illustrated herein can be combined with each other. The components of the embodiments of this application generally described and illustrated in the accompanying drawings can be arranged and designed in various different configurations. Therefore, the detailed description of the embodiments of this application is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

Claims

1. A method for processing hysteroscopic lesion auxiliary diagnostic and treatment information based on multimodal graph contrastive learning, characterized in that, Performed by a computing device, the method includes: The process involves acquiring hysteroscopic images, TCM diagnostic information, and clinical phenotypic data of reference patient samples, as well as acquiring hysteroscopic images, TCM diagnostic information, and clinical phenotypic data of patients to be evaluated; preprocessing all data to form a standardized multimodal dataset; wherein, the reference patient samples include patient samples used for model training, validation, or construction of a reference instance set, and the patients to be evaluated are those for whom lesion risk warning information is to be generated during the model inference stage; The original image was constructed based on the comprehensive similarity between the hysteroscopic image features and clinical phenotypic features of the reference patient sample and the patient to be evaluated. A comparison chart was constructed based on the matching relationship between TCM syndrome features and hysteroscopic image features. A test graph was constructed based on the interaction between clinical phenotypic characteristics and hysteroscopic lesion risk signals. ; the original image Comparison Charts Test chart Construct the first two-layer graph by connecting cross-layer edges. Second double-layer diagram This leads to the formation of a multimodal graphical network model for traditional Chinese and Western medicine. A two-layer random walk algorithm is used to perform two separate random walks in the first two-layer graph. Second double-layer diagram In this process, local sub-image sampling is performed with the node of the patient to be evaluated as the center to generate multi-view sub-images; the multi-view sub-images include the original image sub-image, the comparison image branch enhancement sub-image, the test image branch enhancement sub-image, the cross-layer sub-image between the original image and the comparison image, and the cross-layer sub-image between the original image and the test image. The multi-view subgraph is input into the trained cross-contrast learning model; the cross-contrast learning model obtains node embeddings and subgraph global embeddings through a shared graph neural network encoder, calculates the multi-view consistency scores corresponding to the comparison graph branch and the test graph branch respectively, and fuses the consistency scores corresponding to the two branches. During the model inference phase, a low-risk reference instance set and a high-risk reference instance set are pre-established based on reference patient samples. The multi-view subgraph corresponding to the patient to be evaluated is compared with the reference instances in the low-risk reference instance set and the high-risk reference instance set in multiple rounds to obtain the low-risk reference similarity score and the high-risk reference similarity score. The lesion score of the patient to be evaluated is generated based on the difference between the two. Endometrial lesion risk warning information is generated based on the lesion score; the warning information is used to provide doctors with auxiliary decision-making references and is not directly used as a final clinical diagnosis or treatment prescription.

2. The method for processing hysteroscopic lesion auxiliary diagnostic and treatment information based on multimodal graph contrastive learning according to claim 1, characterized in that, The implementation of preprocessing to form a standardized multimodal dataset includes: Hysteroscopic images are standardized and enhanced, TCM diagnostic information is quantified and encoded to generate TCM syndrome features, and clinical phenotypic data are cleaned and normalized to form a standardized multimodal dataset.

3. The method for processing hysteroscopic lesion auxiliary diagnostic and treatment information based on multimodal graph contrastive learning according to claim 1, characterized in that, The steps for constructing a multimodal graphical network model of traditional Chinese and Western medicine are as follows: The original image was constructed based on the comprehensive similarity of hysteroscopic image features and clinical phenotype features between the reference patient sample and the patients to be evaluated. ,in, This represents the set of nodes in the original graph. Represents the set of edges of the original graph. The set of edge weights represents the original graph; each patient corresponds to a patient node in the original graph; the features of the patient nodes in the original graph are composed of hysteroscopic image features, TCM syndrome features, and clinical phenotype features; the edge weights of the original graph are determined based on the weighted results of the similarity terms of hysteroscopic image features and clinical phenotypes between patient nodes. A comparison graph was constructed based on the matching relationship between TCM syndrome features and hysteroscopic image features between patient nodes. ,in, This represents the set of nodes in the comparison graph. Denotes the set of edges in the comparison graph. The comparison graph represents the set of edge weights; the comparison graph and the original graph are constructed based on the same patient set; the patient node features in the comparison graph are represented using the same multimodal fusion node representation as the original graph; and binary matching results corresponding to multiple comparison indicators are obtained according to preset comparison rules. And determine the edge weights of the comparison graph based on the mean of the binary matching results; A test graph was constructed based on the clinical phenotypic similarity between patient nodes and the interaction of hysteroscopic lesion risk signals. ,in, This represents the set of nodes in the test graph. This represents the set of edges in the test graph. The test graph represents the set of edge weights; the test graph and the original graph are constructed based on the same set of patients; the patient node features in the test graph are represented using the same multimodal fusion node representation as the original graph; for patient nodes... With patient nodes The signal matching degree is calculated using preset signal matching rules. When the input signal and the response signal meet the preset strong matching condition, When partial matching conditions are met, When the matching conditions are not met, The edge weights of the test graph are determined based on the weighted results of the clinical phenotype similarity item and the signal matching item. Original image Comparison Chart The first double-layer diagram is formed by connecting the cross-layer edges. This is used to characterize the matching relationship between hysteroscopic image features and TCM syndrome feature features; the original image... With test chart A second double-layer diagram is formed by connecting the cross-layer edges. It is used to characterize the interaction between clinical phenotypic features, hysteroscopic lesion features and lesion risk signals; the cross-layer edges are used to connect corresponding nodes of the same patient in different layers; the similarity between different patients is characterized by the intra-layer edges in the corresponding layers, thereby forming a multimodal graph network model of traditional Chinese and Western medicine that simultaneously reflects the correlation between hysteroscopic image features, TCM syndrome features and clinical lesion risk.

4. The method for processing hysteroscopic lesion auxiliary diagnostic and treatment information based on multimodal graph contrastive learning according to claim 1, characterized in that, The implementation of generating multi-view subgraphs using a two-layer random walk algorithm includes: In the first double-layer diagram Second double-layer diagram Random walk sampling is performed; let... This indicates the patient to be evaluated in the original image. The corresponding central node, This represents the current patient node during the random walk. Indicates the patient node to be transferred next; let... Indicates the layer where the current node is located. Indicates the next target layer; Random walk from state ( )Transfer to state( The probability of ) is expressed as: ； Where r is the restart probability, used to control the random walk to return to the central patient node in the original graph. The probability of; This is the inter-layer switching coefficient, used to control the ratio between intra-layer expansion and inter-layer switching during random walks; Represents patient node In the current layer The set of first-order neighborhood nodes in the; Represents patient node With patient nodes Edge weights in the current layer; Represents patient node On another layer The same patient corresponding node in the data; Based on the transition probability, in the original graph Medium sampling Generate a subgraph of the original graph from each of the first-order neighbor nodes. In the comparison chart and test chart Sampled separately Generate a comparison graph branch enhancement subgraph from each of the first-order neighbor nodes. and test graph branch enhancement subgraph Cross-layer sampling is performed between the original and comparison maps, and between the original and test maps, to generate cross-layer sub-maps. and ; The original image subgraph, the comparison image branch enhancement subgraph, the test image branch enhancement subgraph, and the two types of cross-layer subgraphs are collectively used as multi-view subgraphs.

5. The method for processing hysteroscopic lesion auxiliary diagnostic and treatment information based on multimodal graph contrastive learning according to claim 4, characterized in that, Patient node With patient nodes Edge weight calculation in each layer includes: When the layer is the original image At that time, patient node With patient nodes The edge weights between them are denoted as The similarity of hysteroscopic image features between patient nodes is represented as follows: ； The clinical phenotype similarity between patient nodes is represented as follows: ； The edge weights of the original graph are represented as follows: ； in, and Representing patient nodes respectively and patient nodes Hysteroscopic image characteristics; and Representing patient nodes respectively and patient nodes Clinical phenotypic characteristics; This is a scaling parameter for clinical phenotype distance; 0、 ,and + =1; When the layer is a comparison image At that time, patient node With patient nodes The edge weights between them are denoted as This is used to represent the degree of matching between the TCM syndrome features and the intrauterine mirror image features between two patients. The calculation formula is as follows: ； Where M represents the total number of comparison indicators; Represents patient node With patient nodes The binary comparison result on the k-th comparison metric; when the k-th comparison metric meets the preset matching condition. ,otherwise ; When the layer is a test image At that time, patient node With patient nodes The edge weights between them are denoted as This is used to represent the comprehensive association between two patients in terms of clinical phenotypic similarity and the interaction of disease risk signals. The formula for its calculation is: ； in, Represents patient node With patient nodes The signal matching degree between the input signal and the response signal; when the input signal and the response signal meet the preset strong matching condition. When partial matching conditions are met, When the matching conditions are not met, ; >0 represents the clinical phenotype distance scaling parameter; 0、 ,and + =1; The original graph edge weights, comparison graph edge weights, and test graph edge weights are all non-negative values, used for subsequent multi-view subgraph sampling and graph neural network encoding processing.

6. The method for processing hysteroscopic lesion auxiliary diagnostic and treatment information based on multimodal graph contrastive learning according to claim 1, characterized in that, The training process of the cross-comparison learning model includes: The shared graph neural network encoder of the cross-comparison learning model obtains the node embeddings in the multi-view sub-graphs corresponding to the patients in the training samples; the node embeddings include the joint representations corresponding to hysteroscopic image features, TCM syndrome features and clinical phenotype features. The global representation of the subgraph is obtained by aggregating the hysteroscopic image features, TCM syndrome features, and clinical phenotype features in the subgraph through the readout module of the cross-comparison learning model. The similarity calculation module of the cross-comparison learning model is used to calculate the similarity of the first two-layer graph. The generated comparison graph branch multi-view subgraph, and the second double-layer graph The generated test graph branches into multiple view subgraphs, calculates various consistency scores between node embeddings, between node embeddings and subgraph global embeddings, and between different subgraph global embeddings, and performs weighted fusion of the consistency scores corresponding to the two two-layer graph branches; Based on the fused consistency scores, a contrastive learning objective function is constructed, which includes node-node-node contrastive loss, node-subgraph-subgraph contrastive loss, and subgraph-subgraph-subgraph contrastive loss. The parameters of the cross-contrast learning model are then optimized and trained.

7. The method for processing hysteroscopic lesion auxiliary diagnostic and treatment information based on multimodal graph contrastive learning according to claim 6, characterized in that, The implementation of calculating the multi-view consistency score corresponding to the comparison graph branch and the test graph branch, and fusing the consistency scores corresponding to the two branches, includes: set up This represents the branch type of a two-layer graph, where d=m represents the comparison graph branch and d=t represents the test graph branch; for any branch d, the node embeddings corresponding to the original graph subgraph, the enhanced subgraph, and the cross-layer subgraph are respectively represented as follows: , and The corresponding global embeddings of the subgraphs are represented as follows: , and ; The node-to-node consistency score for any branch d is represented as: ； The node-subgraph-subgraph consistency score corresponding to any branch d is represented as: ； The subgraph-subgraph-subgraph consistency score corresponding to any branch d is represented as: ； The consistency scores of the comparison graph branch and the test graph branch are fused using a weighted average: ,q∈{NNN,NSS,SSS}; in, , , The learnable weight matrices are calculated for node-level, node-subgraph-level, and subgraph-level similarity, respectively. It is a logical function; , ,and The consistency score after fusion is used to construct the comparative learning objective function and to calculate the subsequent lesion score.

8. The method for processing hysteroscopic lesion auxiliary diagnostic and treatment information based on multimodal graph contrastive learning according to claim 1, characterized in that, The implementation of lesion score calculation includes: During the model training phase, training instance pairs are constructed based on known pathological diagnosis results or risk levels in the training samples. During the model inference phase, low-risk reference instance sets and high-risk reference instance sets are pre-established based on the training set or validation set. The patient to be evaluated is only compared with the samples in the reference instance set in multiple rounds, and the pathological diagnosis results of the patient to be evaluated are not used as input. Let Q low-risk reference instances and Q high-risk reference instances be sampled in each round. The average similarity score between the patient to be evaluated and the low-risk reference instances in the r-th round is expressed as: ； The average similarity score between the patient to be evaluated and the high-risk reference instance in round r is represented as: ； in, This represents the q-th low-risk reference instance obtained from the r-th round of sampling; This represents the q-th high-risk reference instance obtained from the r-th round of sampling; This represents the predicted similarity output by the cross-comparison learning model. Patient nodes to be evaluated The lesion score is expressed as: ； Where R represents the number of sampling rounds for the reference instance; This represents the average similarity score between the patient to be evaluated and the high-risk reference instance in round r; The lesion score represents the average similarity score between the patient to be evaluated and the low-risk reference instance in the r-th round; the lesion score increases when the average similarity between the patient to be evaluated and the high-risk reference instance is higher than the average similarity between the patient and the low-risk reference instance.

9. The method for processing hysteroscopic lesion auxiliary diagnostic and treatment information based on multimodal graph contrastive learning according to claim 8, characterized in that, The method also includes: comparing lesion scores with preset thresholds to generate lesion risk warning information; when ≥ At that time, it generates a higher risk warning message for endometrial lesions; when < At that time, it indicates a lower risk of endometrial lesions; among them, The prompt information is used to provide doctors with auxiliary decision-making references, and is not directly used as the final clinical diagnosis conclusion or treatment prescription, and is either a preset threshold or an auxiliary judgment threshold determined based on the validation set.

10. The method for processing hysteroscopic lesion auxiliary diagnostic and treatment information based on multimodal graph contrastive learning according to claim 1 or 9, characterized in that, The method also includes generating TCM syndrome tendency prompts and re-examination suggestion prompts based on preset syndrome element matching rules or re-examination suggestion mapping rules.

Images