Multimodal data dynamic fusion and annotation method and system for medical decision-making

Abstract

The present invention provides a method and system for dynamic fusion and annotation of multimodal data for medical decision-making, which relates to the field of medical decision-making technology, including collecting medical images, electronic medical records and physiological monitoring data of patients, extracting multimodal features such as lesion morphology, disease type, symptoms, treatment records and physiological indicators, and constructing a dynamic fusion annotation model. The model encodes multimodal features, calculates similarity, constructs feature association graphs and performs feature aggregation, and annotates in combination with a medical knowledge base to generate dynamic annotation vectors. The dynamic annotation vectors are input into a medical decision support model, and based on similar case analysis and deep reinforcement learning, diagnosis and treatment recommendations containing treatment plans and early warning information are generated. The present invention can improve the accuracy and efficiency of medical decision-making and realize personalized diagnosis and treatment through dynamic fusion of multimodal data and online update mechanism.

Description

Technical Field

[0001] The present invention relates to medical decision-making technology, and in particular to a multimodal data dynamic fusion and annotation method and system for medical decision-making. Background Art

[0002] Medical decision support systems are designed to assist doctors in diagnosis and treatment, and to improve medical efficiency and accuracy. In recent years, with the diversification and complexity of medical data, how to effectively integrate and utilize this data has become a challenge for medical decision support systems. Multimodal data, including medical imaging data, electronic medical record data, and physiological monitoring data, contains rich patient information and can provide a more comprehensive basis for medical decision making. However, how to effectively integrate and annotate these multimodal data, and how to use these data for medical decision making, are still urgent issues to be solved.

[0003] Most existing medical decision support systems are based on single-modality data, such as using only imaging data or only electronic medical record data, and cannot fully utilize the complementarity of multimodal data. In addition, existing multimodal data fusion methods usually adopt static fusion methods, which makes it difficult to capture the dynamic changes of patients' conditions over time. Moreover, existing medical decision support systems lack effective annotation methods, making it difficult to perform semantic understanding and knowledge reasoning on multimodal data, limiting their application in complex medical scenarios. Summary of the invention

[0004] The embodiments of the present invention provide a multimodal data dynamic fusion and annotation method and system for medical decision-making, which can solve the problems in the prior art.

[0005] According to a first aspect of the embodiments of the present invention,

[0006] Provide a multimodal data dynamic fusion and annotation method for medical decision-making, including:

[0007] Collect the patient's medical imaging data, electronic medical record data and physiological monitoring data, use the pre-built three-dimensional lesion feature extraction module to perform hierarchical perception on the medical imaging data, obtain the three-dimensional contour of the lesion area, calculate the volume change rate and boundary diffusion of the three-dimensional contour, and extract the density distribution characteristics and texture evolution characteristics of the lesion area; use the hierarchical entity recognition model to extract disease type characteristics, symptom description characteristics and treatment record characteristics from the electronic medical record data, extract physiological indicator fluctuation characteristics after denoising the physiological monitoring data, and form a multimodal feature sample set with the density distribution characteristics, texture evolution characteristics, disease type characteristics, symptom description characteristics, treatment record characteristics and physiological indicator fluctuation characteristics;

[0008] Constructing a dynamic fusion annotation model, the dynamic fusion annotation model receives the multimodal feature sample set, encodes various features in the multimodal feature sample set through a feature encoding network to obtain a feature vector, calculates a similarity weight matrix between the feature vectors, constructs a feature association graph according to the similarity weight matrix, uses a graph convolutional network to perform feature aggregation on the feature association graph to obtain a fused feature vector, annotates the fused feature vector in combination with the diagnosis and treatment specifications in the medical knowledge base, and generates a dynamic annotation vector;

[0009] The dynamic annotation vector is input into the medical decision support model, and a similar case set is obtained based on the similarity analysis between the dynamic annotation vector and historical cases. The diagnosis and treatment records in the similar case set are subjected to time series analysis to obtain the disease development law and treatment response characteristics. A decision generation model is constructed based on a deep reinforcement learning framework, and the dynamic annotation vector and the treatment response characteristics are used as state inputs. The treatment effect in the similar case set is used as a reward signal to generate diagnosis and treatment recommendations including treatment plans and early warning information. The newly generated diagnosis and treatment records and their treatment effects are fed back to the dynamic fusion annotation model for online updating, and the diagnosis and treatment recommendations are stored in a medical knowledge base for subsequent case analysis.

[0010] In an optional embodiment,

[0011] The medical image data is layered sensed using a pre-built three-dimensional lesion feature extraction module to obtain a three-dimensional contour of the lesion area, calculate the volume change rate and boundary diffusion of the three-dimensional contour, and extract the density distribution characteristics and texture evolution characteristics of the lesion area, including:

[0012] Performing histogram matching on the medical image data to obtain standardized image data, and performing data integrity check on the standardized image data to obtain preprocessed medical image data;

[0013] A three-dimensional lesion extraction module is constructed, which includes an encoder and a decoder. The encoder is provided with multiple groups of residual-connected three-dimensional convolution units, each group of three-dimensional convolution units has different convolution kernel sizes and channels, and is used to extract multi-scale spatial features from preprocessed medical image data; the decoder performs feature fusion on the multi-scale spatial features through deconvolution operations, establishes dense skip connections between feature layers of the same resolution, and obtains three-dimensional contour data of the lesion area in combination with a spatial attention mechanism;

[0014] Adaptively tetrahedral meshing is performed on the three-dimensional contour data to obtain a time-series voxel model, and the boundary of the time-series voxel model is tracked by a level set evolution method, and the voxel topological structure changes at adjacent time points are extracted to obtain the volume change rate; a boundary evolution model based on curvature flow is constructed, and the boundary evolution model is combined with a regional growing algorithm to calculate the boundary diffusion of the lesion area;

[0015] A region of interest is determined based on the volume change rate and the boundary diffusion, and an adaptive three-dimensional filter group is constructed. The region of interest is decomposed at multiple scales by adjusting the direction parameters and scale parameters of the adaptive three-dimensional filter group to obtain a three-dimensional texture feature map. The local statistics of the three-dimensional texture feature map are calculated to obtain density distribution characteristics. At the same time, a three-dimensional grayscale co-occurrence matrix considering spatial position constraints is constructed according to the region of interest. The spatial position information is introduced as a constraint condition into the calculation process of the three-dimensional grayscale co-occurrence matrix, and the grayscale co-occurrence features of the region of interest are extracted. The local structural features are obtained by combining the local binary pattern operator. The grayscale co-occurrence features and the local structural features are correlated and clustered in the time dimension to obtain texture evolution features.

[0016] In an optional embodiment,

[0017] Constructing a boundary evolution model based on curvature flow, combining the boundary evolution model with the region growing algorithm, and calculating the boundary diffusion of the lesion area includes:

[0018] Constructing a velocity field term according to the volume change rate, and combining the velocity field term with a curvature term and an image gradient term to construct a curvature flow boundary evolution model;

[0019] Calculating local boundary energy, constructing an exponential decay function according to the local boundary energy to obtain a weight adjustment factor, and using the weight adjustment factor to dynamically adjust the weights of each component of the curvature flow boundary evolution model to obtain an adaptive evolution model;

[0020] Determine the initial boundary point set based on the adaptive evolution model, calculate the grayscale mean and standard deviation of the current region to construct a density similarity term, determine the local homogeneity term in combination with the neighborhood pixel variance of the boundary point set, map the local deviation of the volume change rate into a temporal consistency term, and combine the density similarity term, the local homogeneity term and the temporal consistency term to obtain a regional growing cost function;

[0021] Searching for candidate growth points in the neighborhood of the initial boundary point set, weightedly fusing the gradient information of the regional growth cost function with the adaptive evolution model, calculating the fusion weight coefficient according to the curvature value, gradient intensity and local contrast at the boundary point, and obtaining a spatiotemporal fusion boundary evolution equation;

[0022] Iteratively calculating the boundary displacement field of adjacent time points using the spatiotemporal fusion boundary evolution equation, constructing a diffusion tensor characterizing local motion characteristics based on the boundary displacement field; performing eigenvalue decomposition on the diffusion tensor, and calculating the directional intensity index of boundary diffusion according to the ratio of eigenvalues;

[0023] An attenuation function based on the volume change rate is constructed, the directional strength index is multiplied by the attenuation function to obtain a comprehensive diffusion measurement value, a feature map characterizing the boundary diffusion behavior is generated according to the comprehensive diffusion measurement value, the lesion area is segmented based on the feature map, the boundary contour of the segmented area is calculated, and the boundary diffusion degree of the lesion area is determined by combining the directional strength index and the volume change rate.

[0024] In an optional embodiment,

[0025] The feature association graph is aggregated using a graph convolutional network to obtain a fused feature vector, and the fused feature vector is annotated in combination with the diagnosis and treatment specifications in the medical knowledge base to generate a dynamic annotation vector, including:

[0026] Construct a multi-layer graph convolution network to perform feature aggregation on the feature association graph. Each layer of graph convolution includes a spatial attention branch and a channel attention branch. The spatial attention branch calculates the attention weight matrix of the neighborhood nodes. The channel attention branch generates the weight vector of the feature channel. The attention weight matrix and the weight vector are weighted with the input features to obtain dual-branch features. The dual-branch features and the input features are combined through cross-layer connections to generate enhanced features. The enhanced features are input into the next layer of graph convolution network to obtain multi-layer node features in sequence.

[0027] Constructing a feature pyramid for the multi-layer node features to generate a multi-scale feature sequence, calculating an inter-layer attention map based on the multi-scale feature sequence, converting the inter-layer attention map into a similarity matrix, calculating the fusion weight of the inter-layer features according to the similarity matrix, and performing weighted combination on the multi-layer node features to obtain a fused feature representation;

[0028] Construct a knowledge constraint network with a dual network structure, wherein the first sub-network is a concept association network, which constructs a concept graph based on the concept system in the medical knowledge base, and performs relational constraints on the fused feature representation; the second sub-network is a rule reasoning network, which converts medical norms into reasoning rules, performs rule constraints on the fused feature representation, and outputs knowledge enhancement features;

[0029] Constructing a progressive annotation hierarchy based on the knowledge enhancement feature, constructing a feature comparison module at each annotation hierarchy, the feature comparison module calculating positive sample pairs of concept features at the same level and negative sample pairs of concept features at different levels, generating a comparison feature representation, extracting hierarchical structure information from the concept system of the medical knowledge base, and constructing an annotation consistency constraint based on the hierarchical structure information;

[0030] A joint optimization strategy is designed to calculate the consistency loss based on the annotation consistency constraint, and a multi-task learning framework is constructed by combining the classification loss and contrast loss. The contributions of different loss terms are balanced through dynamic weights, and the model parameters are iteratively updated using an alternating optimization algorithm to obtain a dynamic annotation vector that integrates knowledge constraints.

[0031] In an optional embodiment,

[0032] A joint optimization strategy is designed to calculate the consistency loss based on the annotation consistency constraint. A multi-task learning framework is constructed by combining the classification loss and contrast loss. The contribution of different loss items is balanced through dynamic weights. The model parameters are iteratively updated using an alternating optimization algorithm. The dynamic annotation vector that integrates knowledge constraints is obtained, including:

[0033] Inputting the contrast feature representation into a cross entropy loss function with a temperature coefficient, wherein the temperature coefficient is used to adjust the smoothness of the prediction distribution, outputting a classification loss value, and calculating a category prediction vector based on the classification loss value;

[0034] Calculate the feature similarity matrix of the positive sample pair and the feature similarity matrix of the negative sample pair, construct a contrast loss calculation module based on the temperature parameter, input the feature similarity matrix of the positive sample pair and the feature similarity matrix of the negative sample pair into the contrast loss calculation module, output the contrast loss value, and generate a sample similarity vector;

[0035] Constructing a hierarchical transfer matrix based on the concept hierarchy in the medical knowledge base, multiplying the annotation vectors of different levels with the hierarchical transfer matrix to obtain a hierarchical mapping matrix, calculating the structural constraint loss of the hierarchical mapping matrix, outputting the consistency loss value, and extracting the hierarchical semantic relationship vector;

[0036] Constructing a dynamic weight generation network, inputting the training state vector into the weight generation network to generate a weight vector, normalizing the weight vector to obtain a dynamic weight coefficient, and performing weighted combination of the dynamic weight coefficient with the classification loss value, the contrast loss value, and the consistency loss value to obtain a total loss function;

[0037] Calculating the gradient of the cross entropy loss parameter according to the total loss function, calculating the gradient of the contrast loss parameter according to the contrast loss value, and calculating the gradient of the hierarchical constraint parameter according to the consistency loss value;

[0038] The momentum value at the previous moment is weighted with the current gradients to obtain the updated momentum, and the corresponding parameters are iteratively optimized using the updated momentum until the total loss function converges. The classification prediction vector, contrast feature vector and hierarchical constraint vector corresponding to the optimized parameters are weightedly fused to obtain a dynamic annotation vector with knowledge constraints.

[0039] In an optional embodiment,

[0040] A decision generation model is constructed based on a deep reinforcement learning framework, the dynamic annotation vector and the treatment response feature are used as state inputs, and the treatment effect in the similar case set is used as a reward signal to generate diagnosis and treatment recommendations containing treatment plans and early warning information, including:

[0041] Obtain disease diagnosis and treatment guidelines and clinical diagnosis and treatment data, extract treatment rules and pathway information from the disease diagnosis and treatment guidelines, and construct a disease knowledge graph in combination with the clinical diagnosis and treatment data;

[0042] Constructing a multimodal feature fusion network, wherein the multimodal feature fusion network comprises a feature branch channel and an attention guidance module, calculating attention weights of dynamic annotation vectors and treatment response features through the attention guidance module, and fusing features based on the attention weights to obtain a state vector;

[0043] Extracting a treatment path from the disease knowledge graph through a graph attention network, generating a set of candidate treatment plans based on the treatment path, and discretizing continuous treatment parameters in the set of candidate treatment plans to obtain a set of discretized treatment plans;

[0044] Constructing a deep reinforcement learning network, the deep reinforcement learning network comprising a state encoder and an action evaluator, inputting the state vector into the state encoder to obtain a state representation, and inputting the state representation and a treatment plan in the discretized treatment plan set into the action evaluator to obtain a plan value;

[0045] Extracting time-series treatment records from a set of similar cases, inputting the time-series treatment records into a pre-built causal inference model to obtain a degree of correlation of treatment effects, calculating an experience reward value based on the degree of correlation of treatment effects, inputting the set of candidate treatment plans into an expert rule system to obtain a constraint reward value, and combining the experience reward value and the constraint reward value to obtain a reward signal;

[0046] Building a priority experience replay pool based on the reward signal, storing state transition samples in the priority experience replay pool, calculating the importance score of the state transition samples, setting the sampling probability based on the importance score and extracting training samples, and using the training samples to optimize the parameters of the deep reinforcement learning network to obtain a trained decision generation model;

[0047] The current state vector is input into the trained decision generation model, the treatment plan with the maximum plan value is selected from the discretized treatment plan set as the optimal treatment plan, the reliability score of the optimal treatment plan is calculated, risk warning information is generated based on the plan value distribution of the optimal treatment plan, and the optimal treatment plan, reliability score and risk warning information are combined to form a diagnosis and treatment recommendation.

[0048] In an optional embodiment,

[0049] Extracting time series treatment records from a similar case set, inputting the time series treatment records into a pre-built causal inference model to obtain a treatment effect correlation, and calculating an experience reward value based on the treatment effect correlation includes:

[0050] Extracting treatment regimen sequences, clinical indicator sequences, symptom score sequences, and adverse reaction records from the similar case set, and combining them to generate a time series treatment record;

[0051] Constructing a structural causal model, taking the treatment plan sequence, the clinical indicator sequence, and the symptom score sequence in the time series treatment record as causal graph nodes, establishing causal edges between the causal graph nodes based on an expert knowledge base and data correlation analysis results, and constructing a dynamic causal graph using time series dependencies;

[0052] Constructing a counterfactual reasoning module, inputting the time series treatment record into the counterfactual reasoning module, processing the feature interaction relationship in the time series treatment record through a gated attention mechanism, obtaining an average treatment effect and a conditional average treatment effect, and constructing a causal effect scoring matrix based on the average treatment effect and the conditional average treatment effect, wherein the row vectors of the causal effect scoring matrix represent different treatment plans, the column vectors represent different clinical indicators, and the matrix elements represent the correlation of treatment effects;

[0053] Performing eigendecomposition on the causal effect score matrix, extracting a main eigenvector, using the main eigenvector as a treatment effect weight vector, and performing weighted combination of the treatment effect weight vector with a clinical indicator sequence, a symptom score sequence, and an adverse reaction record, respectively, to obtain a multi-dimensional reward vector;

[0054] A multi-layer reward distribution network is constructed, including a feature extraction layer, a temporal correlation analysis layer and a reward fusion layer. The multi-dimensional reward vector is input into the feature extraction layer, and the high-frequency noise is removed by wavelet transform to obtain a filtered reward vector. Local features are extracted through a multi-head self-attention mechanism to generate an enhanced feature vector. The enhanced feature vector is input into the temporal correlation analysis layer, and the temporal information is embedded through position encoding. The attention scores of different time steps are calculated, and a temporal dependency matrix is ​​constructed based on the attention scores. The temporal dependency matrix is ​​tensor multiplied with the enhanced feature vector to obtain a temporal weighted feature, and a jump connection is used to add the temporal weighted feature to the filtered reward vector. The feature weight is dynamically adjusted through an adaptive gating unit, and the experience reward value is finally output.

[0055] A second aspect of an embodiment of the present invention provides a multimodal data dynamic fusion and annotation system for medical decision-making, including:

[0056] The first unit is used to collect the patient's medical imaging data, electronic medical record data and physiological monitoring data, and use the pre-built three-dimensional lesion feature extraction module to perform hierarchical perception on the medical imaging data to obtain the three-dimensional contour of the lesion area, calculate the volume change rate and boundary diffusion of the three-dimensional contour, and extract the density distribution characteristics and texture evolution characteristics of the lesion area; use the hierarchical entity recognition model to extract the disease type characteristics, symptom description characteristics and treatment record characteristics from the electronic medical record data, perform noise reduction on the physiological monitoring data and then extract the physiological index fluctuation characteristics, and form a multimodal feature sample set with the density distribution characteristics, texture evolution characteristics, disease type characteristics, symptom description characteristics, treatment record characteristics and physiological index fluctuation characteristics;

[0057] The second unit is used to construct a dynamic fusion annotation model, which receives the multimodal feature sample set, encodes various features in the multimodal feature sample set through a feature encoding network to obtain a feature vector, calculates a similarity weight matrix between the feature vectors, constructs a feature association graph according to the similarity weight matrix, uses a graph convolution network to perform feature aggregation on the feature association graph to obtain a fused feature vector, and annotates the fused feature vector in combination with the diagnosis and treatment specifications in the medical knowledge base to generate a dynamic annotation vector;

[0058] The third unit is used to input the dynamic annotation vector into the medical decision support model, obtain a similar case set based on the similarity analysis between the dynamic annotation vector and the historical cases, perform time series analysis on the diagnosis and treatment records in the similar case set to obtain the disease development law and treatment response characteristics, build a decision generation model based on the deep reinforcement learning framework, use the dynamic annotation vector and the treatment response characteristics as state inputs, use the treatment effect in the similar case set as a reward signal, generate diagnosis and treatment recommendations including treatment plans and early warning information, feed back the newly generated diagnosis and treatment records and their treatment effects to the dynamic fusion annotation model for online updating, and store the diagnosis and treatment recommendations in the medical knowledge base for subsequent case analysis.

[0059] A third aspect of the embodiments of the present invention

[0060] An electronic device is provided, comprising:

[0061] processor;

[0062] a memory for storing processor-executable instructions;

[0063] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0064] A fourth aspect of the embodiments of the present invention is:

[0065] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0066] In this embodiment, the accuracy and personalization of medical decision-making are improved through dynamic fusion and feature extraction of multimodal data. Combining medical images, electronic medical records and physiological monitoring data, it is not only possible to fully capture the morphology of lesions, disease types and treatment history, but also to achieve three-dimensional evaluation of diseases through detailed feature analysis such as three-dimensional lesion characteristics and fluctuations in physiological indicators. The dynamic fusion annotation model effectively aggregates multidimensional data, improves the correlation between data, and enhances the accuracy of disease pattern recognition through the application of feature encoding and graph convolutional networks. At the same time, combined with the decision generation model of deep reinforcement learning, the treatment effect can be analyzed according to similar cases, personalized diagnosis and treatment plans can be generated, and continuous optimization can be performed based on the historical feedback mechanism, ultimately improving the efficiency and intelligence level of the medical decision support system, and promoting real-time online updates and self-improvement of the medical knowledge base. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 A schematic diagram of a process of a multimodal data dynamic fusion and annotation method for medical decision-making according to an embodiment of the present invention;

[0068] Figure 2This is a system flow chart of an embodiment of the present invention;

[0069] Figure 3 It is a schematic diagram of the structure of a multimodal data dynamic fusion and annotation system for medical decision-making according to an embodiment of the present invention. DETAILED DESCRIPTION

[0070] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0071] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0072] Figure 1 FIG. 1 is a flow chart of a multimodal data dynamic fusion and annotation method for medical decision-making according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0073] S101. Collect the patient's medical imaging data, electronic medical record data and physiological monitoring data, use the pre-built three-dimensional lesion feature extraction module to perform hierarchical perception on the medical imaging data, obtain the three-dimensional contour of the lesion area, calculate the volume change rate and boundary diffusion of the three-dimensional contour, and extract the density distribution characteristics and texture evolution characteristics of the lesion area; use the hierarchical entity recognition model to extract disease type characteristics, symptom description characteristics and treatment record characteristics from the electronic medical record data, perform noise reduction processing on the physiological monitoring data and extract physiological indicator fluctuation characteristics, and form a multimodal feature sample set with the density distribution characteristics, texture evolution characteristics, disease type characteristics, symptom description characteristics, treatment record characteristics and physiological indicator fluctuation characteristics;

[0074] S102. Construct a dynamic fusion annotation model, wherein the dynamic fusion annotation model receives the multimodal feature sample set, encodes various features in the multimodal feature sample set through a feature encoding network to obtain a feature vector, calculates a similarity weight matrix between the feature vectors, constructs a feature association graph according to the similarity weight matrix, uses a graph convolutional network to perform feature aggregation on the feature association graph to obtain a fused feature vector, annotates the fused feature vector in combination with the diagnosis and treatment specifications in the medical knowledge base, and generates a dynamic annotation vector;

[0075] S103. Input the dynamic annotation vector into the medical decision support model, obtain a similar case set based on the similarity analysis between the dynamic annotation vector and the historical cases, perform time series analysis on the diagnosis and treatment records in the similar case set to obtain the disease development rules and treatment response characteristics, build a decision generation model based on the deep reinforcement learning framework, use the dynamic annotation vector and the treatment response characteristics as state inputs, use the treatment effect in the similar case set as a reward signal, generate diagnosis and treatment recommendations including treatment plans and early warning information, feed back the newly generated diagnosis and treatment records and their treatment effects to the dynamic fusion annotation model for online updating, and store the diagnosis and treatment recommendations in the medical knowledge base for subsequent case analysis.

[0076] Among them, medical imaging data is obtained through image acquisition equipment (such as CT, MRI, X-ray, etc.), which can reveal the morphology, location and changes of lesions. Electronic medical record data contains the patient's historical diagnosis and treatment records, including disease type, symptom description, treatment process and other information. Physiological monitoring data refers to the patient's physiological indicators obtained through physiological monitoring equipment (such as electrocardiogram, blood oxygen monitor, etc.), such as heart rate, blood pressure, body temperature and other data.

[0077] The processing of electronic medical record data relies on a hierarchical entity recognition model. This model can automatically identify disease types, symptom descriptions, and treatment records from complex texts. This process uses natural language processing technology to extract key information from medical records and provide structured data support for subsequent analysis.

[0078] The processing of physiological monitoring data involves denoising. After removing noise and interference signals, the fluctuation characteristics of physiological indicators are extracted. The fluctuation characteristics can reflect the changes in the patient's health status and provide a basis for early warning of diseases and monitoring of treatment effects.

[0079] The dynamic fusion annotation model is a comprehensive and adaptable machine learning model that aims to automatically generate accurate and personalized medical annotations (such as disease type, lesion condition, treatment plan, etc.) through the fusion of multimodal data, dynamic updating of features, and integration with the medical knowledge base.

[0080] The medical decision support model will first perform a similar case analysis and select a group of similar cases based on the similarity between the dynamic annotation vector and the historical case data. These similar cases are selected not only based on the similarity of disease type and symptoms, but also include a comprehensive evaluation of multiple dimensions such as lesion morphology and treatment records. In this way, the model can mine the case set that is most similar to the current patient from historical cases.

[0081] Time series analysis of the medical records in these similar case collections can reveal the development patterns of the disease and the characteristics of treatment response. Time series analysis refers to the process of analyzing and modeling a series of data arranged in chronological order. Its main goal is to identify the patterns, trends, periodicity, and random fluctuations of data over time, and use these characteristics to predict future trends or make decisions. In the medical field, time series analysis is often used to analyze patients' condition changes, treatment responses, fluctuations in vital signs, etc., to help doctors better understand the development of the disease and the effectiveness of treatment.

[0082] In an optional embodiment, the medical image data is layered sensed using a pre-built three-dimensional lesion feature extraction module to obtain a three-dimensional contour of the lesion area, the volume change rate and boundary diffusion of the three-dimensional contour are calculated, and the density distribution characteristics and texture evolution characteristics of the lesion area are extracted, including:

[0083] Performing histogram matching on the medical image data to obtain standardized image data, and performing data integrity check on the standardized image data to obtain preprocessed medical image data;

[0084] A three-dimensional lesion extraction module is constructed, which includes an encoder and a decoder. The encoder is provided with multiple groups of residual-connected three-dimensional convolution units, each group of three-dimensional convolution units has different convolution kernel sizes and channels, and is used to extract multi-scale spatial features from preprocessed medical image data; the decoder performs feature fusion on the multi-scale spatial features through deconvolution operations, establishes dense skip connections between feature layers of the same resolution, and obtains three-dimensional contour data of the lesion area in combination with a spatial attention mechanism;

[0085] Adaptively tetrahedral meshing is performed on the three-dimensional contour data to obtain a time-series voxel model, and the boundary of the time-series voxel model is tracked by a level set evolution method, and the voxel topological structure changes at adjacent time points are extracted to obtain the volume change rate; a boundary evolution model based on curvature flow is constructed, and the boundary evolution model is combined with a regional growing algorithm to calculate the boundary diffusion of the lesion area;

[0086] A region of interest is determined based on the volume change rate and the boundary diffusion, and an adaptive three-dimensional filter group is constructed. The region of interest is decomposed at multiple scales by adjusting the direction parameters and scale parameters of the adaptive three-dimensional filter group to obtain a three-dimensional texture feature map. The local statistics of the three-dimensional texture feature map are calculated to obtain density distribution characteristics. At the same time, a three-dimensional grayscale co-occurrence matrix considering spatial position constraints is constructed according to the region of interest. The spatial position information is introduced as a constraint condition into the calculation process of the three-dimensional grayscale co-occurrence matrix, and the grayscale co-occurrence features of the region of interest are extracted. The local structural features are obtained by combining the local binary pattern operator. The grayscale co-occurrence features and the local structural features are correlated and clustered in the time dimension to obtain texture evolution features.

[0087] For example, first, the collected medical image data usually has certain noise and inconsistency, and needs to be standardized. Through the histogram matching technology, the patient's image data is compared with the standard image, so that the brightness and contrast of the image are consistent with the distribution of the standard image, thereby obtaining the standardized image data. This step not only eliminates the influence of different devices or different shooting conditions, but also provides a clear basis for subsequent feature extraction.

[0088] Then, the standardized image data is checked for data integrity to ensure that the image data is not missing or damaged. For example, the image size, pixel integrity, and whether the target area in the image is complete are checked to ensure data quality. After the integrity check is completed, the preprocessed medical image data is obtained, which will serve as the input of the subsequent 3D lesion feature extraction module.

[0089] Next, based on the preprocessed image data, a 3D lesion feature extraction module is constructed. This module includes an encoder and a decoder. The encoder is responsible for extracting multi-scale spatial features from the image data, while the decoder is responsible for fusing these features and reconstructing the 3D contour of the lesion area.

[0090] In the encoder, multiple groups of residual-connected three-dimensional convolutional units are used, and each group of convolutional units has different convolution kernel sizes and number of channels. This design can effectively capture different scale information in the image data, thereby improving the accuracy and robustness of feature extraction. Through this multi-scale convolutional network, rich spatial feature information can be extracted from the image, providing high-dimensional feature representation for subsequent lesion area analysis.

[0091] The decoder fuses the extracted spatial features through deconvolution operations and establishes dense jump connections between feature layers of the same resolution. This dense connection enables the low-level and high-level features to be effectively transferred, thereby enhancing the expressiveness of the features. In addition, the introduction of the spatial attention mechanism allows the network to pay more attention to the lesion area during the fusion process, thereby obtaining accurate three-dimensional contour data of the lesion area.

[0092] After obtaining the three-dimensional contour data of the lesion area, it needs to be further processed and feature extracted.

[0093] First, the adaptive tetrahedral meshing method is used to divide the three-dimensional contour data into several tetrahedral meshes to generate a time-series voxel model. The time-series voxel model is a time-series three-dimensional mesh data structure that can track the changes in the lesion area at multiple time points. Through this model, the morphological changes of the lesion and its evolution in the time dimension can be better analyzed.

[0094] Secondly, the level set evolution method is used to track the boundaries of the time series voxel model. The level set method is a numerical method on a dynamic grid that can accurately track the boundaries of the lesion area and compare its changes at different time points. This process can extract the volume change rate of the lesion area at different time points, that is, the increase or decrease in the lesion volume, reflecting the growth or shrinkage trend of the lesion.

[0095] In order to analyze the diffusion of the lesion area boundary, a boundary evolution model based on curvature flow is combined. This model provides the smoothness and morphological changes of the lesion boundary by simulating the evolution of the lesion boundary over time. This boundary evolution model can accurately describe the boundary changes of the lesion area at each time point.

[0096] On this basis, the region growing algorithm is combined with the boundary evolution model to further refine the boundary analysis, especially for lesion areas with complex morphology. This algorithm can consider the spatial distribution of the lesion area in the calculation of boundary diffusion, thereby more accurately evaluating the extent of lesion expansion.

[0097] After extracting the volume change rate and boundary diffusion of the lesion area, the next step is to analyze the texture features of the lesion. Texture features in medical images usually reflect the complexity of tissue structure and lesions and can provide important information about the nature of the lesion.

[0098] Based on the region of interest of the lesion area, an adaptive three-dimensional filter group is constructed. The filter group can perform multi-scale decomposition of the lesion area by adjusting the direction parameters and scale parameters, thereby extracting the three-dimensional texture features of the lesion. These features can reflect the texture structure of the lesion area at different scales, including the density distribution and tissue heterogeneity inside the lesion.

[0099] Subsequently, the local statistics of the calculated three-dimensional texture feature map are used to obtain the density distribution characteristics of the lesion area. In addition, by constructing a three-dimensional gray-level co-occurrence matrix, considering the spatial position constraints, and introducing spatial position information in the calculation of the co-occurrence matrix, the texture analysis is made more accurate.

[0100] Finally, the local binary pattern operator (LBP) is used to extract local structural features, and combined with the local statistics in the texture feature map, correlation analysis and clustering are performed in the time dimension to obtain the texture evolution characteristics of the lesion area. This feature can reflect the temporal evolution of the lesion, such as the changes in the texture inside the lesion and whether it has malignant transformation.

[0101] In this embodiment, through multi-scale spatial feature extraction and hierarchical perception, the lesion area can be more accurately identified and segmented, and the accuracy of diagnosis can be improved. By extracting multi-dimensional information such as the volume change rate, boundary diffusion, density distribution characteristics, and texture evolution characteristics of the lesion area, a more comprehensive lesion feature is provided, which helps doctors to understand the development of the disease more comprehensively. By analyzing the time series voxel model, dynamic monitoring of the lesion can be achieved, helping doctors to better evaluate the treatment effect and predict the development trend of the disease.

[0102] In an optional implementation, a boundary evolution model based on curvature flow is constructed, and the boundary evolution model is combined with a region growing algorithm to calculate the boundary diffusion of the lesion region, including:

[0103] Constructing a velocity field term according to the volume change rate, and combining the velocity field term with a curvature term and an image gradient term to construct a curvature flow boundary evolution model;

[0104] Calculating local boundary energy, constructing an exponential decay function according to the local boundary energy to obtain a weight adjustment factor, and using the weight adjustment factor to dynamically adjust the weights of each component of the curvature flow boundary evolution model to obtain an adaptive evolution model;

[0105] Determine the initial boundary point set based on the adaptive evolution model, calculate the grayscale mean and standard deviation of the current region to construct a density similarity term, determine the local homogeneity term in combination with the neighborhood pixel variance of the boundary point set, map the local deviation of the volume change rate into a temporal consistency term, and combine the density similarity term, the local homogeneity term and the temporal consistency term to obtain a regional growing cost function;

[0106] Searching for candidate growth points in the neighborhood of the initial boundary point set, weightedly fusing the gradient information of the regional growth cost function with the adaptive evolution model, calculating the fusion weight coefficient according to the curvature value, gradient intensity and local contrast at the boundary point, and obtaining a spatiotemporal fusion boundary evolution equation;

[0107] Iteratively calculating the boundary displacement field of adjacent time points using the spatiotemporal fusion boundary evolution equation, constructing a diffusion tensor characterizing local motion characteristics based on the boundary displacement field; performing eigenvalue decomposition on the diffusion tensor, and calculating the directional intensity index of boundary diffusion according to the ratio of eigenvalues;

[0108] An attenuation function based on the volume change rate is constructed, the directional strength index is multiplied by the attenuation function to obtain a comprehensive diffusion measurement value, a feature map characterizing the boundary diffusion behavior is generated according to the comprehensive diffusion measurement value, the lesion area is segmented based on the feature map, the boundary contour of the segmented area is calculated, and the boundary diffusion degree of the lesion area is determined by combining the directional strength index and the volume change rate.

[0109] Exemplarily, first, a boundary evolution model based on curvature flow is constructed. The change in the volume of the target area at adjacent time points is calculated to construct a velocity field. This velocity field reflects the growth or contraction trend of the target area. This velocity field is combined with the curvature term that reflects the change in boundary curvature and the image gradient term that reflects the change in local grayscale of the image to construct a curvature flow boundary evolution model. Assuming that the volume of the target area increases by 10 pixels per unit time, the value of the velocity field term is 10. The curvature term is taken according to the convexity and concavity of the boundary curvature, with positive values ​​for convex areas and negative values ​​for concave areas. The image gradient term is taken according to the grayscale difference on both sides of the boundary. The greater the grayscale difference, the greater the value.

[0110] Next, an adaptive evolution model is constructed. The local boundary energy is calculated, which can be defined as the product of the square of the boundary curvature and the image gradient. An exponential decay function is constructed based on the local boundary energy to obtain a weight adjustment factor. This exponential decay function can dynamically adjust the weight according to the size of the local boundary energy. The higher the energy, the smaller the weight. This weight adjustment factor is used to dynamically adjust the weights of the velocity field term, curvature term, and image gradient term in the curvature flow boundary evolution model to obtain an adaptive evolution model. For example, if the local boundary energy is high, the weight of the curvature term is reduced and the weight of the velocity field term is increased, so that the evolution model focuses more on the volume changes in the target area.

[0111] Then, regional growing is performed. Based on the initial boundary point set determined by the adaptive evolution model, the grayscale mean and standard deviation of the current region are calculated, and the density similarity term is constructed. The density similarity term is used to measure the grayscale similarity between the candidate growth point and the current region. At the same time, the variance of the neighborhood pixels of the boundary point set is calculated to determine the local homogeneity term. The local homogeneity term is used to measure the grayscale uniformity of the area around the candidate growth point. In addition, the local deviation of the volume change rate is mapped to a temporal consistency term, which is used to measure whether the growth trend of the candidate growth point is consistent with that of the target area. The density similarity term, the local homogeneity term, and the temporal consistency term are combined to obtain the regional growing cost function. If the grayscale value of a candidate growth point is close to the grayscale mean of the current region, the surrounding area has uniform grayscale, and the growth trend is consistent with that of the target area, then its regional growing cost function value is low.

[0112] Candidate growth points are searched in the neighborhood of the initial boundary point set, and the gradient information of the regional growth cost function is weightedly fused with the adaptive evolution model to obtain the boundary evolution equation of spatiotemporal fusion. The fusion weight coefficient is calculated based on the curvature value, gradient intensity and local contrast at the boundary point. If the curvature at the boundary point is large, the gradient intensity is high, and the local contrast is strong, the adaptive evolution model is given a higher weight.

[0113] The boundary displacement field of adjacent time points is iteratively calculated using the boundary evolution equation of spatiotemporal fusion, and a diffusion tensor that characterizes the local motion characteristics is constructed based on the boundary displacement field. The diffusion tensor is subjected to eigenvalue decomposition, and the directional strength index of boundary diffusion is calculated based on the ratio of eigenvalues. If the maximum eigenvalue of the diffusion tensor is much larger than other eigenvalues, it means that the boundary diffusion has a strong directionality.

[0114] Construct an attenuation function based on the volume change rate, and multiply the directional strength index with the attenuation function to obtain a comprehensive diffusion metric. Generate a feature map that represents the boundary diffusion behavior based on the comprehensive diffusion metric, segment the lesion area based on the feature map, calculate the boundary contour of the segmented area, and determine the boundary diffusion of the lesion area by combining the directional strength index and the volume change rate. If the volume of the target area continues to increase and the boundary diffusion has a strong directionality, the boundary diffusion is high.

[0115] In this embodiment, the advantages of curvature flow and region growing algorithms are combined to more accurately capture the boundary of the lesion area and calculate its diffusion. The weight of the evolution model can be dynamically adjusted according to the local boundary energy to adapt to different types of lesion areas. The temporal consistency term is used to effectively suppress the influence of noise and improve the robustness of segmentation.

[0116] In an optional implementation, a graph convolutional network is used to perform feature aggregation on the feature association graph to obtain a fused feature vector, and the fused feature vector is annotated in combination with the diagnosis and treatment specifications in the medical knowledge base. Generating a dynamic annotation vector includes:

[0117] Construct a multi-layer graph convolution network to perform feature aggregation on the feature association graph. Each layer of graph convolution includes a spatial attention branch and a channel attention branch. The spatial attention branch calculates the attention weight matrix of the neighborhood nodes. The channel attention branch generates the weight vector of the feature channel. The attention weight matrix and the weight vector are weighted with the input features to obtain dual-branch features. The dual-branch features and the input features are combined through cross-layer connections to generate enhanced features. The enhanced features are input into the next layer of graph convolution network to obtain multi-layer node features in sequence.

[0118] Constructing a feature pyramid for the multi-layer node features to generate a multi-scale feature sequence, calculating an inter-layer attention map based on the multi-scale feature sequence, converting the inter-layer attention map into a similarity matrix, calculating the fusion weight of the inter-layer features according to the similarity matrix, and performing weighted combination on the multi-layer node features to obtain a fused feature representation;

[0119] Construct a knowledge constraint network with a dual network structure, wherein the first sub-network is a concept association network, which constructs a concept graph based on the concept system in the medical knowledge base, and performs relational constraints on the fused feature representation; the second sub-network is a rule reasoning network, which converts medical norms into reasoning rules, performs rule constraints on the fused feature representation, and outputs knowledge enhancement features;

[0120] Constructing a progressive annotation hierarchy based on the knowledge enhancement feature, constructing a feature comparison module at each annotation hierarchy, the feature comparison module calculating positive sample pairs of concept features at the same level and negative sample pairs of concept features at different levels, generating a comparison feature representation, extracting hierarchical structure information from the concept system of the medical knowledge base, and constructing an annotation consistency constraint based on the hierarchical structure information;

[0121] A joint optimization strategy is designed to calculate the consistency loss based on the annotation consistency constraint, and a multi-task learning framework is constructed by combining the classification loss and contrast loss. The contributions of different loss terms are balanced through dynamic weights, and the model parameters are iteratively updated using an alternating optimization algorithm to obtain a dynamic annotation vector that integrates knowledge constraints.

[0122] Exemplarily, first, a feature association graph is constructed. The nodes of the graph represent the input medical features, such as the patient's symptoms, signs, test results, etc. The edges between the nodes represent the correlation between the features, such as the coexistence relationship between symptoms, the correlation between signs and diseases, etc. The correlation can be obtained through statistical analysis, expert knowledge, or knowledge graphs. For example, the frequency of two symptoms co-occurring in a patient group can be counted, and the higher the frequency, the stronger the correlation.

[0123] Next, a multi-layer graph convolutional network is used to perform feature aggregation on the feature association graph. Each layer of the graph convolutional network contains a spatial attention branch and a channel attention branch. The spatial attention branch calculates the attention weights of neighboring nodes. For example, for a symptom node, symptom nodes with strong correlations will have higher weights. The channel attention branch generates a weight vector for the feature channel. For example, for multiple feature dimensions of a symptom (such as duration, severity, etc.), important dimensions will have higher weights. The attention weights and input features are weighted to obtain dual-branch features. The dual-branch features are combined with the input features through cross-layer connections to generate enhanced features, which are input to the next layer of the graph convolutional network. Repeat the above process to obtain multi-layer node features.

[0124] Suppose there are three symptom nodes: A (headache), B (fever), and C (cough). The correlation between A and B is strong, and the correlation between A and C is weak. The initial feature vector is A=[1, 2], B=[3, 4], and C=[5, 6]. The spatial attention branch calculates that the weight of A to B is 0.8, and the weight of A to C is 0.2. The channel attention branch calculates the weight vector of the feature channel as [0.7, 0.3]. Then the spatial attention feature of A is 0.8*B+0.2*C=[2.9, 3.6], and the channel attention feature is [0.7*1+0.3*2, 0.7*2+0.3*4]=[1.3, 2.6]. Connect the spatial attention feature, channel attention feature, and original feature A to obtain the enhanced feature [1, 2, 2.9, 3.6, 1.3, 2.6].

[0125] Next, a feature pyramid is constructed for multi-layer node features to generate a multi-scale feature sequence. For example, node features of different layers can be pooled to obtain feature representations of different scales. Inter-layer attention maps are calculated based on the multi-scale feature sequence and converted into a similarity matrix. The fusion weights of inter-layer features are calculated based on the similarity matrix, and multi-layer node features are weighted combined to obtain a fused feature representation.

[0126] Construct a knowledge constraint network with a dual network structure. The first sub-network is a concept association network, which constructs a concept graph based on the concept system (such as the disease classification system) in the medical knowledge base. The fused feature representation is input into the concept association network to perform relationship constraints. The second sub-network is a rule reasoning network, which converts medical norms (such as diagnosis and treatment guidelines) into reasoning rules. The fused feature representation is input into the rule reasoning network to perform rule constraints. Finally, the knowledge enhancement feature is output.

[0127] Assume that the fusion feature is represented as [0.1, 0.8, 0.2]. The concept association network judges that the feature is strongly associated with the concept of "cold" and weakly associated with the concept of "pneumonia" based on the knowledge base, and adjusts the feature to [0.2, 0.9, 0.1]. The rule reasoning network judges that if "fever" and "cough" are present, a "routine blood test" is required based on the diagnosis and treatment guidelines, and adjusts the feature to [0.2, 0.9, 0.1, 0.7], where 0.7 represents the probability of "routine blood test".

[0128] Then, a progressive annotation hierarchy is constructed based on the knowledge-enhanced features. A hierarchy can be constructed based on the disease classification system, from symptoms to major disease categories to specific diseases. A feature comparison module is constructed at each annotation level. The feature comparison module calculates positive sample pairs of concept features at the same level and negative sample pairs of concept features at different levels to generate a comparison feature representation. Hierarchical information is extracted from the concept system of the medical knowledge base, and annotation consistency constraints are constructed based on the hierarchical information. For example, if a patient is diagnosed with "bacterial pneumonia", then its upper-level concepts "pneumonia" and "respiratory diseases" should also be labeled.

[0129] Finally, a joint optimization strategy is designed. The core of this strategy is to calculate the consistency loss based on the "label consistency constraint". The main goal of the label consistency constraint is to ensure that the label output by the model is consistent with the expected label, so as to reduce the impact of the wrong label generated by the model inference on the final result. Specifically, this process includes comparing the labels of each pair of input data and calculating the difference between the model output and the true label to obtain the consistency loss.

[0130] The calculation method of consistency loss relies on processing the multi-dimensional feature representation of the input data. This loss measures the gap between the model output label and the reference label. Common loss functions include cross entropy loss function or mean square error loss function, which can effectively capture the deviation between the model prediction and the true label and optimize the model by minimizing the consistency loss.

[0131] On this basis, a multi-task learning framework is constructed by combining classification loss and contrastive loss. Classification loss is mainly used to supervise the category prediction of the model output and measure the error between the model prediction category and the true category. Contrastive loss is used to learn the similarity between samples. By pulling similar samples closer and pushing different samples farther apart, the model can better learn the implicit relationship between samples. The two work together to optimize the model's classification performance and contrastive learning ability at the same time during the model training process, thereby improving the model's performance in various tasks.

[0132] In order to balance the contribution of different loss items, a dynamic weight balancing mechanism needs to be introduced. The core of this mechanism is to automatically adjust the weights of each loss item according to the different requirements of the task, so that during the training process, the classification loss, contrast loss, and consistency loss can be dynamically adjusted according to their importance. For example, in the initial training stage of the model, it may be necessary to pay more attention to the consistency loss, while as the training progresses, the contribution of classification and contrast loss gradually increases. This dynamic adjustment process ensures that the effects of each task during the model training process are balanced by controlling the weights of each loss item.

[0133] Next, the model parameters are iteratively updated by using the alternating optimization algorithm. The alternating optimization algorithm is a common optimization method in multi-task learning. Its basic idea is to alternately optimize the objectives of different loss functions in each round of iteration. Specifically, the classification loss and contrast loss can be optimized first to ensure that the model can complete the classification task and similarity learning task well, and then the consistency loss can be optimized to make the model more consistent when outputting labels. In each round of optimization, the weights are dynamically adjusted according to the contribution of the loss term to achieve the best training effect.

[0134] Through these optimization steps, the model parameters can be continuously updated to obtain a dynamic annotation vector that incorporates knowledge constraints. This vector integrates a variety of information, including the patient's imaging data, medical record data and their corresponding labels, and has high accuracy and robustness. Ultimately, after multiple rounds of alternating optimization, the model will learn better feature representation and decision-making capabilities, making the annotation process more accurate and in line with the standard specifications in the medical knowledge base.

[0135] In this embodiment, by combining the medical knowledge base and the graph convolutional network, the correlation between features and knowledge constraints can be captured more accurately, thereby improving the annotation accuracy. The annotation vector can be dynamically generated according to the input features, avoiding the limitation of the traditional method that requires manual definition of annotation rules. By introducing knowledge constraints and attention mechanisms, the decision-making process of the model can be better understood and the interpretability of the model can be enhanced.

[0136] In an optional implementation, a joint optimization strategy is designed to calculate the consistency loss based on the annotation consistency constraint, and a multi-task learning framework is constructed by combining the classification loss and the contrast loss. The contributions of different loss items are balanced by dynamic weights, and the model parameters are iteratively updated by an alternating optimization algorithm. The dynamic annotation vector integrating the knowledge constraints is obtained, including:

[0137] Inputting the contrast feature representation into a cross entropy loss function with a temperature coefficient, wherein the temperature coefficient is used to adjust the smoothness of the prediction distribution, outputting a classification loss value, and calculating a category prediction vector based on the classification loss value;

[0138] Calculate the feature similarity matrix of the positive sample pair and the feature similarity matrix of the negative sample pair, construct a contrast loss calculation module based on the temperature parameter, input the feature similarity matrix of the positive sample pair and the feature similarity matrix of the negative sample pair into the contrast loss calculation module, output the contrast loss value, and generate a sample similarity vector;

[0139] Constructing a hierarchical transfer matrix based on the concept hierarchy in the medical knowledge base, multiplying the annotation vectors of different levels with the hierarchical transfer matrix to obtain a hierarchical mapping matrix, calculating the structural constraint loss of the hierarchical mapping matrix, outputting the consistency loss value, and extracting the hierarchical semantic relationship vector;

[0140] Constructing a dynamic weight generation network, inputting the training state vector into the weight generation network to generate a weight vector, normalizing the weight vector to obtain a dynamic weight coefficient, and performing weighted combination of the dynamic weight coefficient with the classification loss value, the contrast loss value, and the consistency loss value to obtain a total loss function;

[0141] Calculating the gradient of the cross entropy loss parameter according to the total loss function, calculating the gradient of the contrast loss parameter according to the contrast loss value, and calculating the gradient of the hierarchical constraint parameter according to the consistency loss value;

[0142] The momentum value at the previous moment is weighted with the current gradients to obtain the updated momentum, and the corresponding parameters are iteratively optimized using the updated momentum until the total loss function converges. The classification prediction vector, contrast feature vector and hierarchical constraint vector corresponding to the optimized parameters are weightedly fused to obtain a dynamic annotation vector with knowledge constraints.

[0143] Exemplarily, first, prepare a medical image dataset and the corresponding initial annotation information, and build a medical knowledge base. The medical image dataset contains various medical images, such as X-rays, CT scan images, MRI images, etc. The initial annotation information can be a category label at the image level or a segmentation mask at the pixel level. The medical knowledge base contains the hierarchical structure and semantic relationships of medical concepts, such as the classification of diseases, the composition of organs, etc.

[0144] Then, extract the image features. Use a pre-trained convolutional neural network (such as ResNet, Inception, etc.) to extract the feature representation of the image. Suppose there is a chest X-ray. After feature extraction, a 1024-dimensional feature vector is obtained.

[0145] Next, the classification loss is calculated. The extracted image features are input into a fully connected layer with a temperature coefficient, and then the classification loss value is calculated using the cross entropy loss function. The temperature coefficient is used to adjust the smoothness of the predicted distribution, for example, it is set to 0.1. Assuming that the chest X-ray belongs to the "pneumonia" category, the classification loss value is calculated to be 0.5. At the same time, the category prediction vector is calculated based on the classification loss value, for example [0.1, 0.8, 0.1], which represents the probability that the image belongs to three categories ("normal", "pneumonia", "other").

[0146] At the same time, calculate the contrast loss. Select images that belong to the same category as the target image as positive samples, and images of different categories as negative samples. Calculate the feature similarity matrix of positive sample pairs and negative sample pairs. Then, input these similarity matrices into a contrast loss calculation module based on temperature parameters, such as using InfoNCELoss. Assuming that the temperature parameter is set to 0.2, the calculated contrast loss value is 0.2. At the same time, generate a sample similarity vector, such as [0.9, 0.1, 0.2], which indicates the similarity between the image and the three samples.

[0147] In addition, the consistency loss is calculated. A hierarchical transfer matrix is ​​constructed based on the concept hierarchy in the medical knowledge base. For example, "pneumonia" belongs to "lung disease", and "lung disease" belongs to "respiratory disease". The hierarchical transfer matrix can represent these hierarchical relationships. The annotation vectors of different levels are multiplied by the hierarchical transfer matrix to obtain a hierarchical mapping matrix. Then, the structural constraint loss of the hierarchical mapping matrix is ​​calculated, for example, using the mean square error. Assume that the calculated consistency loss value is 0.3. At the same time, a hierarchical semantic relationship vector is extracted, such as [0.7, 0.2, 0.1], which indicates the degree of relevance of the image to the three levels of concepts.

[0148] Dynamically adjust loss weights. Build a dynamic weight generation network, input the training state vector (such as the current number of epochs, the trend of loss value changes, etc.) into the network, and generate a weight vector. Then, normalize the weight vector to get the dynamic weight coefficient. For example, in the early stage of training, the weight of classification loss is larger, while in the later stage of training, the weight of consistency loss is larger. Assume that the current dynamic weight coefficients are [0.5, 0.3, 0.2], corresponding to classification loss, contrast loss, and consistency loss, respectively. Multiply these weight coefficients by the corresponding loss values, and then add them together to get the total loss function.

[0149] Iteratively optimize the model parameters. Calculate the gradients of the cross entropy loss parameter, contrast loss parameter, and hierarchical constraint parameter based on the total loss function. Weight the momentum value at the previous moment with the current gradient to obtain the updated momentum. Then, use the updated momentum to iteratively optimize the corresponding parameters, for example, using the Adam optimizer. Repeat the above steps until the total loss function converges.

[0150] Finally, the dynamic annotation vector is obtained by fusion. The classification prediction vector, contrast feature vector and hierarchical semantic relationship vector corresponding to the optimized parameters are weighted and fused to obtain a dynamic annotation vector with knowledge constraints. For example, the three vectors are multiplied by the weights [0.4, 0.3, 0.3] respectively, and then added to obtain the final dynamic annotation vector.

[0151] In this embodiment, by combining knowledge constraints and multi-task learning, the accuracy of annotation can be effectively improved, especially when the annotation information is incomplete or noisy. The dynamic weight mechanism can adaptively balance the contribution of different loss terms, making the model more robust to different data distributions and noise. This method can mine potential semantic relationships in medical images and integrate them into the annotation process, thereby improving the semantic consistency and interpretability of annotations.

[0152] In an optional embodiment, a decision generation model is constructed based on a deep reinforcement learning framework, the dynamic annotation vector and the treatment response feature are used as state inputs, and the treatment effect in the similar case set is used as a reward signal, and the generation of a diagnosis and treatment recommendation including a treatment plan and early warning information includes:

[0153] Obtain disease diagnosis and treatment guidelines and clinical diagnosis and treatment data, extract treatment rules and pathway information from the disease diagnosis and treatment guidelines, and construct a disease knowledge graph in combination with the clinical diagnosis and treatment data;

[0154] Constructing a multimodal feature fusion network, wherein the multimodal feature fusion network comprises a feature branch channel and an attention guidance module, calculating attention weights of dynamic annotation vectors and treatment response features through the attention guidance module, and fusing features based on the attention weights to obtain a state vector;

[0155] Extracting a treatment path from the disease knowledge graph through a graph attention network, generating a set of candidate treatment plans based on the treatment path, and discretizing continuous treatment parameters in the set of candidate treatment plans to obtain a set of discretized treatment plans;

[0156] Constructing a deep reinforcement learning network, the deep reinforcement learning network comprising a state encoder and an action evaluator, inputting the state vector into the state encoder to obtain a state representation, and inputting the state representation and a treatment plan in the discretized treatment plan set into the action evaluator to obtain a plan value;

[0157] Extracting time-series treatment records from a set of similar cases, inputting the time-series treatment records into a pre-built causal inference model to obtain a degree of correlation of treatment effects, calculating an experience reward value based on the degree of correlation of treatment effects, inputting the set of candidate treatment plans into an expert rule system to obtain a constraint reward value, and combining the experience reward value and the constraint reward value to obtain a reward signal;

[0158] Building a priority experience replay pool based on the reward signal, storing state transition samples in the priority experience replay pool, calculating the importance score of the state transition samples, setting the sampling probability based on the importance score and extracting training samples, and using the training samples to optimize the parameters of the deep reinforcement learning network to obtain a trained decision generation model;

[0159] The current state vector is input into the trained decision generation model, the treatment plan with the maximum plan value is selected from the discretized treatment plan set as the optimal treatment plan, the reliability score of the optimal treatment plan is calculated, risk warning information is generated based on the plan value distribution of the optimal treatment plan, and the optimal treatment plan, reliability score and risk warning information are combined to form a diagnosis and treatment recommendation.

[0160] For example, first of all, obtaining disease diagnosis and treatment guidelines and clinical diagnosis and treatment data is the basis of the entire system. These data come from authoritative medical resources, such as the World Health Organization's disease classification standards (such as ICD), clinical treatment manuals, research results in professional medical journals, etc. In addition, clinical diagnosis and treatment data include patient historical case records, pathology reports, imaging data (such as CT, MRI, etc.) and execution records of treatment plans. Through these data, treatment rules and path information can be extracted to construct a disease knowledge graph. The disease knowledge graph is composed of various aspects of the disease, including the type of disease, symptoms, potential complications, treatment methods, drug information, and the relationship between them. The nodes in the graph represent different disease-related entities (such as "diabetes", "insulin therapy"), while the edges represent the relationship between entities (such as the relationship between "treatment plans"). The construction of the knowledge graph is not only a data integration process, but also provides important support for subsequent treatment path reasoning and decision generation.

[0161] On this basis, a multimodal feature fusion network is constructed, which contains multiple feature branch channels and an attention guidance module. The core task of the multimodal feature fusion network is to process multimodal data from different data sources, such as patient imaging data (such as MRI images), diagnostic records and treatment history in electronic medical records, etc. Each data source is processed through an independent feature branch channel. In this process, convolutional neural network (CNN) is first used to extract features from imaging data; natural language processing (NLP) technology is used to process text data, such as disease descriptions and treatment records in electronic medical records. Then, these features from different sources are weighted through the attention guidance module. This module dynamically adjusts the importance of each input feature in the network by calculating the attention weight of each input feature. For example, if a patient's imaging data plays a key role in the treatment process, the weight of the data will be automatically enhanced, so that the network focuses more on imaging information, and vice versa. Emphasize other data. Finally, the network fuses these weighted features into a unified state vector, representing the patient's current condition and treatment response characteristics.

[0162] Extract treatment paths from the disease knowledge graph through the graph attention network. The graph attention network (GAT) is a neural network that processes graph-structured data. It can better capture the semantic information of nodes and edges in the graph by learning the relative importance and adjacency between nodes. In this step, the graph attention network obtains treatment paths related to the patient's disease from the disease knowledge graph. For example, the system can extract a complete treatment path from the disease graph, such as the possible treatment paths for "diabetes" include "drug therapy" → "diet control" → "insulin injection". This path can cover every step from diagnosis to treatment, and can also dynamically adjust the path selection by considering the patient's specific conditions (such as whether there are complications).

[0163] After generating a set of candidate treatment options, the continuous treatment parameters in these options need to be discretized. The purpose of discretization is to convert continuous treatment options (such as drug dosage, treatment cycle, etc.) into a set of discrete decision options. This makes the treatment options more operational and suitable for training and optimization in deep reinforcement learning models. For example, the continuous value of drug dosage may be discretized into several common dosage options (such as low dose, medium dose, high dose).

[0164] Next, a deep reinforcement learning network is constructed, including a state encoder and an action evaluator. The task of the state encoder is to convert the input state vector (generated by the multimodal feature fusion network) into a compact and expressive state representation. This representation condenses information such as the patient's current condition, treatment history, and treatment response into a form that is easy for the network to process. Then, this state representation and treatment plan (from a set of discretized treatment plans) are input into the action evaluator. The goal of the action evaluator is to evaluate the effectiveness of each candidate treatment plan, that is, their therapeutic value under the current patient state. The system outputs an evaluation value for each treatment plan, indicating the potential effect of the plan.

[0165] To further enhance the model's predictive power, the system extracts time-series treatment records from a collection of similar cases. These treatment records include the treatment pathways of historical cases and their outcomes (such as improvement or deterioration of the condition after drug treatment). By inputting the causal inference model, the system is able to evaluate the potential effects of each treatment from these records. For example, some treatments may have worked better in past cases, while others worked less well, and the system is able to infer the correlation between different treatments based on historical data.

[0166] Based on the correlation of historical treatment records, the system calculates an experience reward value, which represents the long-term effect of the treatment plan. At the same time, the system also constrains candidate treatment plans through the expert rule system to ensure that the treatment plan conforms to medical norms and experience. For example, some drugs may have known side effects, and the expert rule system can adjust the treatment plan based on this known information. By combining these two parts of the reward value, the final reward signal is obtained.

[0167] Based on these reward signals, the system builds a priority experience replay pool. In reinforcement learning, the experience replay pool is used to store the historical learning experience of the model. In the priority experience replay pool, the stored samples are sorted according to their importance scores. The higher the importance score of a sample, the higher its priority in subsequent learning. This means that during training, the model will learn more from key samples in order to improve performance faster.

[0168] Through repeated updates and training, the system eventually obtains an optimized decision-making model. This model can select the most appropriate treatment plan from a set of discrete treatment plans based on the current patient's state vector. This plan has the maximum plan value, that is, it can bring the best treatment effect under the current state. At the same time, the system will also generate risk warning information based on the value distribution of the treatment plan, reminding doctors of possible side effects, uncertainty of treatment response and other risks. Finally, the model will output a comprehensive diagnosis and treatment recommendation, including the optimal treatment plan, reliability score (indicating the reliability of the plan) and risk warning information for doctors to refer to and make the final decision.

[0169] In this embodiment, a personalized treatment plan can be generated based on the individual characteristics and disease dynamics of the patient to improve the treatment effect. The potential risks of the treatment plan can be predicted and early warning information can be provided to help doctors and patients make safer decisions. It can provide doctors with diagnosis and treatment suggestions, assist doctors in making clinical decisions, and improve the efficiency and quality of diagnosis and treatment.

[0170] In an optional embodiment, extracting time series treatment records from a set of similar cases, inputting the time series treatment records into a pre-built causal inference model to obtain a treatment effect correlation, and calculating the experience reward value based on the treatment effect correlation includes:

[0171] Extracting treatment regimen sequences, clinical indicator sequences, symptom score sequences, and adverse reaction records from the similar case set, and combining them to generate a time series treatment record;

[0172] Constructing a structural causal model, taking the treatment plan sequence, the clinical indicator sequence, and the symptom score sequence in the time series treatment record as causal graph nodes, establishing causal edges between the causal graph nodes based on an expert knowledge base and data correlation analysis results, and constructing a dynamic causal graph using time series dependencies;

[0173] Constructing a counterfactual reasoning module, inputting the time series treatment record into the counterfactual reasoning module, processing the feature interaction relationship in the time series treatment record through a gated attention mechanism, obtaining an average treatment effect and a conditional average treatment effect, and constructing a causal effect scoring matrix based on the average treatment effect and the conditional average treatment effect, wherein the row vectors of the causal effect scoring matrix represent different treatment plans, the column vectors represent different clinical indicators, and the matrix elements represent the correlation of treatment effects;

[0174] Performing eigendecomposition on the causal effect score matrix, extracting a main eigenvector, using the main eigenvector as a treatment effect weight vector, and performing weighted combination of the treatment effect weight vector with a clinical indicator sequence, a symptom score sequence, and an adverse reaction record, respectively, to obtain a multi-dimensional reward vector;

[0175] A multi-layer reward distribution network is constructed, including a feature extraction layer, a temporal correlation analysis layer and a reward fusion layer. The multi-dimensional reward vector is input into the feature extraction layer, and the high-frequency noise is removed by wavelet transform to obtain a filtered reward vector. Local features are extracted through a multi-head self-attention mechanism to generate an enhanced feature vector. The enhanced feature vector is input into the temporal correlation analysis layer, and the temporal information is embedded through position encoding. The attention scores of different time steps are calculated, and a temporal dependency matrix is ​​constructed based on the attention scores. The temporal dependency matrix is ​​tensor multiplied with the enhanced feature vector to obtain a temporal weighted feature, and a jump connection is used to add the temporal weighted feature to the filtered reward vector. The feature weight is dynamically adjusted through an adaptive gating unit, and the experience reward value is finally output.

[0176] Exemplarily, first, a collection of cases with similar characteristics (e.g., age, gender, disease diagnosis, course of disease, etc.) to the target case is retrieved from the database. A treatment regimen sequence (e.g., drug name, dosage, route of administration, time of administration), a clinical indicator sequence (e.g., blood pressure, heart rate, body temperature, laboratory test results), a symptom score sequence (e.g., pain score, fatigue score, anxiety score), and adverse reaction records (e.g., nausea, vomiting, rash, headache) are extracted from the electronic medical records of these similar cases. The extracted information is combined to generate a time series treatment record. For example, the treatment regimen, clinical indicators, symptom scores, and adverse reaction records at each time point can be organized into a vector, and the vectors of all time points form a matrix, representing a complete time series treatment record of a case. Assuming that a case has three time points, the treatment regimen is drug A and drug B, and the clinical indicators are blood pressure and heart rate, the time series treatment record can be represented as a 3x4 matrix, where each row represents a time point and each column represents a feature.

[0177] Next, construct a structural causal model. Create a causal graph with treatment options, clinical indicators, and symptom scores as nodes in the graph. Establish causal edges between these nodes based on the medical expert knowledge base and data correlation analysis results. For example, if drug A lowers blood pressure, establish a directed edge between the drug A node and the blood pressure node, with the arrow pointing to the blood pressure node. In order to reflect temporal dependencies, use a dynamic causal graph. For example, blood pressure at the current time point may be affected by blood pressure and treatment options at the previous time point, so a directed edge is established between the corresponding nodes.

[0178] Then, a counterfactual reasoning module is constructed. The time-series treatment records are input into the counterfactual reasoning module, which uses a gated attention mechanism to process the feature interaction relationship in the time-series treatment records. For example, the effect of drug A on blood pressure may be different at different time points, and the gated attention mechanism can capture this dynamic relationship. The counterfactual reasoning module outputs the average treatment effect and the conditional average treatment effect. For example, the average treatment effect can be expressed as the average reduction in blood pressure for all patients after taking drug A, while the conditional average treatment effect can be expressed as the average reduction in blood pressure for patients in a specific age group after taking drug A. A causal effect score matrix is ​​constructed based on the average treatment effect and the conditional average treatment effect. The row vectors of the matrix represent different treatment regimens, the column vectors represent different clinical indicators, and the matrix elements represent the degree of association of treatment effects. For example, the elements corresponding to drug A and blood pressure in the matrix represent the degree of effect of drug A on blood pressure. Assuming there are two treatment regimens and two clinical indicators, the causal effect score matrix can be a 2x2 matrix.

[0179] Perform eigendecomposition on the causal effect score matrix and extract the main eigenvector. Use the main eigenvector as the treatment effect weight vector. Perform weighted combination of the treatment effect weight vector with the clinical indicator sequence, symptom score sequence, and adverse reaction record to obtain a multi-dimensional reward vector. For example, multiplying the weight vector with the blood pressure sequence can obtain the reward value brought by the blood pressure change.

[0180] Finally, a multi-layer reward distribution network is constructed. The network includes a feature extraction layer, a temporal correlation analysis layer, and a reward fusion layer. The multi-dimensional reward vector is input into the feature extraction layer, and the wavelet transform is used to remove high-frequency noise to obtain a filtered reward vector. The local features are extracted through the multi-head self-attention mechanism to generate an enhanced feature vector. The enhanced feature vector is input into the temporal correlation analysis layer, the temporal information is embedded through position encoding, the attention scores of different time steps are calculated, and the temporal dependency matrix is ​​constructed based on the attention score. The temporal dependency matrix is ​​tensor multiplied with the enhanced feature vector to obtain the temporal weighted feature. The temporal weighted feature is added to the filtered reward vector using a jump connection, the feature weight is dynamically adjusted through the adaptive gating unit, and the experience reward value is finally output.

[0181] In this embodiment, by constructing a causal graph and a counterfactual reasoning module, the actual treatment effects of different treatment plans on patients can be more accurately evaluated to avoid confusing correlation and causality. Based on the individual characteristics and treatment history of the patient, a more effective personalized treatment plan can be recommended to improve the success rate of treatment. By analyzing the correlation between causality and treatment effect, new treatment strategies and potential drug targets can be discovered to promote the development of the medical field.

[0182] Another embodiment of the present application is as follows:

[0183] Using deep neural networks, we process patients’ text data, imaging data, electrocardiogram (ECG), and electroencephalogram (EEG) data, build a multimodal data model, and perform modal feature extraction and vector coding. The technical solution mainly includes:

[0184] Convert text into vectors through word embedding (including Word2Vec, GloVe, etc.). For example, the patient's condition description, symptoms and other information can be converted into a word vector representation to reflect its semantic characteristics.

[0185] Process image data (such as X-ray or CT images of injured areas) using convolutional neural networks (CNNs) for feature extraction and generating vector representations of images (using pre-trained networks such as ResNet for feature extraction).

[0186] Speech data can be converted into text through a speech recognition model, and then a text vector is generated through a text embedding method; at the same time, audio feature extraction methods (such as MFCC) are used to convert sound into an audio vector.

[0187] Physiological signals can be processed through time series data analysis methods (such as RNN, LSTM, etc.) and converted into vector representations to capture the patient's vital signs, including processing electrocardiogram, electroencephalogram, pulse, and body temperature.

[0188] A vector space model (VSM) of emergency patients is established, and vectors generated in different modalities are subjected to various calculations and operations in multidimensional space to complete the fusion of multimodal data. Depending on the data type and the different life cycles of the data, the following fusion methods are used in combination:

[0189] 1. Weighted method: Assign different weights to different types of data (text, image, audio, etc.), and combine them into a final vector representation according to the weights. This method is suitable for scenarios where different modalities have different importance for label generation.

[0190] 2. Series method: connect the vectors of multiple modes in sequence to generate a long vector. This method can retain the characteristics of each mode. This method is suitable for fusing feature vectors with stable structures.

[0191] 3. Attention method: Using the attention mechanism in deep learning, dynamic weights are assigned to data of different modalities. In the process of label generation, the fusion strategy can be adjusted according to the actual importance of the input data. This method is more suitable for dynamic models with real-time data updates.

[0192] Generate labels for emergency patients, and the real-time update module can automatically update labels according to changes in the patient's condition.

[0193] 1. Choose different generation models according to the label type: For labels with clear rules and high interpretability requirements, traditional classifiers (such as support vector machines, decision trees, neural networks, etc.) are used to generate corresponding labels based on feature vectors. For prediction classes or uncertainty labels, generation models (including Seq2Seq and Transformer) can be used to generate labels based on the merged multimodal vectors.

[0194] 2. Since the condition of emergency patients can change rapidly, the vector space method can also support real-time label updates. By collecting new multimodal data in real time (for example, vital sign monitoring data: decreased blood oxygen or worsening dyspnea), the system can update the patient's label information in real time to reflect the patient's current emergency needs.

[0195] Figure 2 is a system flow chart of an embodiment of the present invention, such as Figure 2 As shown, the system comprises:

[0196] The system first receives input data from multiple sources, including medical images, electronic medical records, and physiological data. Each type of data is preprocessed through a feature encoding module to extract core feature information. The extracted feature information is mapped to an SVM (support vector machine) graph structure, where the nodes represent different key entities, such as patients, diseases, images, and signs, and the edges between the nodes represent the associations between these entities.

[0197] Based on the graph structure, the system processes it through a multi-scale feature extraction module, using multiple convolutional layers and fusion modules to extract features at different levels and capture multi-scale information representation. Subsequently, the extracted features are weighted through the attention mechanism module to enhance the influence of important features. The weighted features are further propagated in the graph through a graph neural network (GNN) to integrate the semantic relationship between nodes.

[0198] Finally, the features processed by the graph neural network are passed to the decision module for feature aggregation. After integrating all important information, the system generates personalized optimization decisions. This process is suitable for in-depth analysis and optimization of complex data, especially in the medical field, and can effectively support the implementation of precision medicine and personalized services.

[0199] Figure 3 FIG. 1 is a schematic diagram of a structure of a multimodal data dynamic fusion and annotation system for medical decision-making according to an embodiment of the present invention. Figure 3 As shown, the system comprises:

[0200] The first unit is used to collect the patient's medical imaging data, electronic medical record data and physiological monitoring data, and use the pre-built three-dimensional lesion feature extraction module to perform hierarchical perception on the medical imaging data to obtain the three-dimensional contour of the lesion area, calculate the volume change rate and boundary diffusion of the three-dimensional contour, and extract the density distribution characteristics and texture evolution characteristics of the lesion area; use the hierarchical entity recognition model to extract the disease type characteristics, symptom description characteristics and treatment record characteristics from the electronic medical record data, perform noise reduction on the physiological monitoring data and then extract the physiological index fluctuation characteristics, and form a multimodal feature sample set with the density distribution characteristics, texture evolution characteristics, disease type characteristics, symptom description characteristics, treatment record characteristics and physiological index fluctuation characteristics;

[0201] The second unit is used to construct a dynamic fusion annotation model, which receives the multimodal feature sample set, encodes various features in the multimodal feature sample set through a feature encoding network to obtain a feature vector, calculates a similarity weight matrix between the feature vectors, constructs a feature association graph according to the similarity weight matrix, uses a graph convolution network to perform feature aggregation on the feature association graph to obtain a fused feature vector, and annotates the fused feature vector in combination with the diagnosis and treatment specifications in the medical knowledge base to generate a dynamic annotation vector;

[0202] The third unit is used to input the dynamic annotation vector into the medical decision support model, obtain a similar case set based on the similarity analysis between the dynamic annotation vector and the historical cases, perform time series analysis on the diagnosis and treatment records in the similar case set to obtain the disease development law and treatment response characteristics, build a decision generation model based on the deep reinforcement learning framework, use the dynamic annotation vector and the treatment response characteristics as state inputs, use the treatment effect in the similar case set as a reward signal, generate diagnosis and treatment recommendations including treatment plans and early warning information, feed back the newly generated diagnosis and treatment records and their treatment effects to the dynamic fusion annotation model for online updating, and store the diagnosis and treatment recommendations in the medical knowledge base for subsequent case analysis.

[0203] According to a third aspect of the embodiments of the present invention,

[0204] An electronic device is provided, comprising:

[0205] processor;

[0206] a memory for storing processor-executable instructions;

[0207] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0208] A fourth aspect of the embodiments of the present invention is:

[0209] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0210] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0211] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal data dynamic fusion and annotation method for medical decision-making, characterized in that: include: Collect the patient's medical imaging data, electronic medical record data and physiological monitoring data, use the pre-built three-dimensional lesion feature extraction module to perform hierarchical perception on the medical imaging data, obtain the three-dimensional contour of the lesion area, calculate the volume change rate and boundary diffusion of the three-dimensional contour, and extract the density distribution characteristics and texture evolution characteristics of the lesion area; use the hierarchical entity recognition model to extract disease type characteristics, symptom description characteristics and treatment record characteristics from the electronic medical record data, extract physiological indicator fluctuation characteristics after denoising the physiological monitoring data, and form a multimodal feature sample set with the density distribution characteristics, texture evolution characteristics, disease type characteristics, symptom description characteristics, treatment record characteristics and physiological indicator fluctuation characteristics; Constructing a dynamic fusion annotation model, the dynamic fusion annotation model receives the multimodal feature sample set, encodes various features in the multimodal feature sample set through a feature encoding network to obtain a feature vector, calculates a similarity weight matrix between the feature vectors, constructs a feature association graph according to the similarity weight matrix, uses a graph convolutional network to perform feature aggregation on the feature association graph to obtain a fused feature vector, annotates the fused feature vector in combination with the diagnosis and treatment specifications in the medical knowledge base, and generates a dynamic annotation vector; The dynamic annotation vector is input into the medical decision support model, and a similar case set is obtained based on the similarity analysis between the dynamic annotation vector and historical cases. The diagnosis and treatment records in the similar case set are subjected to time series analysis to obtain the disease development law and treatment response characteristics. A decision generation model is constructed based on a deep reinforcement learning framework, and the dynamic annotation vector and the treatment response characteristics are used as state inputs. The treatment effect in the similar case set is used as a reward signal to generate diagnosis and treatment recommendations including treatment plans and early warning information. The newly generated diagnosis and treatment records and their treatment effects are fed back to the dynamic fusion annotation model for online updating, and the diagnosis and treatment recommendations are stored in a medical knowledge base for subsequent case analysis.

2. The method according to claim 1, characterized in that The medical image data is layered sensed using a pre-built three-dimensional lesion feature extraction module to obtain a three-dimensional contour of the lesion area, calculate the volume change rate and boundary diffusion of the three-dimensional contour, and extract the density distribution characteristics and texture evolution characteristics of the lesion area, including: Performing histogram matching on the medical image data to obtain standardized image data, and performing data integrity check on the standardized image data to obtain preprocessed medical image data; A three-dimensional lesion extraction module is constructed, which includes an encoder and a decoder. The encoder is provided with multiple groups of residual-connected three-dimensional convolution units, each group of three-dimensional convolution units has different convolution kernel sizes and channels, and is used to extract multi-scale spatial features from preprocessed medical image data; the decoder performs feature fusion on the multi-scale spatial features through deconvolution operations, establishes dense skip connections between feature layers of the same resolution, and obtains three-dimensional contour data of the lesion area in combination with a spatial attention mechanism; Adaptively tetrahedral meshing is performed on the three-dimensional contour data to obtain a time-series voxel model, and the boundary of the time-series voxel model is tracked by a level set evolution method, and the voxel topological structure changes at adjacent time points are extracted to obtain the volume change rate; a boundary evolution model based on curvature flow is constructed, and the boundary evolution model is combined with a regional growing algorithm to calculate the boundary diffusion of the lesion area; A region of interest is determined based on the volume change rate and the boundary diffusion, and an adaptive three-dimensional filter group is constructed. The region of interest is decomposed at multiple scales by adjusting the direction parameters and scale parameters of the adaptive three-dimensional filter group to obtain a three-dimensional texture feature map. The local statistics of the three-dimensional texture feature map are calculated to obtain density distribution characteristics. At the same time, a three-dimensional grayscale co-occurrence matrix considering spatial position constraints is constructed according to the region of interest. The spatial position information is introduced as a constraint condition into the calculation process of the three-dimensional grayscale co-occurrence matrix, and the grayscale co-occurrence features of the region of interest are extracted. The local structural features are obtained by combining the local binary pattern operator. The grayscale co-occurrence features and the local structural features are correlated and clustered in the time dimension to obtain texture evolution features.

3. The method according to claim 2, characterized in that Constructing a boundary evolution model based on curvature flow, combining the boundary evolution model with the region growing algorithm, and calculating the boundary diffusion of the lesion area includes: Constructing a velocity field term according to the volume change rate, and combining the velocity field term with a curvature term and an image gradient term to construct a curvature flow boundary evolution model; Calculating local boundary energy, constructing an exponential decay function according to the local boundary energy to obtain a weight adjustment factor, and using the weight adjustment factor to dynamically adjust the weights of each component of the curvature flow boundary evolution model to obtain an adaptive evolution model; Determine the initial boundary point set based on the adaptive evolution model, calculate the grayscale mean and standard deviation of the current region to construct a density similarity term, determine the local homogeneity term in combination with the neighborhood pixel variance of the boundary point set, map the local deviation of the volume change rate into a temporal consistency term, and combine the density similarity term, the local homogeneity term and the temporal consistency term to obtain a regional growing cost function; Searching for candidate growth points in the neighborhood of the initial boundary point set, weightedly fusing the gradient information of the regional growth cost function with the adaptive evolution model, calculating the fusion weight coefficient according to the curvature value, gradient intensity and local contrast at the boundary point, and obtaining a spatiotemporal fusion boundary evolution equation; Iteratively calculating the boundary displacement field of adjacent time points using the spatiotemporal fusion boundary evolution equation, constructing a diffusion tensor characterizing local motion characteristics based on the boundary displacement field; performing eigenvalue decomposition on the diffusion tensor, and calculating the directional intensity index of boundary diffusion according to the ratio of eigenvalues; An attenuation function based on the volume change rate is constructed, the directional strength index is multiplied by the attenuation function to obtain a comprehensive diffusion measurement value, a feature map characterizing the boundary diffusion behavior is generated according to the comprehensive diffusion measurement value, the lesion area is segmented based on the feature map, the boundary contour of the segmented area is calculated, and the boundary diffusion degree of the lesion area is determined by combining the directional strength index and the volume change rate.

4. The method according to claim 1, characterized in that The feature association graph is aggregated using a graph convolutional network to obtain a fused feature vector, and the fused feature vector is annotated in combination with the diagnosis and treatment specifications in the medical knowledge base to generate a dynamic annotation vector, including: Construct a multi-layer graph convolution network to perform feature aggregation on the feature association graph. Each layer of graph convolution includes a spatial attention branch and a channel attention branch. The spatial attention branch calculates the attention weight matrix of the neighborhood nodes. The channel attention branch generates the weight vector of the feature channel. The attention weight matrix and the weight vector are weighted with the input features to obtain dual-branch features. The dual-branch features and the input features are combined through cross-layer connections to generate enhanced features. The enhanced features are input into the next layer of graph convolution network to obtain multi-layer node features in sequence. Constructing a feature pyramid for the multi-layer node features to generate a multi-scale feature sequence, calculating an inter-layer attention map based on the multi-scale feature sequence, converting the inter-layer attention map into a similarity matrix, calculating the fusion weight of the inter-layer features according to the similarity matrix, and performing weighted combination on the multi-layer node features to obtain a fused feature representation; Construct a knowledge constraint network with a dual network structure, wherein the first sub-network is a concept association network, which constructs a concept graph based on the concept system in the medical knowledge base, and performs relational constraints on the fused feature representation; the second sub-network is a rule reasoning network, which converts medical norms into reasoning rules, performs rule constraints on the fused feature representation, and outputs knowledge enhancement features; Constructing a progressive annotation hierarchy based on the knowledge enhancement feature, constructing a feature comparison module at each annotation hierarchy, the feature comparison module calculating positive sample pairs of concept features at the same level and negative sample pairs of concept features at different levels, generating a comparison feature representation, extracting hierarchical structure information from the concept system of the medical knowledge base, and constructing an annotation consistency constraint based on the hierarchical structure information; A joint optimization strategy is designed to calculate the consistency loss based on the annotation consistency constraint, and a multi-task learning framework is constructed by combining the classification loss and contrast loss. The contributions of different loss terms are balanced through dynamic weights, and the model parameters are iteratively updated using an alternating optimization algorithm to obtain a dynamic annotation vector that integrates knowledge constraints.

5. The method according to claim 4, characterized in that A joint optimization strategy is designed to calculate the consistency loss based on the annotation consistency constraint. A multi-task learning framework is constructed by combining the classification loss and contrast loss. The contribution of different loss items is balanced through dynamic weights. The model parameters are iteratively updated using an alternating optimization algorithm. The dynamic annotation vector that integrates knowledge constraints is obtained, including: Inputting the contrast feature representation into a cross entropy loss function with a temperature coefficient, wherein the temperature coefficient is used to adjust the smoothness of the prediction distribution, outputting a classification loss value, and calculating a category prediction vector based on the classification loss value; Calculate the feature similarity matrix of the positive sample pair and the feature similarity matrix of the negative sample pair, construct a contrast loss calculation module based on the temperature parameter, input the feature similarity matrix of the positive sample pair and the feature similarity matrix of the negative sample pair into the contrast loss calculation module, output the contrast loss value, and generate a sample similarity vector; Constructing a hierarchical transfer matrix based on the concept hierarchy in the medical knowledge base, multiplying the annotation vectors of different levels with the hierarchical transfer matrix to obtain a hierarchical mapping matrix, calculating the structural constraint loss of the hierarchical mapping matrix, outputting the consistency loss value, and extracting the hierarchical semantic relationship vector; Constructing a dynamic weight generation network, inputting the training state vector into the weight generation network to generate a weight vector, normalizing the weight vector to obtain a dynamic weight coefficient, and performing weighted combination of the dynamic weight coefficient with the classification loss value, the contrast loss value, and the consistency loss value to obtain a total loss function; Calculating the gradient of the cross entropy loss parameter according to the total loss function, calculating the gradient of the contrast loss parameter according to the contrast loss value, and calculating the gradient of the hierarchical constraint parameter according to the consistency loss value; The momentum value at the previous moment is weighted with the current gradients to obtain the updated momentum, and the corresponding parameters are iteratively optimized using the updated momentum until the total loss function converges. The classification prediction vector, contrast feature vector and hierarchical constraint vector corresponding to the optimized parameters are weightedly fused to obtain a dynamic annotation vector with knowledge constraints.

6. The method according to claim 1, characterized in that A decision generation model is constructed based on a deep reinforcement learning framework, the dynamic annotation vector and the treatment response feature are used as state inputs, and the treatment effect in the similar case set is used as a reward signal to generate diagnosis and treatment recommendations containing treatment plans and early warning information, including: Obtain disease diagnosis and treatment guidelines and clinical diagnosis and treatment data, extract treatment rules and pathway information from the disease diagnosis and treatment guidelines, and construct a disease knowledge graph in combination with the clinical diagnosis and treatment data; Constructing a multimodal feature fusion network, wherein the multimodal feature fusion network comprises a feature branch channel and an attention guidance module, calculating attention weights of dynamic annotation vectors and treatment response features through the attention guidance module, and fusing features based on the attention weights to obtain a state vector; Extracting a treatment path from the disease knowledge graph through a graph attention network, generating a set of candidate treatment plans based on the treatment path, and discretizing continuous treatment parameters in the set of candidate treatment plans to obtain a set of discretized treatment plans; Constructing a deep reinforcement learning network, the deep reinforcement learning network comprising a state encoder and an action evaluator, inputting the state vector into the state encoder to obtain a state representation, and inputting the state representation and a treatment plan in the discretized treatment plan set into the action evaluator to obtain a plan value; Extracting time-series treatment records from a set of similar cases, inputting the time-series treatment records into a pre-built causal inference model to obtain a degree of correlation of treatment effects, calculating an experience reward value based on the degree of correlation of treatment effects, inputting the set of candidate treatment plans into an expert rule system to obtain a constraint reward value, and combining the experience reward value and the constraint reward value to obtain a reward signal; Building a priority experience replay pool based on the reward signal, storing state transition samples in the priority experience replay pool, calculating the importance score of the state transition samples, setting the sampling probability based on the importance score and extracting training samples, and using the training samples to optimize the parameters of the deep reinforcement learning network to obtain a trained decision generation model; The current state vector is input into the trained decision generation model, the treatment plan with the maximum plan value is selected from the discretized treatment plan set as the optimal treatment plan, the reliability score of the optimal treatment plan is calculated, risk warning information is generated based on the plan value distribution of the optimal treatment plan, and the optimal treatment plan, reliability score and risk warning information are combined to form a diagnosis and treatment recommendation.

7. The method according to claim 6, characterized in that Extracting time series treatment records from a similar case set, inputting the time series treatment records into a pre-built causal inference model to obtain a treatment effect correlation, and calculating an experience reward value based on the treatment effect correlation includes: Extracting treatment regimen sequences, clinical indicator sequences, symptom score sequences, and adverse reaction records from the similar case set, and combining them to generate a time series treatment record; Constructing a structural causal model, taking the treatment plan sequence, the clinical indicator sequence, and the symptom score sequence in the time series treatment record as causal graph nodes, establishing causal edges between the causal graph nodes based on an expert knowledge base and data correlation analysis results, and constructing a dynamic causal graph using time series dependencies; Constructing a counterfactual reasoning module, inputting the time series treatment record into the counterfactual reasoning module, processing the feature interaction relationship in the time series treatment record through a gated attention mechanism, obtaining an average treatment effect and a conditional average treatment effect, and constructing a causal effect scoring matrix based on the average treatment effect and the conditional average treatment effect, wherein the row vectors of the causal effect scoring matrix represent different treatment plans, the column vectors represent different clinical indicators, and the matrix elements represent the correlation of treatment effects; Performing eigendecomposition on the causal effect score matrix, extracting a main eigenvector, using the main eigenvector as a treatment effect weight vector, and performing weighted combination of the treatment effect weight vector with a clinical indicator sequence, a symptom score sequence, and an adverse reaction record, respectively, to obtain a multi-dimensional reward vector; A multi-layer reward distribution network is constructed, including a feature extraction layer, a temporal correlation analysis layer and a reward fusion layer. The multi-dimensional reward vector is input into the feature extraction layer, and the high-frequency noise is removed by wavelet transform to obtain a filtered reward vector. Local features are extracted through a multi-head self-attention mechanism to generate an enhanced feature vector. The enhanced feature vector is input into the temporal correlation analysis layer, and the temporal information is embedded through position encoding. The attention scores of different time steps are calculated, and a temporal dependency matrix is ​​constructed based on the attention scores. The temporal dependency matrix is ​​tensor multiplied with the enhanced feature vector to obtain a temporal weighted feature, and a jump connection is used to add the temporal weighted feature to the filtered reward vector. The feature weight is dynamically adjusted through an adaptive gating unit, and the experience reward value is finally output.

8. A multimodal data dynamic fusion and annotation system for medical decision-making, used to implement the method described in any one of claims 1 to 7, characterized in that: include: The first unit is used to collect the patient's medical imaging data, electronic medical record data and physiological monitoring data, and use the pre-built three-dimensional lesion feature extraction module to perform hierarchical perception on the medical imaging data to obtain the three-dimensional contour of the lesion area, calculate the volume change rate and boundary diffusion of the three-dimensional contour, and extract the density distribution characteristics and texture evolution characteristics of the lesion area; use the hierarchical entity recognition model to extract the disease type characteristics, symptom description characteristics and treatment record characteristics from the electronic medical record data, perform noise reduction on the physiological monitoring data and then extract the physiological index fluctuation characteristics, and form a multimodal feature sample set with the density distribution characteristics, texture evolution characteristics, disease type characteristics, symptom description characteristics, treatment record characteristics and physiological index fluctuation characteristics; The second unit is used to construct a dynamic fusion annotation model, which receives the multimodal feature sample set, encodes various features in the multimodal feature sample set through a feature encoding network to obtain a feature vector, calculates a similarity weight matrix between the feature vectors, constructs a feature association graph according to the similarity weight matrix, uses a graph convolution network to perform feature aggregation on the feature association graph to obtain a fused feature vector, and annotates the fused feature vector in combination with the diagnosis and treatment specifications in the medical knowledge base to generate a dynamic annotation vector; The third unit is used to input the dynamic annotation vector into the medical decision support model, obtain a similar case set based on the similarity analysis between the dynamic annotation vector and the historical cases, perform time series analysis on the diagnosis and treatment records in the similar case set to obtain the disease development law and treatment response characteristics, build a decision generation model based on the deep reinforcement learning framework, use the dynamic annotation vector and the treatment response characteristics as state inputs, use the treatment effect in the similar case set as a reward signal, generate diagnosis and treatment recommendations including treatment plans and early warning information, feed back the newly generated diagnosis and treatment records and their treatment effects to the dynamic fusion annotation model for online updating, and store the diagnosis and treatment recommendations in the medical knowledge base for subsequent case analysis.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method described in any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images