Large model training system and method based on multimodal medical data fusion

Abstract

The present invention relates to the field of artificial intelligence technology, and in particular to a large-model training system and method based on multimodal medical data fusion. The method comprises: collecting multimodal medical data, extracting and precalculating features of the multimodal medical data; fusing and analyzing the information discrimination value evaluation of sample features and the label association strength to obtain a contextual adaptability adjustment factor of the sample data; performing modal consistency analysis and modal significance analysis on multimodal medical data samples to obtain an information value deepening adjustment factor of the sample data; optimizing and adjusting an original attention score using the contextual adaptability adjustment factor and the information value deepening adjustment factor to obtain a final attention score; and fusing multimodal data features and performing model training using the final attention score, thereby improving the accuracy of the model in learning medical knowledge and improving the final performance of the model trained by the training system on the target task.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a large model training system and method based on multimodal medical data fusion. Background Art

[0002] Multimodal feature fusion is a fundamental and critical technology in the application of artificial intelligence in healthcare, particularly for tasks involving multiple information sources (such as medical images, text reports, and electronic medical records). This technology aims to integrate feature information extracted from independent modal data through specific algorithms or strategies to generate a unified feature representation that comprehensively reflects the sample information. This unified representation serves as a direct input for subsequent machine learning models (such as classifiers, segmentation models, and generative models) that perform complex analysis, prediction, or decision-making. Currently, common approaches to achieving multimodal feature fusion include: first, directly concatenating the feature vectors extracted from each modality along the dimension to form a longer combined vector; second, performing element-wise mathematical operations on the feature vectors of each modality, such as element-wise addition, multiplication, or averaging; and third, employing methods based on attention mechanisms. These methods learn the importance weights of different modal features based on the data content and perform a weighted sum of the features based on these weights, resulting in a dynamically adjusted fused representation. Fusion methods based on attention mechanisms are considered relatively advanced and effective due to their ability to adaptively focus on more relevant modal information.

[0003] However, existing multimodal feature fusion techniques, particularly those based on attention mechanisms, suffer from a technical limitation when calculating the contribution of each modality to the final fused representation. Attention mechanisms typically generate a raw attention score by calculating the similarity between a query vector (query) and a key vector (key) derived from features from each modality. This score is then normalized to form the final attention weight. This core mechanism, which relies solely on numerical similarity between vectors to determine information contribution, fails to fully account for the inherently complex nature of medical multimodal data, namely, the inherent differences in the reliability of data signals and the relative criticality of information content. On the one hand, a feature from one modality may generate a strong numerical signal due to noise, artifacts, or low-quality representations in the raw data (e.g., blurred image areas, templated or nonspecific descriptions in reports). This signal may accidentally generate a high similarity score with the query vector or the key vector of another modality, but it does not reflect real, valuable information. The standard attention score calculation process lacks a step to assess the reliability of the source of this signal, making it easily misled by such "spurious" signals and assigning inappropriately high attention to low-quality or noisy sources. On the other hand, even if the signal source is reliable, the information value carried by different modalities in a specific sample is not equal; a high similarity score may correspond to a common description of general background information or a normal state (constituting information redundancy), or it may correspond to a precise description of a key, diagnostically distinguishing pathological feature (constituting key information). Existing similarity-based attention score calculation mechanisms are inherently unable to effectively distinguish the information value differences behind these two "high-similarity" scenarios. As a result, attention resources may be evenly distributed or incorrectly focused on redundant information, while failing to fully amplify the truly critical and decision-making modal information.

[0004] In summary, the core link of the existing attention fusion mechanism, which only calculates the attention score based on vector similarity, is unable to simultaneously evaluate the reliability of the signal and distinguish the criticality and redundancy of information, resulting in insufficient accuracy and effectiveness of attention allocation. The resulting fused feature representation fails to optimally focus on real and critical medical information, which directly limits the performance potential of downstream artificial intelligence systems that rely on this representation for learning and reasoning. Summary of the Invention

[0005] In order to solve the technical problems of insufficient accuracy and effectiveness of the above-mentioned attention allocation, the present invention aims to propose a large model training system and method based on multimodal medical data fusion to improve the accuracy of vector weight evaluation in the attention mechanism.

[0006] To achieve the above object, the technical solution of the present invention is achieved as follows:

[0007] In a first aspect, the present application provides a large model training method based on multimodal medical data fusion, the method comprising the following steps:

[0008] Step S1: Collect multimodal medical data, and perform feature extraction and pre-calculation on the multimodal medical data;

[0009] Step S2: Obtaining the contextual adaptability adjustment factor of the sample data by evaluating and integrating the distinguishing value of the sample feature information of the medical data and the correlation strength of the medical data labels;

[0010] Step S3: Performing modal consistency analysis and modal significance analysis on the multimodal medical data samples to obtain the information value deepening adjustment factor of the sample data;

[0011] Step S4: Optimize and adjust the original attention score through the context adaptability adjustment factor and the information value deepening adjustment factor to obtain the final attention score;

[0012] Step S5: Perform multimodal data feature fusion and model training through the final attention score.

[0013] Furthermore, according to the above-mentioned multimodal medical data, feature extraction and pre-calculation of the multimodal medical data are performed, specifically including:

[0014] Collect multimodal medical data samples in the medical information system and perform desensitization processing to construct a multimodal medical data set; for any sample in the multimodal medical data set, obtain its medical imaging data, text diagnosis report data and target label data; perform standardization preprocessing on the medical imaging data and input it into a pretrained deep convolutional neural network to extract image feature vectors; perform text preprocessing on the text diagnosis report data and input it into a pretrained language model to extract text feature vectors; classify all samples according to the target label, and for any target category and any modality, calculate and store the modal feature mean vector and covariance matrix of the samples in the category; for any modality, calculate and store the average L2 norm of the feature vector of the modality on all samples in the data set.

[0015] Furthermore, the contextual adaptability adjustment factor of the sample data is obtained by evaluating and integrating the distinguishing value of the sample characteristic information of the medical data and the correlation strength of the medical data labels, including:

[0016] Acquire the characteristic vectors of sample data in a multimodal medical dataset and the categories to which the characteristic vectors of the sample data belong; obtain a first typicality evaluation of the sample data by performing an intra-class difference analysis on the characteristic vectors of the sample data; obtain a first heterogeneous separation evaluation of the sample data by performing a heterogeneous center difference analysis on the characteristic vectors of the sample data; obtain a first discrimination degree of the sample data by performing a discrimination value analysis on the first typicality evaluation and the first heterogeneous separation evaluation of the sample data; obtain the label association strength of the sample data by performing a target label association analysis on the characteristic vectors of the sample data; obtain a contextual adaptability adjustment factor of the sample data by performing a fusion analysis on the first discrimination degree of the sample data and the label association strength.

[0017] Further, according to the above, performing intra-class difference analysis on the characteristic vectors of the sample data to obtain a first typicality evaluation of the sample data; performing heterogeneous center difference analysis on the characteristic vectors of the sample data to obtain a first heterogeneous separation evaluation of the sample data; and performing discrimination value analysis on the first typicality evaluation and the first heterogeneous separation evaluation of the sample data to obtain a first discrimination degree of the sample data, specifically including:

[0018] Obtain the characteristic vector of the target sample, call the mean vector and covariance matrix of its target category, evaluate the multidimensional distance between the three, standardize the evaluation results according to the characteristic dimension, and obtain a first typicality evaluation; compare the characteristic vector of the target sample with the mean vectors of all other target categories one by one, select the minimum distance and take its square value to obtain a first heterogeneous separation evaluation; obtain the atypical threshold of the target category, use the first heterogeneous separation evaluation as the numerator, the constant one and the sum of the first typicality evaluation as the denominator to form a fraction, which is used as a first discrimination degree evaluation factor; subtract the difference between the first typicality evaluation and the threshold value through Sigmoid function mapping to obtain a second discrimination degree evaluation factor; multiply the first discrimination degree evaluation factor by the second discrimination degree evaluation factor to obtain the first discrimination degree of the sample data.

[0019] Furthermore, the target label association analysis is performed on the feature vector of the sample data to obtain the label association strength of the sample data, and the context adaptability adjustment factor of the sample data is obtained by fusion analysis of the first discrimination degree of the sample data and the label association strength, which specifically includes:

[0020] The distance between the sample data and the nearest heterogeneous category mean vector and the distance to the category mean vector are obtained respectively, and the label association strength of the sample data is determined by the ratio of the two; a first discrimination degree non-negative fusion weight coefficient and a label association strength non-negative fusion weight coefficient are set, the first discrimination degree and the label association strength are logarithmically mapped respectively, and weighted summation is performed according to the weight coefficient to obtain a first fusion evaluation; the first fusion evaluation is subjected to hyperbolic tangent mapping and output as the context adaptability adjustment factor of the sample data.

[0021] Furthermore, by performing modal consistency analysis and modal significance analysis on multimodal medical data samples, the information value deepening adjustment factor of the sample data is obtained, which specifically includes:

[0022] Acquire the eigenvectors of sample data in a multimodal medical dataset, and use the average cosine similarity between the eigenvector of any dimension of the sample data and the eigenvectors of all other dimensions of the sample data as the inter-modal similarity of the sample data; perform a significance assessment on the eigenvectors of the sample data to obtain the relative significance of the sample data; and perform a fusion assessment of the inter-modal similarity and relative significance of the sample data to obtain the information value deepening adjustment factor of the sample data.

[0023] Furthermore, the relative significance of the sample data is obtained by performing significance evaluation on the characteristic vector of the sample data, including:

[0024] Obtain the eigenvector of the sample data and the average L2 norm of the eigenvectors of any modality of the sample data on all samples of the multimodal medical dataset; divide the L2 norm of the eigenvector of any modality of the sample data by the average L2 norm of the eigenvectors of the modality on all samples of the multimodal medical dataset, and use the result as the relative significance of the sample data.

[0025] Furthermore, by fusion evaluation of the inter-modal similarity and relative significance of the sample data, an information value deepening adjustment factor of the sample data is obtained, which specifically includes:

[0026] An adjustment weight for the relative significance influence is set; a result obtained by adding a constant 1 to the degree of similarity between modalities of the sample data is used as a first adjustment factor; a result obtained by multiplying the adjustment weight for the relative significance influence, the relative significance of the sample data, and the degree of similarity between modalities of the sample data is used as a second adjustment factor; a result obtained by mapping the result obtained by adding the first adjustment factor and the second adjustment factor to a hyperbolic tangent function is used as a first mapping evaluation; and a result obtained by adding the constant 1 to the first mapping evaluation is used as an information value deepening adjustment factor for the sample data.

[0027] Furthermore, the original attention score is optimized and adjusted according to the context adaptability adjustment factor and the information value deepening adjustment factor to obtain the final attention score, which specifically includes:

[0028] Obtain the original attention score, contextual adaptability adjustment factor, and information value deepening adjustment factor of the sample data; use the result of multiplying the contextual adaptability adjustment factor and the information value deepening adjustment factor as the optimization weight, and use the result of multiplying the optimization weight and the original attention score as the final attention score.

[0029] In the second aspect, the present application provides a large model training system based on multimodal medical data fusion, including: a processor and a memory, the memory storing computer program instructions, and when the computer program instructions are executed by the processor, a large model training method based on multimodal medical data fusion is implemented.

[0030] Compared with the prior art, the present invention has the following advantages:

[0031] The large-scale model training system and method based on multimodal medical data fusion described in the present invention collects multimodal medical data and extracts and pre-calculates features of the multimodal medical data; obtains the contextual adaptability adjustment factor of the sample data by fusing the information discrimination value evaluation of the sample features with the label association strength; obtains the information value deepening adjustment factor of the sample data by performing modal consistency analysis and modal significance analysis on the multimodal medical data samples; optimizes and adjusts the original attention score by using the contextual adaptability adjustment factor and the information value deepening adjustment factor to obtain the final attention score; performs multimodal data feature fusion and model training based on the final attention score, wherein the contextual adaptability adjustment factor and the information value deepening adjustment factor are used to optimize and adjust the original attention score. The information value deepening adjustment factor optimizes and adjusts the original attention score and normalizes it through the standard Softmax function during the model training process to obtain the final attention weight that can accurately reflect the comprehensive contribution value of each modality. The value vectors corresponding to each modality are weighted and combined according to the final attention weight to generate an optimized multimodal fusion feature representation, so that the optimized multimodal fusion feature representation intelligently enhances the proportion of key non-redundant information in its fusion process, while suppressing the interference of low-value or redundant information. Therefore, compared with the feature representation generated by the existing technology, it can more effectively encapsulate the core information of the sample, thereby improving the efficiency and accuracy of the model learning medical knowledge, and improving the final performance of the model trained by the training system on the target task. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In the attached figure:

[0033] Figure 1This is a flow chart of the method for the large model training system and method based on multimodal medical data fusion described in an embodiment of the present invention. DETAILED DESCRIPTION

[0034] The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0035] See also Figure 1 , is a flow chart of a large model training system and method based on multimodal medical data fusion provided by the first embodiment of the present invention, such as Figure 1 As shown, the large model training system and method based on multimodal medical data fusion may include:

[0036] S1, collects multimodal medical data and performs feature extraction and pre-calculation on the multimodal medical data.

[0037] It should be noted that during the data collection process, relevant privacy protection regulations are strictly observed and all information involving patient identity is completely anonymized or pseudonymized.

[0038] First, medical data samples containing multiple information sources (multiple modalities) are collected from the medical information system. For each sample, ensure that the following core data is obtained:

[0039] Medical imaging data : Such as CT (computed tomography), MRI (magnetic resonance imaging), X-ray, ultrasound and other digital image files.

[0040] Text diagnostic report :With image data The corresponding text-based diagnostic report written by a radiologist or other clinician contains a description of the imaging findings, measurement values, diagnostic opinions, etc.

[0041] Target Label : The final analysis target or ground truth label corresponding to each sample i. This label is crucial for subsequent calculations and is used to define the category C to which the sample belongs ( ).For example It can be the name of the pathologically confirmed disease (such as "lung adenocarcinoma", "benign nodule"), clinical diagnosis stage, and risk level.

[0042] After that, basic feature extraction and data space definition are performed:

[0043] The collected raw data is preprocessed and the basic eigenvectors of each mode are extracted. These eigenvectors form the basis for subsequent calculations. All eigenvectors exist in the eigenspace of their respective modes.

[0044] Image feature extraction: image data Perform standardized preprocessing and then input it into a pre-trained deep convolutional neural network (CNN) model (for example, ResNet, EfficientNet, etc. pre-trained on a large general image dataset or medical image dataset) to extract its deep visual features and obtain a fixed-dimensional image feature vector All image feature vectors It constitutes the feature space of the imaging modality.

[0045] Text feature extraction: text diagnostic report Perform text preprocessing (word segmentation, stop word removal, and conversion to lowercase), then input it into a pre-trained language model to extract its semantic representation to obtain a fixed-dimensional text feature vector All text feature vectors It constitutes the feature space of text modality.

[0046] After that, global statistical information is pre-calculated and stored:

[0047] In order to support the calculation of the adjustment factor in the subsequent steps, it is necessary to (Include The following global statistics are pre-calculated and stored on each modal feature space extracted from the basic feature extraction and data space definition:

[0048] Category-related statistics: For each target category C (consisting of all possible value definition) and each modal ( ), calculate and store the mean vector of the modal features of all samples in the category C ; Calculate and store the modalities of all samples in category C Covariance matrix of features .

[0049] Global modal statistics: For each mode , calculate and store the mode In the entire training data set The average L2 norm of the feature vectors on samples (used for subsequent calculation of relative significance).

[0050] These pre-computed global statistics (category mean vector, category covariance matrix, modal mean norm) will be used as known constants or lookup tables and called in subsequent steps.

[0051] Finally, confirm the data alignment: confirm that for each sample , its data of different modalities ( , ), the corresponding target label And the extracted basic feature vector ( , ) is accurate.

[0052] After completing the above steps, the obtained data set and pre-calculated information can be used for subsequent optimization and fusion processing.

[0053] S2, by evaluating and integrating the distinguishing value of sample feature information of medical data and the correlation strength of medical data labels, the contextual adaptability adjustment factor of sample data is obtained.

[0054] The core of existing attention-based multimodal fusion methods is to determine attention allocation by calculating the numerical similarity between the query vector and the key vectors of each modality. Although this mechanism realizes data-driven weight adjustment, its limitation is that similarity itself is a relative and lacks context-aware measurement. It cannot independently reflect the true information value and contribution potential of a modal feature in the specific analysis context of the current specific sample. Even after the data has been cleaned and obvious errors and noise have been eliminated, this scoring method based on a single vector similarity still faces problems with the ambiguity of feature space expression and the nonlinear relationship between similarity and information contribution.

[0055] Among them, in terms of the ambiguity of feature space expression, the feature vector extracted from complex raw data (high-dimensional images, long reports) is a compressed expression of the original information. In this process, different raw signal patterns may be mapped to similar positions in the feature space, or the importance of the same raw signal in different contexts cannot be fully reflected in a single feature vector, which means that a high similarity score , the corresponding original information pattern behind it and its actual significance in the current sample analysis task are uncertain. For example, in lung CT image analysis, a characteristic pattern representing a small circular shadow is ubiquitous. When the analysis task is to identify early-stage lung cancer, this pattern is a key signal that requires high attention, so its characteristics will have a high similarity with the query vector for searching for suspicious lesions. However, in routine screening or samples with a consistent benign history, the same small circular shadow characteristic pattern is highly likely to represent common benign calcifications or scars, and its diagnostic importance is significantly reduced. Although the visual appearance and characteristic pattern are similar, resulting in a high numerical similarity between the pattern and the query vector, the standard attention mechanism, based on its working principle, cannot and is not designed to be able to determine whether this shadow is a key suspected signal or common benign background information in the current specific situation based on this similarity alone.

[0056] On the other hand, the nonlinear relationship between similarity and information contribution is not linear in medical diagnostic analysis. A feature that is highly similar to the query but represents commonly known or known information, such as a description of a normal anatomical structure, will have a low marginal contribution to the final decision. Conversely, a feature that is not necessarily highly similar to the query but represents some discrepant information, such as an imaging sign or a suggestive differential diagnosis in a report, will have a higher marginal contribution. Standard attention scores, calculated as proportional to similarity, fail to capture this complex, nonlinear, or even inversely correlated relationship between contribution and similarity. They tend to amplify highly similar information regardless of its true contribution value.

[0057] Therefore, relying solely on vector similarity to allocate attention will inevitably lead to deviations in the true value assessment of each modal information in the current specific sample analysis context. To address the problem of true value assessment deviation, the present invention performs intra-class difference analysis on the eigenvectors of the sample data to obtain a first typicality assessment of the sample data; performs heterogeneous center difference analysis on the eigenvectors of the sample data to obtain a first heterogeneous separation assessment of the sample data; performs a discrimination value analysis on the first typicality assessment and the first heterogeneous separation assessment of the sample data to obtain a first discrimination degree of the sample data; performs target label association analysis on the eigenvectors of the sample data to obtain the label association strength of the sample data; and performs a fusion analysis on the first discrimination degree of the sample data and the label association strength to obtain a contextual adaptability adjustment factor of the sample data.

[0058] The present invention constructs context-adaptive adjustment factors for sample data, which no longer rely on judging absolute “signal quality” or whether they conform to “universal priors” (to avoid accidentally harming rare cases), but instead focuses on evaluating the relative value of feature information in the current context:

[0059] Assessing the discrimination and specificity of information: Valuable information should help distinguish the current sample from other possible disease states. This requires assessing the extent to which a modality feature is a unique representative of its category (rather than a universally shared feature) and the degree to which it differs from other categories. This invention achieves this assessment by calculating specific statistical indicators, aiming to identify feature sources that are more likely to point to key diagnostic clues or rare manifestations.

[0060] The direct relevance of the evaluation information to the sample analysis goal: The ultimate value of a feature is reflected in its contribution to the achievement of the analysis goal. This invention quantifies the feature by directly comparing the geometric relationship between the feature and its category and the closest different category. and sample target labels The distance information in the feature space is used to determine the extent to which a feature clearly points to its true category, thereby evaluating its actual utility for the current task goal.

[0061] The first typicality evaluation of the sample data is obtained by performing intra-class difference analysis on the characteristic vectors of the sample data; the first heterogeneous separation evaluation of the sample data is obtained by performing heterogeneous center difference analysis on the characteristic vectors of the sample data; and the first discrimination degree of the sample data is obtained by performing discrimination value analysis on the first typicality evaluation and the first heterogeneous separation evaluation of the sample data. Specifically, the method includes:

[0062] Obtaining a eigenvector of the sample data and a mean vector and covariance matrix of a target category corresponding to the eigenvector; performing a Mahalanobis distance evaluation between the eigenvector and the category center using the eigenvector of the sample data, the mean vector of the target category corresponding to the eigenvector of the sample data, and the covariance matrix of the target category corresponding to the eigenvector of the sample data to obtain a first Mahalanobis distance of the eigenvector of the sample data; dividing the square of the first Mahalanobis distance by the dimension of the eigenvector of the sample data by a calculation result as a first typicality evaluation of the sample data;

[0063] In one embodiment, assuming that In the sample The eigenvector of a mode is , based on the categories pre-calculated from the training dataset Medium mode The covariance matrix of , based on the categories pre-calculated from the training dataset Medium mode The mean vector of , then In the sample The calculation formula for the first typicality evaluation of a mode is:

[0064]

[0065] in, Indicates the In the sample First typicality assessment of each modality; Indicates the In the sample The eigenvector of each mode; Represents the categories pre-calculated based on the training dataset Medium mode The mean vector of ; Represents the categories pre-calculated based on the training dataset Medium mode The covariance matrix of Represents the feature vector Dimensions, Represents the transpose of a vector.

[0066] Afterwards, the closest heterogeneous mean vector of the sample data is obtained by evaluating the distance between the feature vector of the sample data and the mean vectors corresponding to all other target categories; the square of the Euclidean distance between the feature vector of the sample data and the closest heterogeneous mean vector is used as the first heterogeneous separation evaluation of the sample data;

[0067] Obtain an atypicality threshold within a target category of sample data; use the first heterogeneous separation evaluation of the sample data as a numerator and the result of adding a constant 1 to the first typicality evaluation of the sample data as a denominator to form a fraction as a first discrimination evaluation factor; perform Sigmoid function mapping on the result of subtracting the first typicality evaluation of the sample data from the atypicality threshold within the target category of the sample data as a second discrimination evaluation factor; and use the result of multiplying the first discrimination evaluation factor by the second discrimination evaluation factor as the first discrimination degree of the sample data.

[0068] In one embodiment, assuming that In the sample The first heterogeneous separation evaluation of the modalities is ; The atypical threshold within the target category of the sample data is , then In the sample The calculation formula for the first degree of discrimination of a mode is:

[0069]

[0070] in, Indicates the In the sample the first degree of differentiation of the modalities; Indicates the In the sample First heterogeneous separation evaluation of the modalities; Indicates the In the sample First typicality assessment of each modality; Indicates the atypical threshold within the target category of the sample data; express Activation function.

[0071] It should be noted that, considering that in medical applications, only those features that can effectively stay away from outliers and show a certain uniqueness (not completely typical) in the same category have the greatest distinguishing value, the above formula ensures that only when the feature satisfies both the requirements of staying away from outliers and being sufficiently unique in the same category, its degree of distinction will be significantly improved when it exceeds the threshold. In this embodiment, the initial value of the atypicality threshold within the target category of the sample data is set to 1. In actual training, the 75th percentile of the squared Mahalanobis distance distribution of the modal feature of the corresponding category can be used for setting.

[0072] After evaluating the discriminative value of a feature, it is also necessary to measure its direct correlation strength with the current sample analysis task goal. The value of medical information is ultimately reflected in its contribution to achieving a specific clinical goal (such as accurate diagnosis). To this end, the present invention calculates the correlation strength index between the feature and the target label. This index is intended to quantify the feature. To what extent is it clearly pointed to its true category This is achieved by comparing the distance of the feature to the nearest alien center with the distance to the center of its own category. The larger the distance ratio, the more clearly it belongs to its own category and the stronger the association with the target label.

[0073] Specifically, after obtaining the first degree of discrimination of the sample data, target label association analysis can be performed on the feature vector of the sample data to obtain the label association strength of the sample data; and the context adaptability adjustment factor of the sample data can be obtained by fusion analysis of the first degree of discrimination of the sample data and the label association strength, including:

[0074] Obtain the mean vector of the nearest other target categories corresponding to the feature vector of the sample data and the mean vector of the target category corresponding to the feature vector of the sample data; use the mean vector of the nearest other target categories corresponding to the feature vector of the sample data as the nearest heterogeneous cluster mean vector of the sample data; use the mean vector of the target category corresponding to the feature vector of the sample data as the category mean vector of the sample data;

[0075] The label association strength of the sample data is calculated by taking the Euclidean distance between the feature vector of the sample data and the nearest heterogeneous cluster mean vector of the sample data as the numerator and the Euclidean distance between the feature vector of the sample data and the class mean vector of the sample data plus a very small positive number as the denominator.

[0076] In one embodiment, assuming that In the sample data The mean vector of the nearest other target category corresponding to the feature vector of each modality is Rule No. In the sample data The calculation formula for the label association strength of a modality is:

[0077]

[0078] in, Indicates the In the sample data The strength of the label association of each modality; Indicates the In the sample The eigenvector of each mode; Indicates the In the sample data The mean vector of the nearest other target category corresponding to the feature vector of each modality; Represents the categories pre-calculated based on the training dataset Medium mode The mean vector of ; represents the Euclidean distance; Represents a very small positive number to prevent the denominator from being 0.

[0079] It should be noted that this formula directly calculates the ratio of the minimum distance between classes to the distance within a class. Higher values ​​indicate modality Features The closer the correlation is with the goal of completing the current sample analysis task.

[0080] Obtain the set first discrimination degree non-negative fusion weight coefficient and the set label association strength non-negative fusion weight coefficient; perform logarithmic mapping on the result of adding the constant 1 to the first discrimination degree of the sample data as the first logarithmic mapping result; perform logarithmic mapping on the result of adding the constant 1 to the label association strength of the sample data as the second logarithmic mapping result; perform weighted summation calculation on the first logarithmic mapping result and the second logarithmic mapping result respectively using the first discrimination degree non-negative fusion weight coefficient and the label association strength non-negative fusion weight coefficient, and use the weighted summation calculation result as the first fusion evaluation; perform hyperbolic tangent function mapping on the first fusion evaluation as the contextual adaptability adjustment factor of the sample data.

[0081] In one embodiment, it is assumed that the non-negative fusion weight coefficient of the first differentiation level is ; The non-negative fusion weight coefficient of label association strength is Rule No. In the sample data The calculation formula of the contextual adaptability adjustment factor of each modality is:

[0082]

[0083] in, Indicates the In the sample data Contextual adaptation moderators for each modality; Indicates the non-negative fusion weight coefficient of the first distinction level. In the embodiment of the present invention, the initial value of the weight coefficient is set to ; Indicates the In the sample the first degree of differentiation of the modalities; Indicates the non-negative fusion weight coefficient of the label association strength. In the embodiment of the present invention, the initial value of the weight coefficient is set to ; Indicates the In the sample data The strength of the label association of each modality; represents the hyperbolic tangent function; Indicates The logarithmic function of base .

[0084] It should be noted that the contextual adaptation adjustment factor of the sample data designed by the present invention is Its core goal is to make up for the defect that the existing attention mechanism only relies on vector similarity to calculate the attention score, but cannot accurately measure the true contribution potential of information in a specific sample context. To achieve this goal, The construction process closely combines the characteristics of medical data and the requirements of analysis tasks, reflecting clear causal logic and problem-solving ideas.

[0085] In order to solve the problem that the existing attention mechanism is difficult to distinguish the value of information behind high similarity, the present invention first calculates the relative density within the class and the inter-class separation index of the feature. To assess the discriminative value of information. Using the Mahalanobis distance to calculate intra-class distances can account for the complex inter-dimensional correlations in medical feature data, resulting in a more accurate measure of feature typicality. Combining the Euclidean distance to the nearest outlier center directly correlates features with their ability to distinguish between different class states.

[0086] The first degree of distinction of sample data is introduced based on Sigmoid function and threshold This design simulates the emphasis on information specificity in clinical judgment: only information that effectively distinguishes from other categories and exhibits a certain degree of uniqueness (not completely typical) within its category is considered to have high discriminatory value. This refined quantification of discriminatory value effectively overcomes the limitation of similarity alone, which cannot determine whether information is discriminatory.

[0087] In order to solve the problem that the existing attention mechanism lacks task-oriented considerations, this paper calculates the correlation strength index between features and target labels. . Using geometric relationships in feature space to evaluate features and its true category label By directly calculating the ratio of the Euclidean distance from the feature to the nearest heterogeneous center to the Euclidean distance to its own category center, It intuitively quantifies the extent to which the feature clearly points to its true category rather than other categories. A feature that is close to the center of its own category and far from the center of the other category has a The value is naturally high, indicating that it is strongly associated with the target label.

[0088] By using logarithmic transformation smoothing and combining weighted summation with activation function Way to get , ensuring that the adjustment factor can comprehensively and balancedly reflect the distinguishing value and task contribution of modal features in the current context.

[0089] At this point, by performing modal consistency analysis and modal significance analysis on the samples, the information value deepening adjustment factor of the sample modality is obtained.

[0090] S3, by performing modal consistency analysis and modal significance analysis on multimodal medical data samples, the information value deepening adjustment factor of the sample data is obtained.

[0091] The sample data context adaptation adjustment factor is calculated in step S2 Finally, a preliminary assessment of the expected contribution of sample data in a specific context (based on its discriminative power and task relevance) is made. This step improves our understanding of the value of attention allocation to individual modalities, but the fundamental flaw of existing attention mechanisms remains unresolved: they fail to fully consider the interrelationships between different information sources (modalities), particularly the degree of consistency in their content and the strength of the signals themselves, which is crucial for ultimately determining the optimal attention weights.

[0092] Specifically, even if a modal The sample data feature vector Assessed as having a high expected contribution value (i.e. The information content conveyed by a modality with a high value may be highly similar to that of other modalities with similarly high value. In medical practice, images, reports, and medical records often describe the same condition or physiological state from different perspectives. The information between them is mutually reinforcing and there is also duplication. To improve its attention score, it is easy to give too much cumulative attention to these repeated information, but dilute the attention allocated to other modalities that may contain unique perspectives or key supplementary information. On the other hand, although the contribution value of a modality is initially evaluated, It is not the highest, but its information content is significantly different from other modalities, or although the content is similar, its signal strength is much higher than other modalities. These situations indicate that this modality may play a particularly important role in the current sample.

[0093] Therefore, in order to compensate for After the adjustment, the problem of inaccurate attention allocation due to failure to fully consider the consistency and relative strength between modalities still exists. The present invention introduces an information value deepening adjustment factor. The evaluation is further based on the modality The attention contribution of the modality can be adjusted more finely based on the consistency of the information content of the modality with other modalities and the relative strength of its own signal.

[0094] The idea behind constructing the information value deepening moderating factor is to measure these two key interactive characteristics:

[0095] One is to evaluate the consistency of modal content: by calculating the modal The average similarity between the features of the modality and the features of all other modalities directly understands the extent to which the modality information overlaps or coordinates with other source information. High consistency means that information is repeated; low consistency implies the existence of unique differences.

[0096] The second is to evaluate the relative strength of the sample data: by comparing the modes The signal strength of a feature and its average strength across the entire dataset determine whether the modality is particularly prominent or unusually weak in the current sample. Modalities with high signal strength tend to convey clearer information, and their strength alone provides additional clues for judgment, regardless of whether their content is consistent with other modalities.

[0097] Although the sample data context adaptation adjustment factor The problem that the existing attention mechanism cannot evaluate the value of information distinction and the strength of task association has been initially solved. Correcting the raw attention score The fundamental limitation of calculating scores based on similarity alone has not been completely overcome. After the attention score is improved, the corresponding modality information may be more likely to be compared with other modalities that also receive high attention scores. The rating modality is consistent, resulting in information redundancy; or a Modes with moderate values ​​provide key complementary or conflicting information with other modes. To solve the suboptimal attention allocation problem caused by ignoring the interaction between modalities after adjustment, the present invention further designs an information value deepening adjustment factor, which is designed to be based on the modality The consistency level with other modal information and its own relative significance have a great impact on the After the initial adjustment, the attention contribution potential is recalibrated to ultimately achieve the goal of prioritizing key non-redundant information.

[0098] The specific steps for obtaining the information value deepening adjustment factor of sample data include obtaining the characteristic vector of the sample data in the multimodal medical data set, performing inter-modal consistency analysis on the characteristic vector of the sample data to obtain the inter-modal similarity of the sample data; performing significance evaluation on the characteristic vector of the sample data to obtain the relative significance of the sample data; and performing a fusion evaluation of the inter-modal similarity and relative significance of the sample data to obtain the information value deepening adjustment factor of the sample data.

[0099] First, the inter-modal similarity of the sample data is obtained by performing inter-modal consistency analysis on the eigenvectors of the sample data. Specifically, the eigenvectors of the sample data are obtained, and the average cosine similarity between the eigenvector of any dimension of the sample data and the eigenvectors of all other dimensions of the sample data is used as the inter-modal similarity of the sample data.

[0100] Afterwards, the relative significance of the sample data is obtained by performing a significance evaluation on the eigenvector of the sample data. Specifically, the eigenvector of the sample data and the average L2 norm of the eigenvectors of any modality of the sample data on all samples of the multimodal medical dataset are obtained; the L2 norm of the eigenvector of any modality of the sample data is divided by the average L2 norm of the eigenvector of the modality on all samples of the multimodal medical dataset, and the result of calculation is used as the relative significance of the sample data.

[0101] Finally, by fusing and evaluating the inter-modal similarity and relative significance of the sample data, the information value deepening adjustment factor of the sample data is obtained. Specifically, the adjustment weight of the set relative significance influence is obtained; the calculation result of adding the constant 1 and the inter-modal similarity of the sample data is used as the first adjustment factor; the calculation result of multiplying the adjustment weight of the relative significance influence, the relative significance of the sample data and the inter-modal similarity of the sample data is used as the second adjustment factor; the mapping result of the calculation result of the addition of the first adjustment factor and the second adjustment factor by the hyperbolic tangent function is used as the first mapping evaluation; and the calculation result of adding the constant 1 and the first mapping evaluation is used as the information value deepening adjustment factor of the sample data.

[0102] In one embodiment, assuming that In the sample data The inter-modal similarity of the modes is ;No. In the sample data The relative significance of each mode is ; The adjustment weight of the relative significant effect is , then In the sample data The calculation expression of the information value deepening adjustment factor of each modality is:

[0103]

[0104] in, Indicates the In the sample data The information value deepening modifier of each modality; Indicates the In the sample data The degree of inter-modal similarity of the modalities; Indicates the In the sample data The relative significance of each mode; Moderation weights indicating relative significance of effects; represents the hyperbolic tangent function; Represents the constant 1.

[0105] It should be noted that, in order to solve the information redundancy problem that may still exist after the initial adjustment, the present invention calculates the average cosine similarity To quantify the consistency between modalities. This metric provides a direct clue for identifying modalities that highly overlap with other sources of information. In order to distinguish the signal strength, a relative significance metric is introduced. ,The use of the normalized L2 norm can effectively capture the prominence of the current modal signal strength relative to the average level. pass( ) item explicitly improves those that are inconsistent with other modalities ( low) information weight, solving the Issues where key supplementary or conflicting information is not adequately highlighted after adjustment. Items, retaining and enhancing those that are both consistent ( higher values) and significantly ( This avoids excessive suppression of all consistent information and solves the bias caused by only considering inconsistency.

[0106] At this point, by performing modal consistency analysis and modal significance analysis on multimodal medical data samples, the information value deepening adjustment factor of the sample data is obtained.

[0107] S4, optimizes and adjusts the original attention score through the context adaptability adjustment factor and the information value deepening adjustment factor to obtain the final attention score.

[0108] In obtaining the adjustment factor that reflects the contextual adaptability of the sample data and regulatory factors reflecting the deepening of information value These two factors are then applied to existing attention-based feature fusion methods. Specifically, they modify the calculation of the original attention score. This modification aims to integrate the present invention's assessment of modal contribution potential into the final determination of attention weights, overcoming the limitation of the original mechanism's reliance solely on vector similarity.

[0109] This step is performed within the standard attention fusion framework. First, follow the standard process of the existing attention mechanism for the current sample. and query vector , calculate each mode Basic characteristics of The corresponding key vector Sum value vector , and calculate the raw attention score between the query vector and the key vector (Through dot product similarity ). Subsequently, the present invention uses the original attention score To make corrections, use a two-step product adjustment method to adjust the adjustment factor calculated previously and Act on the original scores in turn to get the final adjusted attention score , the specific calculation is shown in the following formula:

[0110]

[0111] in, Indicates the In the sample The final attention score of each modality; Indicates the In the sample Raw attention scores for each modality; Indicates the In the sample data Contextual adaptation moderators for each modality; Indicates the In the sample data The information value deepening regulatory factor of each modality.

[0112] At this point, the original attention score is optimized and adjusted through the context adaptability adjustment factor and the information value deepening adjustment factor to obtain the final attention score.

[0113] S5, multimodal data feature fusion and model training are performed through the final attention score.

[0114] After calculating the adjusted attention scores of each modality, this step completes the final feature fusion and applies the results to the training of the machine learning model. The adjusted attention scores are normalized by the standard Softmax function to obtain the final attention weights that can accurately reflect the comprehensive contribution value of each modality. Based on these optimized weights, the value vectors corresponding to each modality are weighted and combined to generate an optimized multimodal fusion feature representation. This optimized representation has intelligently enhanced the proportion of key, non-redundant information during the fusion process, while suppressing the interference of low-value or redundant information. Therefore, compared with the representation generated by the prior art, it can more effectively encapsulate the core information of the sample. Finally, in the training system of the present invention, the optimized fusion feature representation generated in this way is used as input data for training machine learning models that need to process multimodal medical data. The optimized feature representation obtained by the method of the present invention is used for training, aiming to improve the efficiency and accuracy of the model in learning medical knowledge, thereby improving the final performance of the model trained by the training system on the target task.

[0115] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A large model training method based on multimodal medical data fusion, characterized by: The method comprises the following steps: Step S1: Collect multimodal medical data, and perform feature extraction and pre-calculation on the multimodal medical data; Step S2: Obtaining the contextual adaptability adjustment factor of the sample data by evaluating and integrating the distinguishing value of the sample feature information of the medical data and the correlation strength of the medical data labels; Step S3: Performing modal consistency analysis and modal significance analysis on the multimodal medical data samples to obtain the information value deepening adjustment factor of the sample data; Step S4: Optimize and adjust the original attention score through the context adaptability adjustment factor and the information value deepening adjustment factor to obtain the final attention score; Step S5: Multimodal data feature fusion and model training are performed using the final attention score; The method of obtaining a contextual adaptability adjustment factor of sample data by evaluating and fusing the distinguishing value of sample feature information of medical data and the label association strength of medical data includes: obtaining the feature vector of sample data in a multimodal medical data set and the category to which the feature vector of the sample data belongs; obtaining a first typicality evaluation of the sample data by performing an intra-class difference analysis on the feature vector of the sample data; obtaining a first heterogeneous separation evaluation of the sample data by performing a heterogeneous center difference analysis on the feature vector of the sample data; obtaining a first degree of discrimination of the sample data by performing a distinguishing value analysis on the first typicality evaluation of the sample data and the first heterogeneous separation evaluation; obtaining the label association strength of the sample data by performing a target label association analysis on the feature vector of the sample data; and obtaining the contextual adaptability adjustment factor of the sample data by performing a fusing analysis on the first degree of discrimination of the sample data and the label association strength.

2. The large model training method based on multimodal medical data fusion according to claim 1 is characterized in that: Collecting multimodal medical data and performing feature extraction and pre-calculation on the multimodal medical data specifically includes: Multimodal medical data samples in a medical information system are collected and desensitized to construct a multimodal medical data set; for any sample in the multimodal medical data set, its medical imaging data, text diagnosis report data and target label data are obtained; standardization preprocessing is performed on the medical imaging data and input into a pre-trained deep convolutional neural network to extract image feature vectors; text preprocessing is performed on the text diagnosis report data and input into a pre-trained language model to extract text feature vectors; all samples are classified according to the target label, and for any target category and any modality, the modal feature mean vector and covariance matrix of the samples in the category are calculated and stored; for any modality, the average L2 norm of the feature vector of the modality on all samples in the data set is calculated and stored.

3. The large model training method based on multimodal medical data fusion according to claim 1 is characterized in that: According to the method, by performing intra-class difference analysis on the characteristic vectors of the sample data, a first typicality evaluation of the sample data is obtained; by performing heterogeneous center difference analysis on the characteristic vectors of the sample data, a first heterogeneous separation evaluation of the sample data is obtained; By performing a differentiation value analysis on the first typicality evaluation and the first heterogeneous separation evaluation of the sample data, the first differentiation degree of the sample data is obtained, specifically including: Obtain the characteristic vector of the target sample, call the mean vector and covariance matrix of its target category, evaluate the multidimensional distance between the three, standardize the evaluation results according to the characteristic dimension, and obtain a first typicality evaluation; compare the characteristic vector of the target sample with the mean vectors of all other target categories one by one, select the minimum distance and take its square value to obtain a first heterogeneous separation evaluation; obtain the atypical threshold of the target category, use the first heterogeneous separation evaluation as the numerator, the constant one and the sum of the first typicality evaluation as the denominator to form a fraction, as the first discrimination degree evaluation factor; subtract the difference between the first typicality evaluation and the threshold through Sigmoid function mapping to obtain a second discrimination degree evaluation factor; the product of the first discrimination degree evaluation factor and the second discrimination degree evaluation factor is used as the first discrimination degree of the sample data.

4. The large model training method based on multimodal medical data fusion according to claim 1 is characterized in that: The target label association analysis is performed on the feature vector of the sample data to obtain the label association strength of the sample data, and the context adaptability adjustment factor of the sample data is obtained by performing a fusion analysis on the first discrimination degree of the sample data and the label association strength, specifically including: The distance between the feature vector of the sample data and the nearest mean vector of the heterogeneous category and the distance to the mean vector of the current category are obtained respectively, and the label association strength of the sample data is determined by the ratio of the two; a non-negative fusion weight coefficient of the first discrimination degree and a non-negative fusion weight coefficient of the label association strength are set, the first discrimination degree and the label association strength are logarithmically mapped respectively, and weighted summed according to the weight coefficients to obtain a first fusion evaluation; the result of the hyperbolic tangent mapping of the first fusion evaluation is used as the context adaptability adjustment factor of the sample data.

5. The large model training method based on multimodal medical data fusion according to claim 1 is characterized in that: The information value deepening adjustment factor of the sample data is obtained by performing modal consistency analysis and modal significance analysis on the multimodal medical data samples, specifically including: Acquire the eigenvectors of sample data in a multimodal medical dataset, and use the average cosine similarity between the eigenvector of any dimension of the sample data and the eigenvectors of all other dimensions of the sample data as the inter-modal similarity of the sample data; perform a significance assessment on the eigenvectors of the sample data to obtain the relative significance of the sample data; perform a fusion assessment of the inter-modal similarity and the relative significance of the sample data to obtain the information value deepening adjustment factor of the sample data.

6. The large model training method based on multimodal medical data fusion according to claim 5 is characterized in that: The relative significance of the sample data is obtained by performing significance evaluation on the characteristic vector of the sample data, including: Obtain the eigenvector of the sample data and the average L2 norm of the eigenvectors of any modality of the sample data on all samples of the multimodal medical dataset; and use the ratio of the L2 norm of the eigenvector of any modality of the sample data to the average L2 norm of the eigenvector of the modality on all samples of the multimodal medical dataset as the relative significance of the sample data.

7. The large model training method based on multimodal medical data fusion according to claim 5 is characterized in that: According to the above, by fusion evaluation of the similarity and relative significance between the modalities of the sample data, the information value deepening adjustment factor of the sample data is obtained, which specifically includes: The adjustment weight of the relative significance influence is set; the result of adding the constant 1 and the degree of similarity between the modalities of the sample data is used as the first adjustment factor; the result of multiplying the adjustment weight of the relative significance influence, the relative significance of the sample data and the degree of similarity between the modalities of the present data is used as the second adjustment factor; the result of mapping the result of adding the first adjustment factor and the second adjustment factor by the hyperbolic tangent function is used as the first mapping evaluation; the result of adding the constant 1 and the first mapping evaluation is used as the information value deepening adjustment factor of the sample data.

8. The large model training method based on multimodal medical data fusion according to claim 1 is characterized in that: The original attention score is optimized and adjusted by the context adaptability adjustment factor and the information value deepening adjustment factor to obtain the final attention score, specifically including: Obtain the original attention score, contextual adaptability adjustment factor and information value deepening adjustment factor of the sample data; use the product of the contextual adaptability adjustment factor and the information value deepening adjustment factor as the optimization weight, and use the calculation result of multiplying the optimization weight and the original attention score as the final attention score.

9. A large model training system based on multimodal medical data fusion, characterized by: include: A processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, a large model training method based on multimodal medical data fusion according to any one of claims 1 to 8 is implemented.

Images