Medical data processing method, system, computer device and readable storage medium

Abstract

The application discloses a medical data processing method, comprising: through a first encoder, a second encoder and a third encoder, respectively corresponding to mapping medical images, clinical texts and structured test indexes to a unified feature dimension, outputting an image feature sequence, a text feature sequence and an index feature vector; through a bidirectional cross-modal cross-attention fusion module, performing two-way cross-attention calculation on the image feature sequence, the text feature sequence and the index feature vector, obtaining three groups of bidirectional fusion representations; through a gated adaptive weighting network, dynamically adjusting the contribution weights of the three groups of bidirectional fusion representations, to output a fusion representation; inputting the fusion representation into a strategy network, outputting a probability distribution of a medical data classification action by the strategy network, and determining a multi-modal fusion classification result based on the probability distribution. The application also provides a medical data processing system, a computer device and a readable storage medium. The application can improve the classification accuracy and robustness.

Description

Technical Field

[0001] This invention relates to the field of medical artificial intelligence technology, specifically to a medical data processing method, system, computer device, and readable storage medium. Background Technology

[0002] The healthcare system continuously accumulates a vast amount of heterogeneous, multimodal data. Medical images, electronic medical records, and structured laboratory indicators describe a patient's health status from different dimensions, containing rich and complementary semantic information. Single-modal data often fails to fully depict complex clinical situations, requiring clinicians to simultaneously refer to multiple sources of information, including imaging findings, medical history, and laboratory indicators, to make accurate diagnoses.

[0003] In existing technologies, multimodal medical data fusion methods are mainly divided into three paradigms: early fusion, late fusion, and intermediate fusion. Early fusion achieves information aggregation by splicing original features, which easily leads to high-dimensional sparsity problems; late fusion only integrates at the decision level and cannot fully capture the interaction information between modalities. Intermediate layer fusion strategies based on deep learning are gradually becoming the mainstream research direction, but most existing attention fusion methods follow the standard architecture of general vision or natural language processing fields, lacking customized designs for the heterogeneity of medical data, making it difficult to effectively handle the fine-grained alignment problem between local lesions in images and text descriptions, and their accuracy and robustness are not high.

[0004] Therefore, there is an urgent need for a cross-modal processing method for heterogeneous medical data that can achieve tight structural coupling and dynamic adaptive fusion, thereby improving classification accuracy and robustness. Summary of the Invention

[0005] Based on this, it is necessary to propose a medical data processing method, system, computer equipment, and readable storage medium to address the above problems, which can achieve tight structural coupling and dynamic adaptive fusion, thereby improving classification accuracy and robustness.

[0006] The technical problem solved by this application is achieved by the following technical solution: This application provides a medical data processing method, which includes the following steps: mapping medical images, clinical text, and structured test indicators in multimodal medical data to a unified feature dimension using a first encoder, a second encoder, and a third encoder, respectively, and outputting image feature sequences, text feature sequences, and indicator feature vectors; performing pairwise bidirectional cross-attention calculations on the image feature sequences, text feature sequences, and indicator feature vectors using a bidirectional cross-modal attention fusion module to obtain three sets of bidirectional fusion representations; dynamically adjusting the contribution weights of the three sets of bidirectional fusion representations using a gated adaptive weighted network to output fusion representations; and outputting a probability distribution of medical data classification actions based on the fusion representations and a policy network, and determining the multimodal fusion classification result based on the probability distribution.

[0007] In one embodiment, the step of mapping the medical image to the unified feature dimension using the first encoder and then outputting the image feature sequence includes: dividing the medical image into multiple non-overlapping image blocks; flattening the image blocks and mapping them to the unified feature dimension using a first learnable linear transformation matrix, and attaching a first learnable position encoding vector to each image block; and performing deep feature extraction on the image block sequence processed by the first learnable linear transformation matrix and the first learnable position encoding vector using the first encoder to output the image feature sequence.

[0008] In one embodiment, the step of mapping the clinical text to the unified feature dimension through the second encoder and then outputting the text feature sequence includes: mapping each word in the clinical text to a corresponding word embedding vector and superimposing a second learnable positional encoding vector to obtain a word embedding sequence; and performing contextual semantic encoding on the word embedding sequence based on the second encoder to output the text feature sequence.

[0009] In one embodiment, the step of mapping the structured test index to the unified feature dimension through the third encoder and then outputting the index feature vector includes: mapping the structured test index to the hidden layer dimension through the third encoder and performing a nonlinear transformation through a first activation function to obtain hidden layer features; mapping the hidden layer features to the unified feature dimension and performing feature distribution stabilization processing through a second activation function and a layer normalization unit to output the index feature vector.

[0010] In one embodiment, the step of performing pairwise bidirectional cross-attention calculations on the image feature sequence, the text feature sequence, and the indicator feature vector through the bidirectional cross-modal cross-attention fusion module to obtain three sets of bidirectional fusion representations includes: taking any two of the image feature sequence, the text feature sequence, and the indicator feature vector as the first modality feature sequence and the second modality feature sequence; performing a linear transformation on the first modality feature sequence through a query projection matrix to generate a query matrix; performing a linear transformation on the second modality feature sequence through a key projection matrix and a value projection matrix respectively to generate a key matrix and a value matrix; calculating the cross-attention output from the first modality to the second modality direction based on the query matrix, the key matrix, and the value matrix, and calculating the reverse cross-attention output from the second modality to the first modality direction to obtain a bidirectional cross-attention output set; concatenating the outputs of each attention head in the bidirectional cross-attention output set and performing a linear projection through an output projection matrix to obtain the bidirectional fusion representation.

[0011] In one embodiment, the step of concatenating the outputs of each attention head in the bidirectional cross-attention output set and performing linear projection through the output projection matrix to obtain the bidirectional fusion representation includes: obtaining the three sets of bidirectional fusion representations based on the following formula: =MultiHead( Zm , Zn =Concat( h 1,…, hH ) ; in, The first modal feature sequence Zm and the second modal feature sequence Zn Bidirectional fusion representation, MultiHead( ) is the multi-head cross-attention function, hi = CrossAttni( Zm , Zn ) represents the output of the i-th attention head, and H represents the total number of attention heads. Let R represent the output projection matrix, and let R represent the set of real numbers. d k This represents the dimension of each attention head. d Represents the unified feature dimension, Concat( () indicates a splicing operation.

[0012] In one embodiment, the step of dynamically adjusting the contribution weights of the three sets of bidirectional fusion representations through the gated adaptive weighted network to output the fusion representation includes: concatenating each of the bidirectional fusion representations and its corresponding global classification label; calculating a gate value using a gate weight matrix and a gate bias term; normalizing the gate value to obtain the contribution weight of each modal branch; and performing a weighted summation of the three sets of bidirectional fusion representations based on the contribution weights to output the fusion representation.

[0013] In one embodiment, the step of outputting the probability distribution of the medical data classification action based on the policy network according to the fusion representation, and determining the multimodal fusion classification result based on the probability distribution, includes: taking the fusion representation as a state input; the policy network linearly projecting the state input through a first policy fully connected layer weight matrix and a first policy bias term, and performing a nonlinear transformation through a first policy activation function to obtain a first policy hidden layer feature; inputting the first policy hidden layer feature into a second policy fully connected layer weight matrix and a second policy bias term for linear mapping, and generating the probability distribution of the medical data classification action through a normalization function; and selecting the medical data classification action with the highest probability based on the probability distribution of the medical data classification action to generate and output the multimodal fusion classification result corresponding to the multimodal medical data.

[0014] In one embodiment, the method employs a two-stage joint training strategy: First stage: Sample multimodal medical data is input into the multimodal encoding module, and after feature fusion through the bidirectional cross-modal attention fusion module and the gated adaptive weighted network to obtain sample fusion representations, these representations are input into the classification head. Using classification loss as the optimization objective, a first loss weight, a second loss weight, and a third loss weight are set to optimize the classification loss, thereby pre-training the multimodal encoding module to obtain pre-trained encoding parameters. Second stage: The shallow network parameters of the multimodal encoding module are frozen, while the deep network parameters of the multimodal encoding module remain in a trainable state. The pre-trained encoding parameters are used as the initial parameters of the multimodal encoding module. The sample fusion representations corresponding to the sample multimodal medical data are connected to the policy network and the value network. The first loss weight, the second loss weight, and the third loss weight are adjusted, using the total loss function as the optimization objective, to drive end-to-end collaborative adjustment of the policy network, the value network, the bidirectional cross-modal attention fusion module, the gated adaptive weighted network, and the deep network parameters of the multimodal encoding module.

[0015] This application also provides a medical data processing system, including: a multimodal encoding module (also known as a multimodal encoder), a bidirectional cross-modal cross-attention fusion module (also known as a BiCMA module), a gated adaptive weighting module (also known as a GAWN module), and an agent reasoning module. The multimodal encoding module includes a first encoder, a second encoder, and a third encoder in parallel, configured to map medical images, clinical text, and structured test indicators from multimodal medical data to a unified feature dimension, and output image feature sequences, text feature sequences, and indicator feature vectors. The bidirectional cross-modal cross-attention fusion module is configured to perform pairwise bidirectional cross-attention calculations on the image feature sequences, text feature sequences, and indicator feature vectors to obtain three sets of bidirectional fused representations. The gated adaptive weighting module includes a gated adaptive weighting network configured to dynamically adjust the contribution weights of the three sets of bidirectional fused representations and output fused representations. The agent reasoning module includes a policy network configured to output a probability distribution of medical data classification actions based on the fused representations, and determine the multimodal fusion classification result based on the probability distribution.

[0016] This application also provides a computer device including a processor and a memory: the processor is used to execute a computer program stored in the memory to implement the method as described above.

[0017] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method as described above.

[0018] The embodiments of this application have the following beneficial effects: This application uses a parallel first encoder, a second encoder, and a third encoder, configured to map medical images, clinical text, and structured test indicators from multimodal medical data to a unified feature dimension, and output image feature sequences, text feature sequences, and indicator feature vectors. It also achieves fine-grained semantic alignment between heterogeneous modalities through a bidirectional cross-modal cross-attention mechanism and dynamically adjusts modal contributions through a gated adaptive weighted network. Therefore, it can achieve tightly coupled structure and dynamic adaptive fusion, thereby improving the accuracy and robustness of multimodal fusion classification.

[0019] The above description is merely an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it according to the contents of the specification, and to make the above and other objects, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and do not limit this application. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating a medical data processing method provided in one embodiment.

[0022] Figure 2 This is a schematic diagram of the architecture of a medical data processing system provided in one embodiment.

[0023] Figure 3 This is a schematic diagram of the structure of a multimodal feature extraction module provided in one embodiment.

[0024] Figure 4 This is a schematic diagram of a bidirectional cross-modal cross-attention fusion mechanism provided in one embodiment.

[0025] Figure 5 This is a schematic diagram of the structure of a gated adaptive weighted network provided in one embodiment.

[0026] Figure 6 A schematic diagram of the overall architecture of a medical data processing system provided for another embodiment.

[0027] Figure 7 This is a schematic diagram of the structure of an agent reasoning module based on a policy network, provided in one embodiment.

[0028] Figure 8 This is a flowchart illustrating a two-stage joint training strategy provided in one embodiment.

[0029] Figure 9 This is a schematic diagram of multi-task loss function collaborative optimization and gradient flow provided in one embodiment.

[0030] Figure 10 This is a schematic diagram of the internal structure of a computer device provided in one embodiment. Detailed Implementation

[0031] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0032] Figure 1This is a flowchart illustrating a medical data processing method according to one embodiment. Figure 1 The medical data processing method shown can be applied to, for example... Figure 2 The medical data processing system shown is also referenced. Figure 1 and Figure 2 Medical data processing methods include: S11: After mapping medical images, clinical texts and structured test indicators in multimodal medical data to a unified feature dimension through the first encoder, the second encoder and the third encoder respectively, the image feature sequence, the text feature sequence and the indicator feature vector are output. In one embodiment, the multimodal coding module (also known as a multimodal encoder) includes a first encoder, a second encoder, and a third encoder in parallel. Specifically, the first encoder can be a VisionTransformer encoder (also known as a ViT encoder), configured to map medical images in the multimodal medical data to a unified feature dimension and output an image feature sequence. The second encoder can be a Clinical Biomedical Pre-trained Language Model Encoder (also known as a BioClinicalBERT encoder or BERT encoder), configured to map clinical text to a unified feature dimension and output a text feature sequence. The third encoder is configured to map structured test indicators to a unified feature dimension and output indicator feature vectors. The third encoder may include a two-layer fully connected network (Multi-Layer Perceptron, MLP) and a layer normalization unit; the third encoder can also be called an MLP encoder. Specifically, if the multimodal medical data is... ,in, This represents a medical image, where H, W, and C represent the height, width, and number of channels of the image, respectively. Given a clinical text word sequence of length L, It is a word element. For d SThe structured test index vector is defined as follows: R refers to the set of real numbers, y is the reference label, N is the total number of samples, and i represents the i-th sample. In other embodiments, the first encoder may also employ other visual feature extraction networks, such as convolutional neural networks (CNNs), residual networks (ResNets), etc., as long as they can map medical images to the unified feature dimension and output image feature sequences. This application does not limit this. The second encoder may also employ other pre-trained language models for clinical text, such as general BERT, ClinicalBERT, or PubMedBERT, as long as they can map clinical text to the unified feature dimension and output text feature sequences. The third encoder may also employ other structured data mapping networks, such as multilayer perceptrons with Dropout regularization or autoencoders, as long as they can map structured test indices to the unified feature dimension and output index feature vectors.

[0033] In one embodiment, step S11, which maps the medical image to a unified feature dimension using a first encoder and outputs an image feature sequence, includes: dividing the medical image into multiple non-overlapping image blocks; flattening the image blocks and mapping them to a unified feature dimension using a first learnable linear transformation matrix; attaching a first learnable positional encoding vector to each image block to preserve the spatial structure information of the medical image; receiving the image block sequence processed by the first learnable linear transformation matrix and the first learnable positional encoding vector through the first encoder; performing deep feature extraction on the received image block sequence; and outputting an image feature sequence. The deep feature extraction can be a nonlinear semantic transformation based on a multi-head self-attention mechanism and a feedforward network, performed by a multi-layer visual transformer encoder after the first learnable linear transformation matrix (linear projection) and the first learnable positional encoding vector (shallow preprocessing).

[0034] Specifically, medical images can be divided into: The size is The image patch, where P represents the side length of the image patch, is fed into a multilayer vision transformer encoder (also known as a vision transformer) after passing through a first learnable linear transformation matrix and a first learnable position encoding vector to obtain the image feature sequence. : (1) in, This represents a composite mapping function based on a Vision Transformer encoder, which takes a medical image as input. The image is processed sequentially through image block segmentation, flattening, projection onto the first learnable linear transformation matrix, superposition of the first learnable positional encoding vectors, and multi-layer Transformer encoder, resulting in an output image feature sequence. . This is an embedded representation of image classification labels in the global classification labeling. This represents the feature embedding of the j-th image patch. To unify the feature dimensions. Flattening an image patch refers to flattening a P×P×C image patch into a P... 2 A C-dimensional vector. The first learnable linear transformation matrix, i.e., the linear projection matrix E∈R. d ×(P 2 C) The configuration maps the flattened image block vectors to a unified feature dimension d through matrix multiplication. The parameters of the first learnable linear transformation matrix are updated during training through the backpropagation algorithm. This encoding method can preserve the spatial structure information of the image and provide a foundation for subsequent cross-modal fine-grained alignment.

[0035] In one embodiment, the step of mapping the clinical text to a unified feature dimension through the second encoder and outputting the text feature sequence in S11 includes: mapping each word in the clinical text to a corresponding word embedding vector and superimposing a second learnable position encoding vector to preserve the sequence position information of the clinical text; and performing contextual semantic encoding on the word embedding sequence based on the second encoder to output the text feature sequence.

[0036] Specifically, the step of mapping the clinical text to the unified feature dimension through the second encoder and then outputting the text feature sequence includes: The clinical text is converted into a word sequence containing L words. The word sequence contains L words w_i, i=1,2,…,L; Each of the lexical units w_i is mapped to a corresponding word embedding vector, and a second learnable position encoding vector is superimposed to preserve the sequence position information, thereby obtaining a lexical embedding sequence. By appending the text classification tag embedding representation to the first digit of the word embedding sequence, an input sequence containing L+1 vectors is obtained; Based on the second encoder, BioClinicalBERT, contextual semantic encoding is performed on the input sequence to output the text feature sequence. It can be represented as: (2) in An embedded representation of text classification tags. Let i represent the feature embedding of the i-th word, i=1,2,…,L, and the dimension of each feature embedding in the text feature sequence is the unified feature dimension d.

[0037] In one embodiment, the step of S11, which maps the structured test index to a unified feature dimension through a third encoder and outputs the index feature vector, includes: mapping the structured test index to a hidden layer dimension and performing a nonlinear transformation through a first activation function to obtain hidden layer features; mapping the hidden layer features to a unified feature dimension and performing feature distribution stabilization processing through a second activation function and a layer normalization unit to output the index feature vector.

[0038] Specifically, if the structured testing index is The indicator feature vector is Then the structured test index The input is fed into the first fully connected layer of the third encoder, and then passed through the first weight matrix by the third encoder. and the first bias term Perform linear projection to map the structured test indicators to the hidden layer dimension. The hidden layer features are obtained by performing a nonlinear transformation on the output of the linear projection through the first activation function. The hidden layer features H are passed as input to the second fully connected layer of the third encoder, and then passed by the third encoder through the second weight matrix. and the second bias term Perform a linear mapping to transfer the hidden layer features from the hidden layer dimension d. h Mapped to a unified feature dimension d, and then subjected to feature distribution stabilization processing via a second activation function and a layer normalization unit (LayerNorm), the output index feature vector is determined. In one embodiment, the first activation function and / or the second activation function is a modified linear unit function (ReLU), configured to independently perform nonlinear thresholding on each dimension of the input feature vector: setting negative values ​​to zero and keeping positive values ​​unchanged, thereby introducing nonlinear expressive power and suppressing negative feature activation.

[0039] Right now, (3) like Figure 3 As shown, the outputs of the first to third encoders that receive multimodal medical data are all mapped to a unified feature space of dimension d. Through the aforementioned parallel first, second, and third encoders, heterogeneous medical images, clinical texts, and structured test indicators are mapped to a unified feature space of dimension d. dThe unified feature space can achieve preliminary alignment of heterogeneous data feature spaces while preserving the modality-specific semantic information, laying the foundation for subsequent cross-modal fusion and avoiding the dimensionality curse and sparsity problems caused by directly splicing the original high-dimensional features in early fusion.

[0040] S12, through the bidirectional cross-modal cross-attention fusion module, performs pairwise bidirectional cross-attention calculations on the image feature sequence, text feature sequence and indicator feature vector to obtain three sets of bidirectional fusion representations; Specifically, in one embodiment, step S12 includes: taking any two of the image feature sequence, text feature sequence, and index feature vector as the first modality feature sequence and the second modality feature sequence; performing a linear transformation on the first modality feature sequence through a query projection matrix to generate a query matrix; performing a linear transformation on the second modality feature sequence through a key projection matrix and a value projection matrix to generate a key matrix and a value matrix, respectively; calculating the cross-attention output from the first modality to the second modality based on the query matrix, the key matrix, and the value matrix, and calculating the reverse cross-attention output from the second modality to the first modality to obtain a bidirectional cross-attention output set; concatenating the outputs of each attention head in the bidirectional cross-attention output set, and performing a linear projection through the output projection matrix to obtain a bidirectional fused representation.

[0041] Specifically, in one embodiment, step S12 includes: selecting any two from the image feature sequence Zv, the text feature sequence Zt, and the index feature vector Zs, as the first modality feature sequence Zm and the second modality feature sequence Zn, respectively, where m,n∈{v,t,s} and m≠n; the bidirectional cross-modal cross-attention fusion module converts the first modality feature sequence Zm into a projection matrix. Perform a linear transformation to generate the query matrix Qm→n, where d k This represents the dimension of each attention head; the second modality feature sequence Zn is projected onto the key matrix by the bidirectional cross-modal cross-attention fusion module. And value projection matrix A linear transformation is performed to generate the key matrix Kn→m and the value matrix Vn→m. Based on the query matrix Qm→n, the key matrix Kn→m, and the value matrix Vn→m, the bidirectional cross-modal cross-attention fusion module calculates the cross-attention output CrossAttn(Zm,Zn) from the first modality to the second modality, and then calculates the reverse cross-attention output CrossAttn(Zn,Zm) from the second modality to the first modality to obtain the bidirectional cross-attention output set. The bidirectional cross-modal cross-attention fusion module (also known as the BiCMA module) concatenates the outputs of each attention head in the bidirectional cross-attention output set and outputs them through a projection matrix. Perform linear projection to obtain a bidirectional fused representation. .

[0042] Specifically, bidirectional fusion representation It can be calculated using the following multi-head attention formula: =MultiHead( Zm , Zn =Concat( h 1,…, hH ) W O ; ; in, The first modal feature sequence Zm and the second modal feature sequence Zn The bidirectional fusion representation, MultiHead() is the multi-head cross-attention function, hi=CrossAttni( Zm , Zn ) represents the cross-attention output of the i-th attention head, softmax represents the normalized exponential function, and H represents the total number of attention heads. Let R represent the output projection matrix, and let d represent the set of real numbers. k This represents the dimension of each attention head, d represents the uniform feature dimension, and Concat(.) represents the concatenation operation. This indicates the matrix transpose.

[0043] like Figure 4 As shown, taking the bidirectional cross-attention calculation of image feature sequences and text feature sequences as an example: (4) (5) (6) in To learn the query projection matrix, key projection matrix, and value projection matrix in the projection matrix, Dimensions for each attention head Indicates the first The output of the attention head. Equation (5) means that attention aggregation is performed using image features as queries and text features as key-value pairs, so that each image patch in the image feature sequence can adaptively focus on the semantically related text description fragments in the text feature sequence.

[0044] Through the aforementioned bidirectional cross-modal cross-attention fusion module, feature sequences from any two modalities can engage in deep semantic interaction with each other as query objects. Compared to self-attention mechanisms or unidirectional cross-attention mechanisms within a single modality, this module can establish fine-grained semantic correspondences between local lesion regions in images and text description fragments, as well as numerical-semantic associations between image spatial features and structured test indicators, thereby improving the accuracy of feature alignment between heterogeneous modalities.

[0045] S13, dynamically adjust the contribution weights of the three sets of bidirectional fusion representations through a gated adaptive weighted network to output the fusion representation; In one embodiment, the step of dynamically adjusting the contribution weights of the three sets of bidirectional fusion representations through a Gated Adaptive Weighting Network (GAWN) to output the fusion representation includes: adjusting the contribution weights of each bidirectional fusion representation... and its corresponding global classification tags The data is spliced ​​together using a gating weight matrix. W g and gated bias items b g Calculate the gate value g m ; for the gate value g m Normalization is performed to obtain the normalized weights of each modal branch. α m The normalized weights αm That is, the contribution weight of each modal branch in the fusion process: based on contribution weight α m For each bidirectional fusion characterization Z m A weighted summation is performed to output a fused representation, wherein each modal branch corresponds to one of the three sets of bidirectional fused representations, and the modal branch index... ∈{ } , Indicates image-text branching. Indicates the image-indicator branch. This indicates a text-indicator branch.

[0046] Specifically, such as Figure 5 As shown, the calculation process for contribution weights is as follows: (7) (8) in It is the sigmoid activation function. This indicates a splicing operation. The global classification label for the corresponding modality can also be represented as Z. cls , and For learnable parameters, Represents the modal branch index, where m′ is the summation index variable, and the set of all modal branch indices {vt, vs, ts} is traversed; g m ' represents the gating value of the m'-th modal branch; ∑m'exp(g m ′) represents the sum of the gate values ​​of all modal branches after exponential transformation, which serves as the normalization denominator to ensure that the sum of the normalized weights of each modal branch is 1. Equation (8) is obtained through Normalization ensures that the sum of the weights is 1. The final fusion representation is a weighted sum of the modal branches: (9) The gated adaptive weighted network of one embodiment of this application, through the aforementioned gated calculation and normalized weighting, enables the model to automatically identify the most discriminative modality combination when faced with different multimodal medical data inputs. For example, for cases with obvious imaging features, the normalized weights corresponding to the image-text branch will be amplified; while for cases with abnormal structured test indicators as the main manifestation, the weighted sum of the normalized weights corresponding to the structured indicator-related branches will be correspondingly increased, thereby achieving adaptive generation of fusion representation samples. Through the aforementioned gated adaptive weighted network, the model can dynamically adjust the contribution weights of each modality branch in the fusion process according to the characteristics of the currently input multimodal medical data samples, so that the fusion representation adaptively highlights the modality information most discriminative for the current diagnostic task, avoiding the imbalance of modality contributions or loss of key information caused by static weight allocation in different clinical scenarios.

[0047] S14. Based on the fusion representation, output the probability distribution of medical data classification actions based on the policy network, and determine the multimodal fusion classification result based on the probability distribution.

[0048] In one implementation, such as Figure 6 As shown, the agent reasoning module in the medical data processing system may further include a value network. In one embodiment, the policy network is obtained by optimization based on the Proximal Policy Optimization (PPO) algorithm. In other embodiments, the policy network may also be constructed based on other reinforcement learning algorithms, such as a policy network based on an Actor-Critic architecture or a policy network based on a Soft Actor-Critic (SAC) algorithm, as long as it can output the probability distribution of medical data classification actions based on the fused representation.

[0049] Please refer to Figure 7Specifically, in S14, based on the fusion representation, the probability distribution of medical data classification actions is output by the policy network, and the multimodal fusion classification result is determined based on the probability distribution, including: Fusion representation As status input Policy Network Based on state input The first strategy fully connected layer weight matrix and the first strategy bias term A linear projection is performed, followed by a nonlinear transformation using the first policy activation function to obtain the first policy hidden layer features; these features are then input into the weight matrix of the second policy fully connected layer. and the second strategy bias term Perform a linear mapping and generate the probability distribution of medical data classification action 'a' via a normalization function: (10) Where a∈A represents candidate actions in the medical data classification action space, and the medical data classification action space A contains candidate actions that correspond one-to-one with the sample reference labels; Based on the probability distribution of medical data classification actions, the medical data classification action with the highest probability is selected to generate and output the multimodal fusion classification result corresponding to the multimodal medical data.

[0050] The first policy fully connected layer weight matrix, the first policy bias term, the second policy fully connected layer weight matrix, and the second policy bias term are learnable parameters of the policy network. Initial values ​​are generated randomly during training, and gradients are calculated based on the policy gradient loss via backpropagation, and iteratively updated using the proximal policy optimization algorithm. During application, the parameters of the multimodal encoding module, the bidirectional cross-modal attention fusion module, the gated adaptive weighted network, the policy network, and the value network remain fixed after training and do not participate in gradient updates. The nonlinear transformation of the first policy activation function refers to applying a nonlinear mathematical function (e.g., ...) after the first fully connected layer of the policy network. The function processes the linear mapping results, thereby giving the policy network the ability to learn the boundaries of complex classification decisions.

[0051] This embodiment can construct a direct path from multi-source data perception to diagnostic decision output by directly using the fused representation as the state input of the policy network. This enables the perception module and the decision module to be tightly coupled at the structural level, which is different from the separate architecture in the prior art where the fused representation and the decision network are trained independently. This reduces the structural deviation between the fused representation and the downstream decision target.

[0052] In one embodiment, S14, based on the fusion representation, the probability distribution of medical data classification actions is output by the policy network, and the multimodal fusion classification result is determined based on the probability distribution. This further includes: the value network, based on the state input, maps through a fully connected layer symmetric to the policy network to output a value estimate of the current state input. ( ).

[0053] In one embodiment, the method further includes a training phase prior to the application phase. This training phase employs a two-stage joint training strategy to achieve collaborative optimization between the perception module (i.e., the multimodal encoding module, the bidirectional cross-modal cross-attention fusion module, and the gated adaptive weighting module) and the decision-making module (i.e., the agent reasoning module). This two-stage joint training strategy is integrated into a unified training loop. Figure 8 As shown, the training phase includes supervised pre-training, frozen shallow PPO fine-tuning, and progressive loss weight adjustment.

[0054] The first stage (supervised pre-training): Multimodal medical data is input into the multimodal encoding module. After feature fusion via the bidirectional cross-modal attention fusion module and the gated adaptive weighted network to obtain sample fusion representations, these representations are input into the classification head. Using classification loss as the optimization objective, first, second, and third loss weights are set to optimize the classification loss, thereby pre-training the multimodal encoding module and obtaining pre-trained encoding parameters. For example, the first loss weight can be set to its maximum value, while the second and third loss weights can be zero or less than the first loss weight. Specifically, in the first stage of training… The first stage only optimizes the classification objective; the second stage gradually increases the... and For example, adjust to This allows decision loss to participate in backpropagation and guides the fused representation to adjust in a direction that is conducive to outputting high-return medical data classification actions.

[0055] The second stage (PPO fine-tuning) involves freezing the shallow network parameters of the multimodal coding module, retaining the deep network parameters of the multimodal coding module as trainable, using the pre-trained coding parameters as the initial parameters of the multimodal coding module, connecting the sample fusion representation corresponding to the sample multimodal medical data to the policy network and the value network, adjusting the first loss weight, the second loss weight, and the third loss weight, and using the total loss function as the optimization objective to drive the end-to-end collaborative adjustment of the policy network, the value network, the bidirectional cross-modal cross-attention fusion module, the gated adaptive weighted network, and the deep network parameters of the multimodal coding module.

[0056] In one embodiment, the shallow network parameters can be the parameters of the first few layers or the first half of the network layers in the image encoder and text encoder; the deep network parameters refer to the remaining network parameters in the multimodal coding module other than the shallow network parameters.

[0057] Specifically, the end-to-end collaborative adjustment includes: passing the joint gradient of the policy gradient loss, the value network loss, and the supervised classification loss layer by layer from the policy network and the value network to the deep network parameters of the gated adaptive weighted network, the bidirectional cross-modal cross-attention fusion module, and the multimodal coding module through a backpropagation algorithm, thereby achieving gradient sharing across modules; and synchronously updating the parameters of the policy network, the value network, the bidirectional cross-modal cross-attention fusion module, the gated adaptive weighted network, and the deep network parameters of the multimodal coding module in a single optimization iteration. This enables the sample fusion representation to maintain its classification and discrimination capabilities while enhancing its sensitivity to high-reward actions in the medical data classification action space, thus reshaping the fusion representation from a reconstruction-oriented approach to a decision-oriented one.

[0058] This embodiment utilizes the aforementioned two-stage joint training strategy. In the first stage, the multimodal coding module acquires basic feature extraction and cross-modal alignment capabilities. In the second stage, by freezing the shallow network parameters and retaining the deep network parameters as trainable, the gradients of the policy gradient loss and the value network loss can be backpropagated from the decision module to the deep network parameters of the gated adaptive weighted network, the bidirectional cross-modal cross-attention fusion module, and the multimodal coding module. This enables end-to-end collaborative adjustment of the perception module and the decision module at the parameter optimization level, allowing the fusion representation to maintain its classification and discrimination capabilities while enhancing its sensitivity to high-reward actions in the medical data classification action space. This achieves closed-loop optimization from multi-source data perception to fusion classification output.

[0059] Specifically, the steps in the second phase may include: Forward propagation and trajectory acquisition during the training phase: Sample multimodal medical data is input into the multimodal encoding module, and features are fused through the bidirectional cross-modal cross-attention fusion module and the gated adaptive weighted network to obtain a sample fusion representation; the sample fusion representation is then used as the state input of the agent. Feed into the policy network respectively and value network The policy network classifies actions based on the probability distribution of the state input-output medical data. (a∣ The value network outputs a value estimate of the current state based on the state input. ( ); Trajectory Acquisition: Based on the probability distribution of the aforementioned medical data classification actions, action sampling is performed on training sample instances to obtain the action executed at time t; a reward signal is generated based on the consistency between the predicted label corresponding to the executed action and the sample reference label. The reward signal includes a basic reward term, which takes a positive value Rpos when the predicted label and the sample reference label are consistent, and a negative value Rneg when they are inconsistent. The information gain reward term can be determined based on the decrease in the diagnostic category distribution entropy after introducing the modal information of the next stage; trajectory data is collected ( ); Dominance estimation and strategy optimization: Based on the aforementioned trajectory data, calculate the generalized dominance estimate. and the probability ratio of new and old strategies ,in Represent the old policy network parameters; the aforementioned generalized advantage estimate... and the probability ratio The input is given to the near-end policy optimization objective function, which is configured to use a pruning threshold. Constrain the policy update magnitude and generate the policy gradient loss: (11) in, This is the clipping function.

[0060] In one embodiment, a reward signal is defined based on the consistency between the predicted label and the sample reference label. : (12) in and These represent the positive reward when the predicted label matches the sample reference label and the negative penalty when they do not. Specifically, in one implementation, to encourage the agent to choose further information fusion rather than make a hasty decision when uncertainty is high, an additional information gain reward term, rinfo, is introduced. This term can be determined based on the decrease in the diagnostic category distribution entropy after introducing the modality information of the next stage. It is equal to the basic reward term (R... pos The total reward (Rt) and the generalized advantage estimate together with the proximal policy optimization algorithm are influenced by the cumulative reward (Rt) and the generalized advantage estimate. Ultimately, this guides the proximal policy optimization algorithm to optimize a fusion classification strategy that is both accurate and makes full use of multimodal information.

[0061] Parameter update: Based on policy gradient loss and value network loss Calculate the total loss function Based on the total loss function, the parameters θ of the policy network and the parameters of the value network are updated using a proximal policy optimization algorithm. The parameters include the deep network parameters of the multimodal coding module, the network parameters of the bidirectional cross-modal attention fusion module, and the network parameters of the gated adaptive weighted network; wherein the shallow network parameters of the multimodal coding module are kept frozen.

[0062] Among them, the value network loss In To accumulate discounted returns, the algorithm is configured to measure the deviation between the current state value estimate and the long-term cumulative return, thereby assisting the near-end policy optimization algorithm in calculating the generalized advantage estimate.

[0063] The total loss function is: (13) in, For classification loss, for example =CE( y ^, y ), CE ) is the cross-entropy function. y ^ represents the predicted label, and y represents the sample reference label.

[0064] like Figure 9 As shown, the progressive loss weight scheduling strategy enables bidirectional gradient flow between the multimodal encoding module, the policy network, and the value network. The gradients of the policy optimization loss and the value function loss are backpropagated through the policy network and the value network to the deep network parameters of the gated adaptive weighted network, the bidirectional cross-modal cross-attention fusion module, and the multimodal encoding module. This allows the fusion representation to maintain its classification discrimination ability while enhancing its sensitivity to high-reward actions in the medical data classification action space. Meanwhile, the shallow network parameters of the multimodal encoding module remain frozen, maintaining the modality-specific prior knowledge learned in the first stage, thus achieving closed-loop optimization from multi-source data perception to fusion classification output.

[0065] The two-stage scheduling scheme described above is integrated into a unified training loop, as detailed in the following steps. The algorithm completes a closed loop between perception and decision-making: gradients flow through the deep network in the second stage, thereby reshaping the fused representation to highlight decision-related features rather than reconstruction-oriented features; while the weights of the shallow encoder remain consistent with the modality-specific prior knowledge learned in the first stage.

[0066] Specifically, the two-stage scheduling scheme has the following steps: Input: Multimodal sample dataset Pre-training epochs E1; Fine-tuning epochs E2; Pruning threshold ; Output: trained encoder (fenc), policy network Value Network ; Step 1 (Initialization): Initialize encoder fnc, classification head ψ, and policy network. Value Network ; Step 2 (Phase 1: Supervised Pre-training): Set the loss weights as λ1←1.0, λ2←0, λ3←0; and the learning rate as η←5×10. 5 For rounds epoch=1 to E1: Sample a small batch of samples B; Computational fusion representation ←fenc(B); Predict label y^←ψ( ) ; By minimizing the classification loss =CE(y^,y) updates the encoder fnc and the classification head ψ; End the loop; Step 3 (Stage 2: Freeze shallow PPO fine-tuning): Freeze the shallow network parameters of the encoder FINC; Set the loss weights as λ1←0.3, λ2←0.5, λ3←0.2; and the learning rate as η←1×10⁻⁶. 5 ; For rounds epoch=1 to E2: Through the old strategy network Trajectories were collected from sampled cases. ); Calculate generalized dominance estimation and probability ratio ; Calculate PPO loss ← ; Calculate value loss ← ; Calculate total loss ← ; The deep parameters of the policy network, value network, bidirectional cross-modal cross-attention fusion module, gated adaptive weighted network, and encoder FINC are updated by minimizing the total loss; the learning rate η is decayed. End the loop; Step 4 (Return): Return to the trained encoder FINC and policy network Value Network .

[0067] To verify the effectiveness of the multimodal medical data processing method and system proposed in this application, experimental verification was conducted on two publicly available multimodal medical datasets.

[0068] Datasets: The first dataset is the Open-i Indiana University Chest X-ray Collection, containing de-identified chest X-ray images and their corresponding radiology reports. After report-image pairing and filtering to retain one image for each examination, a total of 1,800 multimodal examination samples were retained, covering 12 diagnostic categories, including normal, cardiomegaly, pneumonia, pleural effusion, atelectasis, edema, consolidation, pneumothorax, pulmonary lesions, pulmonary opacities, fractures, and other findings. Each sample includes a 224×224 chest X-ray image, radiology report text, and a 37-dimensional structured indicator vector extracted from the report text using a negation recognition rule. The second dataset is the Cancer Genome Atlas 5 Cancer Dataset (also known as the TCGA-5 Cancer dataset), containing five cancer types: TCGA-BRCA, TCGA-LUAD, TCGA-COAD, TCGA-GBM, and TCGA-KIRC, with 60 cases for each cancer type, totaling 300 samples. Each sample includes a H&E pathology image patch, pathology report text, and 64-dimensional gene expression features. Both datasets were divided into training, validation, and test sets in a 7:1:2 ratio. Detailed statistical information for the datasets is shown in Table 1.

[0069] Table 1 Evaluation Metrics and Experimental Environment: Four metrics were selected for evaluation: accuracy, macro-average area under the receiver operating characteristic curve (Macro-AUC), macro-average F1 score (Macro-F1), and weighted precision. All experiments were performed on four NVIDIA A100 80GB GPUs using the AdamW optimizer, with an initial learning rate of 5 × 10⁻⁶. 5. Batch size is 32, and the total number of training rounds is 80. The first stage, pre-training, lasts for 50 rounds, and the second stage, proximal policy optimization (PPO) fine-tuning, lasts for 30 rounds. Each experiment is run independently 5 times, and the mean and standard deviation are taken.

[0070] Experimental results comparing with baseline methods: To comprehensively evaluate the effectiveness of the proposed model, eight representative baseline methods were selected for comparison. Single-modal methods included the Vision Transformer Baseline Model (ViT-Base) using only images, the BioClinicalBERT Baseline Model using only text, and the Multilayer Perceptron Baseline Model (MLP) using only structured metrics. Multimodal fusion methods included Early Concat, Late Voting, Bilinear Fusion, Standard Transformer Self-Attention Fusion (SA-Fusion), and Multimodal Cooperative Attention Transformer (MCAT). The comparison results of each method on the Open-i dataset are shown in Table 2.

[0071] Table 2 As shown in Table 2, our proposed method achieved the best results across all four metrics. Compared to the best-performing multimodal collaborative attention transformer, it improved accuracy by 2.3 percentage points and macro-average AUC by 2.3 percentage points. The overall performance of the single-modal method was significantly lower than that of the multimodal fusion method, which validates the importance of multi-source information integration for complex medical data classification tasks. The comparison results on the TCGA-5 Cancer dataset are shown in Table 3. This dataset has a relatively small sample size but a higher degree of modal heterogeneity, making it more suitable for testing the model's fusion capabilities in highly heterogeneous scenarios.

[0072]

[0073] Table 3 Our proposed method also maintains a significant performance advantage on the TCGA dataset. Compared to the multimodal collaborative attention transformer, it improves accuracy by 1.67 percentage points and macro-average AUC by 0.20 percentage points. The TCGA dataset involves three heterogeneous modalities: H&E pathology image patches, pathology reports, and gene expression, with greater modal differences, which can further validate the model's fusion capability in highly heterogeneous scenarios.

[0074] Ablation experiments and validation of key module effectiveness: To verify the contribution of each core module to the overall performance, a set of ablation experiments were designed on the Open-i dataset. The ablation schemes included: removing the bidirectional cross-modal cross-attention fusion module (w / o BiCMA), removing the gated adaptive weighted network (w / o GAWN), removing the proximal policy optimization fine-tuning module and retaining only the classification head (w / o PPO), and replacing cross-attention with standard self-attention (Replace BiCMA→SA). The experimental results are shown in Table 4.

[0075]

[0076] Table 4 As shown in Table 4, removing the bidirectional cross-modal cross-attention fusion module resulted in a 6.5 percentage point decrease in accuracy, the most significant performance degradation among all ablation schemes. This fully demonstrates the crucial role of the cross-modal cross-attention mechanism in heterogeneous feature alignment. Replacing the bidirectional cross-modal cross-attention fusion module with standard self-attention resulted in a 5.2 percentage point decrease in accuracy, further confirming the superiority of the bidirectional cross-query approach over self-attention in cross-modal scenarios. Removing the gated adaptive weighted network and the proximal policy optimization fine-tuning module resulted in performance degradations of 2.3 and 1.8 percentage points, respectively, indicating that dynamic modality weighting and end-to-end decision optimization both made substantial contributions to the final performance. A quantitative comparison of intra-class and inter-class distances for each method is shown in Table 5.

[0077]

[0078] Table 5 As shown in Table 5, the proposed method has the smallest average intra-class distance (2.91), the largest average inter-class distance (8.74), and a silhouette coefficient of 0.548, all significantly better than other methods. This indicates that the bidirectional cross-modal cross-attention fusion module, combined with a gated adaptive weighted network, can generate more discriminative and structurally clearer fusion feature representations.

[0079] Figure 10 An internal structural diagram of a computer device in one embodiment is shown. This computer device can specifically be a terminal or a server. Figure 10 As shown, the computer device includes a processor, memory, and network interface connected via a system bus. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and may also store a computer program, which, when executed by the processor, causes the processor to implement the aforementioned methods. The internal memory may also store a computer program, which, when executed by the processor, causes the processor to perform the aforementioned medical data processing methods. Those skilled in the art will understand that… Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0080] In one embodiment, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the steps of the method as described in any of the foregoing embodiments.

[0081] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0082] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0083] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A medical data processing method, characterized in that, The method includes the following steps: After mapping medical images, clinical texts, and structured test indicators in multimodal medical data to a unified feature dimension through the first encoder, second encoder, and third encoder respectively, the output image feature sequence, text feature sequence, and indicator feature vector are output. The image feature sequence, the text feature sequence, and the index feature vector are subjected to pairwise bidirectional cross-attention calculations by the bidirectional cross-modal cross-attention fusion module to obtain three sets of bidirectional fusion representations. The contribution weights of the three sets of bidirectional fusion representations are dynamically adjusted by a gated adaptive weighted network to output the fusion representation; Based on the fusion representation, a probability distribution of medical data classification actions is output based on the policy network, and the multimodal fusion classification result is determined based on the probability distribution.

2. The medical data processing method according to claim 1, characterized in that, The step of mapping the medical image to the unified feature dimension through the first encoder and then outputting the image feature sequence includes: The medical image is divided into multiple non-overlapping image blocks; The image blocks are flattened and mapped to the unified feature dimension through a first learnable linear transformation matrix, and a first learnable position encoding vector is attached to each image block. The first encoder performs deep feature extraction on the image block sequence processed by the first learnable linear transformation matrix and the first learnable position encoding vector to output the image feature sequence.

3. The medical data processing method according to claim 1, characterized in that, The step of mapping the clinical text to the unified feature dimension using the second encoder and then outputting the text feature sequence includes: Each word in the clinical text is mapped to a corresponding word embedding vector, and a second learnable position encoding vector is superimposed to obtain a word embedding sequence. Based on the second encoder, the word embedding sequence is subjected to contextual semantic encoding to output the text feature sequence.

4. The medical data processing method according to claim 1, characterized in that, The step of mapping the structured inspection index to the unified feature dimension using the third encoder and then outputting the index feature vector includes: The structured inspection index is mapped to the hidden layer dimension by the third encoder and then subjected to a nonlinear transformation by the first activation function to obtain the hidden layer features. The hidden layer features are mapped to the unified feature dimension, and the feature distribution is stabilized through the second activation function and layer normalization unit to output the index feature vector.

5. The medical data processing method according to claim 1, characterized in that, The steps of performing pairwise bidirectional cross-attention calculations on the image feature sequence, the text feature sequence, and the indicator feature vector through the bidirectional cross-modal cross-attention fusion module to obtain three sets of bidirectional fused representations include: Any two of the image feature sequence, the text feature sequence, and the index feature vector are used as the first modality feature sequence and the second modality feature sequence; The first modal feature sequence is linearly transformed using a query projection matrix to generate a query matrix; The second modality feature sequence is linearly transformed using the key projection matrix and the value projection matrix to generate the key matrix and the value matrix, respectively. Based on the query matrix, the key matrix, and the value matrix, the cross-attention output from the first modality to the second modality is calculated, and the reverse cross-attention output from the second modality to the first modality is calculated to obtain a bidirectional cross-attention output set; The outputs of each attention head in the bidirectional cross-attention output set are concatenated and linearly projected through the output projection matrix to obtain the bidirectional fusion representation.

6. The medical data processing method according to claim 5, characterized in that, The steps of concatenating the outputs of each attention head in the bidirectional cross-attention output set and performing linear projection through the output projection matrix to obtain the bidirectional fused representation include: The three sets of bidirectional fusion representations are obtained based on the following formula: =MultiHead( Zm , Zn )=Concat( h 1,…, hH ) ; in, The first modal feature sequence Zm and the second modal feature sequence Zn Bidirectional fusion representation, MultiHead( ) is the multi-head cross-attention function, hi = CrossAttni( Zm , Zn ) represents the output of the i-th attention head, and H represents the total number of attention heads. Let R represent the output projection matrix, and let R represent the set of real numbers. d k This represents the dimension of each attention head. d Represents the unified feature dimension, Concat( () indicates a splicing operation.

7. The medical data processing method according to any one of claims 1 to 6, characterized in that, The steps of dynamically adjusting the contribution weights of the three sets of bidirectional fused representations through the gated adaptive weighted network to output the fused representations include: The bidirectional fusion representations and their corresponding global classification labels are concatenated, and the gate value is calculated using the gate weight matrix and the gate bias term. The gate value is normalized to obtain the contribution weight of each modal branch; The three sets of bidirectional fusion representations are weighted and summed based on the contribution weights to output the fusion representation.

8. The medical data processing method according to any one of claims 1 to 6, characterized in that, The steps of determining the multimodal fusion classification result based on the probability distribution of the medical data classification action output by the policy network according to the fusion representation include: The fused representation is used as the state input. Based on the state input, the policy network performs linear projection through the weight matrix of the first policy fully connected layer and the first policy bias term, and performs nonlinear transformation through the first policy activation function to obtain the first policy hidden layer features. The hidden layer features of the first strategy are input into the weight matrix of the fully connected layer of the second strategy and the bias term of the second strategy for linear mapping, and the probability distribution of the medical data classification action is generated by the normalization function. Based on the probability distribution of the medical data classification actions, the medical data classification action with the highest probability is selected to generate and output the multimodal fusion classification result corresponding to the multimodal medical data.

9. The medical data processing method according to claim 1, characterized in that, The method employs a two-stage joint training strategy: The first stage involves inputting multimodal medical data into the multimodal coding module, and then performing feature fusion through the bidirectional cross-modal attention fusion module and the gated adaptive weighted network to obtain the sample fusion representation. This representation is then input into the classification head, where the classification loss is optimized by setting a first loss weight, a second loss weight, and a third loss weight. This optimizes the classification loss and pre-trains the multimodal coding module to obtain pre-trained coding parameters. The second stage involves freezing the shallow network parameters of the multimodal coding module, retaining the deep network parameters of the multimodal coding module as trainable, using the pre-trained coding parameters as the initial parameters of the multimodal coding module, connecting the sample fusion representation corresponding to the sample multimodal medical data to the policy network and the value network, adjusting the first loss weight, the second loss weight, and the third loss weight, and using the total loss function as the optimization objective to drive the end-to-end collaborative adjustment of the policy network, the value network, the bidirectional cross-modal cross-attention fusion module, the gated adaptive weighted network, and the deep network parameters of the multimodal coding module.

10. A medical data processing system, characterized in that, include: The multimodal coding module includes a first encoder, a second encoder, and a third encoder in parallel. The first encoder, the second encoder, and the third encoder are respectively configured to map medical images, clinical text, and structured test indicators in multimodal medical data to a unified feature dimension, and then output image feature sequences, text feature sequences, and indicator feature vectors. The bidirectional cross-modal cross-attention fusion module is configured to perform pairwise bidirectional cross-attention calculations on the image feature sequence, the text feature sequence, and the indicator feature vector to obtain three sets of bidirectional fusion representations. A gated adaptive weighting module includes a gated adaptive weighting network configured to dynamically adjust the contribution weights of the three sets of bidirectional fusion representations to output a fusion representation; The intelligent agent reasoning module includes a policy network configured to output a probability distribution of medical data classification actions based on the fused representation, and to determine a multimodal fusion classification result based on the probability distribution.

11. A computer device, characterized in that, Including processor and memory; The processor is used to execute a computer program stored in the memory to implement the medical data processing method as described in any one of claims 1 to 9.

12. A readable storage medium, characterized in that, The readable storage medium stores a computer program that, when executed by a processor, implements the medical data processing method as described in any one of claims 1 to 9.

Images