A multi-modal data adaptive denoising and missing reconstruction method and system

Abstract

The application discloses a multi-modal data adaptive denoising and missing reconstruction method and system, relates to the technical field of point data denoising and reconstruction, and obtains multi-modal original data to be processed and a modality missing mask; image data in the multi-modal original data is subjected to unsupervised denoising to obtain denoised images, and then multi-modal data is obtained; the multi-modal data is input into a double-flow encoder for processing to obtain multi-modal embedding vectors aligned across modalities; after being mapped and added with position embedding, the modality embedding vectors obtain coding features of each modality; missing modalities are determined based on the modality missing mask; available modality coding features are input into a retrieval-enhanced expert model based on prototype memory for missing reconstruction to obtain multi-modal joint representation; and the multi-modal joint representation is mapped to a task output space through a full connection layer. By introducing unsupervised denoising, double-flow coding alignment and mixed reconstruction of modality knowledge experts, robust representation learning and information complementation are realized under the condition that multi-modal data has noise and modality missing.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of point data denoising and reconstruction, and particularly relates to a multi-modal data adaptive denoising and missing reconstruction method and system. BACKGROUND

[0002] The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute prior art.

[0003] Multi-modal medical data (such as CT, MRI, microscope images, etc.) contains rich complementary information and is widely used in biomedical diagnosis, multi-modal emotion recognition, cross-modal retrieval, etc. However, two core challenges are faced in practical applications:

[0004] Data noise interference: noise (such as Gaussian noise of medical images, Poisson noise of fluorescence microscopes) is easily introduced in the process of multi-modal data acquisition. Traditional denoising methods either rely on a large amount of clean training data (which is difficult to obtain in the biomedical field) or have high computational complexity, which cannot meet the real-time processing requirements of resource-limited scenarios;

[0005] Modality missing problem: factors such as sensor damage, privacy protection, and data acquisition conditions often lead to missing of part of the modal data. Existing methods mainly fall into two categories: one is to synthesize data or features based on generative models (such as VAE, GAN or simple cross-modal mapping networks). This kind of method tries to implicitly learn the mapping relationship between modalities through network parameters, but in the case of severe missing (such as only text input), due to the lack of prior constraints, the generated features often produce 'feature hallucinations' that do not conform to the medical anatomy logic or pathological distribution, resulting in a significant decline in diagnostic reliability; the other is to simply zero-fill or mean-fill the missing modalities based on feature level, which completely ignores the semantic association between modalities, resulting in severe information loss and significant decline in model performance.

[0006] At the same time, existing multi-modal processing frameworks lack collaborative optimization of the whole link of 'denoising-representation-reconstruction', the denoising process easily destroys the semantic association between modalities, and the missing reconstruction lacks effective use of cross-modal knowledge, resulting in insufficient overall processing accuracy and robustness. SUMMARY

[0007] To overcome the deficiencies of the prior art, the present application provides a multi-modal data adaptive denoising and missing reconstruction method and system, which aims to effectively preserve data detail features and improve processing robustness in the presence of missing modalities without clean training data and limited computational resources.

[0008] To achieve the above object, one or more embodiments of the present application provide the following technical solutions:

[0009] In a first aspect, the present application provides a multi-modal data adaptive denoising and missing reconstruction method, comprising:

[0010] Obtaining multi-modal original data to be processed and a modal missing mask;

[0011] Unsupervisedly denoising image data in the multi-modal original data to obtain denoised images, and replacing the image data in the multi-modal original data with the denoised images to obtain multi-modal data;

[0012] Inputting the multi-modal data into a double-flow encoder for processing to obtain cross-modal aligned multi-modal embedding vectors;

[0013] After the modal embedding vectors are mapped and position embedding is added, modal encoding features are obtained, the missing modal is determined based on the modal missing mask, the available modal encoding features are input into a retrieval enhanced expert model based on prototype memory for missing reconstruction to obtain multi-modal joint representation, and the multi-modal joint representation is mapped to a task output space through a full connection layer.

[0014] In a further technical solution, the denoised images are obtained, and specifically:

[0015] Inputting the image data into a lightweight denoising subnetwork, filtering random noise of the image based on the lightweight denoising subnetwork to obtain a preliminary denoised image;

[0016] Performing a pixel rearrangement operation on the preliminary denoised image to destroy noise spatial correlation to obtain a refined image;

[0017] Fusing the preliminary denoised image and the refined image to obtain the denoised image.

[0018] In a further technical solution, the lightweight denoising subnetwork is trained through a Noise2Noise loss, a detail enhancement loss and a consistency loss to obtain a trained lightweight denoising subnetwork.

[0019] In a further technical solution, the double-flow encoder adopts a CLIP model, comprising an image encoder and a text encoder, and is trained through image-text pair contrast learning to encode the image and the text respectively.

[0020] In a further technical solution, the retrieval enhanced expert model based on prototype memory comprises a generative expert branch and a retrieval memory branch, the generative expert branch comprises a visual expert and a text expert, and a cross-modal knowledge transfer mechanism is used to reconstruct the representation of the missing modal; the retrieval memory branch addresses and retrieves from a missing modal prototype memory bank based on the available modal encoding features to reconstruct reference prototype features.

[0021] Further technical solutions, the representation of the missing modality is reconstructed by using a cross-modal knowledge transfer mechanism, specifically: the available modality encoding features are respectively input into multiple modality experts for representation, and the source modality features are generated after being processed by the available modality experts, and the cross-modal generated features are generated after being processed by the missing modality expert.

[0022] Further technical solutions, based on the cross-modal generated features and the reference prototype features, the feature reconstruction is performed through a gated residual mechanism to obtain the missing modality reconstruction representation, and then the missing modality reconstruction representation and the source modality features are spliced to obtain the multi-modal joint representation.

[0023] In a second aspect, the present application provides a multi-modal data adaptive denoising and missing reconstruction system, comprising:

[0024] The data acquisition module is configured to acquire the multi-modal original data to be processed and the modality missing mask;

[0025] The denoising module is configured to perform unsupervised denoising on the image data in the multi-modal original data to obtain denoised images, replace the image data in the multi-modal original data with the denoised images, and obtain the multi-modal data.

[0026] The alignment encoding module is configured to input the multi-modal data into a double-flow encoder for processing to obtain a multi-modal embedding vector aligned across modalities.

[0027] The missing reconstruction module is configured to obtain modality encoding features after the modality embedding vector is mapped and added with a position embedding, determine the missing modality based on the modality missing mask, input the available modality encoding features into a prototype memory-based retrieval enhanced expert model for missing reconstruction to obtain a multi-modal joint representation, and map the multi-modal joint representation to a task output space through a fully connected layer.

[0028] In a third aspect, the present application provides a computer readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the steps of the multi-modal data adaptive denoising and missing reconstruction method of the first aspect.

[0029] In a fourth aspect, the present application provides a computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to implement the steps of the multi-modal data adaptive denoising and missing reconstruction method of the first aspect.

[0030] The above one or more technical solutions have the following beneficial effects:

[0031] The application reconstructs the missing modality representation by introducing unsupervised denoising, double-flow encoding alignment and a retrieval enhanced expert model based on prototype memory, and realizes robust representation learning and information complementation under the condition of noise and modality missing in multi-modal data.

[0032] The application innovatively proposes a retrieval enhanced expert model based on prototype memory, when reconstructing the missing modality, the generative branch utilizes cross-modal expert knowledge for semantic transfer, and the retrieval branch obtains the most relevant semantic prototype by querying external prior knowledge (a double-modal prototype memory bank), and the two are dynamically fused through a gating residual mechanism, so that the reconstruction process of the missing modality contains both the specific details of the input data and the general standard semantic knowledge, effectively overcoming the "feature hallucination" problem of pure generative methods under severe missing, and significantly improving the performance stability and reliability of the model in the incomplete modality scene.

[0033] The application is optimized through an end-to-end global joint loss function (fusion denoising loss, cross-modal contrast loss, task loss and prototype clustering loss), ensuring the collaborative work of the whole process from denoising, alignment to reconstruction, while keeping the model lightweight, realizing high-precision multi-modal information processing.

[0034] The application can complete high-quality denoising without clean labels, the unsupervised denoising mechanism based on NFD is free from the dependence on ideal reference images, and is suitable for various medical scenes of rare real noise data; the detail retention capability is strong, the three-stage progressive denoising strategy effectively retains key structures such as cell boundaries and micro lesions, and meets the clinical diagnosis demand; the robust inference under any modality missing is supported, through the retrieval enhanced expert model based on prototype memory, it is ensured that reliable results can still be output under extreme incomplete conditions; low resource consumption, easy to deploy, the core denoising network is extremely simple, suitable for integration in portable devices or remote diagnosis and treatment platforms; privacy friendly, without uploading original data to the cloud, local processing ensures patient information security; strong universality, can be extended to multiple modalities (voice, physiological signals), multiple tasks (segmentation, classification, generation), and has wide application prospects. BRIEF DESCRIPTION OF DRAWINGS

[0035] The drawings accompanying the specification of the application form part of the application and serve to provide further understanding of the application, the illustrative embodiments of the application and their description serve to explain the application, and do not constitute an improper limitation of the application.

[0036] Figure 1 is a flowchart of a multi-modal data adaptive denoising and missing reconstruction method according to an embodiment of the application;

[0037] Figure 2 is a flowchart of a multi-modal data adaptive denoising and missing reconstruction method according to an embodiment of the application. DETAILED DESCRIPTION

[0038] It should be noted that the following detailed description is exemplary in nature and is intended to provide further description of the application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0039] It is also to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, steps, operations, elements, components, and / or groups thereof, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0040] The embodiments in the application and the features in the embodiments can be combined with each other without conflict.

[0041] Embodiment one

[0042] As shown in Figure 1 , Figure 2 The embodiment discloses a multi-modal data adaptive denoising and missing reconstruction method, which comprises the following steps:

[0043] S1: obtaining multi-modal original data to be processed and a modal missing mask;

[0044] In this embodiment, the multi-modal original data at least includes any one or a combination of multiple kinds of image data and text data, the image data is selected from CT images, MRI images, X-ray images, ultrasound images or fluorescence microscope images, and the text data is selected from doctor's diagnosis report, electronic medical record and patient's subjective emotion description text.

[0045] The multi-modal original data to be processed allows partial modal missing, that is, the input can be single modal or partial modal combination, and the missing modal information is marked by the modal missing mask.

[0046] The modal missing mask is a vector (such as {only text available}) or a binary identifier (such as [1, 0] representing that image data exists and text data is missing), and it is known through the modal missing mask which modal has original data and which modal needs modal reconstruction or completion.

[0047] S2: unsupervised denoising is performed on the image data in the multi-modal original data to obtain a denoised image, the image data in the multi-modal original data is replaced by the denoised image, and multi-modal data is obtained;

[0048] In this embodiment, lightweight unsupervised denoising is performed on the image data. Specifically, a lightweight denoising sub-network is constructed based on NFD technology to denoise the original image. The input is fed into a pre-trained lightweight denoising sub-network (a minimal convolutional neural network containing only three convolutional layers with fewer than 1000 parameters), and a three-stage progressive denoising strategy is employed to obtain a denoised image. The inference process is as follows: random noise in the image is filtered out based on the denoising sub-network to obtain a preliminary denoised image; pixel rearrangement is performed on the preliminary denoised image to destroy the spatial correlation of noise, resulting in a thinned image; the preliminary denoised image and the thinned image are then fused to obtain the final denoised image.

[0049] The training process for the lightweight denoising subnetwork is as follows:

[0050] Phase 1: Preliminary denoising. Two randomly downsampled copies of the same image are generated. , Using the Noise2Noise principle, , The model learns to filter out the most obvious random noise without cleaning the image, resulting in a preliminary denoised image. .

[0051] In the first stage, a convolutional neural network is trained using the Noise2Noise loss function:

[0052]

[0053] in, Loss due to Noise2Noise This is the first noisy copy generated through random downsampling. This is a second noisy copy generated through random downsampling. This is the denoising prediction result of the convolutional network for the first noisy copy.

[0054] The second stage: Background refinement involves "pixel-shuffle." The first stage often leaves behind some grid-like or diagonal artifacts (caused by downsampling), affecting the initially denoised image. Pixel rearrangement disrupts spatial correlation of noise, and the network is retrained to remove artifacts, resulting in a thinned image. This step smooths the background by disrupting the local spatial correlation of pixels, specifically removing these residual background noises.

[0055] Phase 3: Foreground Enhancement and Detail Restoration. This involves fusing the initially denoised images. With thinned image Fine-tuning is performed using detail enhancement loss to obtain the denoised image. To recover the key details such as cell boundaries and micro lesions, the network weights are fine-tuned by re-introducing the original noisy image as a guide against the over-smoothing caused by the first two steps.

[0056]

[0057] where, is the detail enhancement loss, is the denoised output of the network for the input image is the sub-image obtained by down-sampling the input image with a stride of 2 using the first diagonal mean down-sampling kernel , is the denoised output of the network for the input image , is the sub-image obtained by down-sampling the input image with a stride of 2 using the second diagonal mean down-sampling kernel , is the denoised output of the network for the input image .

[0058] To prevent overfitting, an additional consistency loss is introduced to constrain the matching degree between the denoised image and the down-sampled output:

[0059]

[0060] where, is the consistency loss, is the input image, is the denoised result of the network for the input image is the low-resolution view obtained by using the second diagonal mean down-sampling kernel , is the low-resolution view obtained by using the first diagonal mean down-sampling kernel .

[0061] The two down-sampled views , are generated by stride-2 convolution using the diagonal mean kernel:

[0062]

[0063] where, is the first diagonal mean down-sampling kernel, is the second diagonal mean down-sampling kernel.

[0064] In the detail enhancement stage of the image denoising phase, the total loss function + ​​The minimal convolutional neural network is retrained to purify the noise while retaining key texture features for subsequent semantic alignment. Total loss Ensure that the texture of the denoised image (such as the text OCR area, object edge) is not lost, and adapt to the feature extraction requirements of the CLIP image encoder.

[0065] During the training process of the denoising subnetwork, the cross-modal similarity of the pre-trained CLIP model is introduced as an additional supervision signal to construct a semantic preservation loss, preventing over-smoothing during the denoising process that causes key discriminative features to be filtered out, thereby maintaining strong association between the image and text semantics while purifying noise.

[0066] Through the above technical solution, in the adaptive denoising step, the image modal adopts a multi-stage pixel rearrangement refinement + detail enhancement retraining process, and the NFD lightweight denoising architecture adopted by the present application has good modal expansion, and can be processed through visual conversion according to requirements to realize targeted denoising of text, voice and other modal data.

[0067] S3: inputting the multi-modal data into the dual-flow encoder for processing to obtain cross-modal aligned multi-modal embedding vectors;

[0068] In this embodiment, based on the missing modal mask, the encoder input path corresponding to the missing modal is gated (such as input zero or shielding), and only the feature extraction path of the valid modal is activated.

[0069] The dual-flow encoder adopts a CLIP architecture model, including an image encoder and a text encoder , which respectively process the denoised image and text data to realize cross-modal feature alignment and encoding.

[0070] The image encoder receives the denoised image , and outputs an image embedding vector , and the text encoder receives the text description , and outputs a text embedding vector , wherein is the dimension of the embedding vector, i.e. the dimension of the shared semantic space to which the image and text are mapped.

[0071] The dual-flow encoder is pre-trained using a contrastive learning objective:

[0072]

[0073] wherein, is the contrastive loss function, which is used to pull the distance between the matched image-text pairs in the embedding space, and push the distance between the unmatched pairs. Temperature coefficient, used to adjust the sharpness of softmax, control the sensitivity of the model to distinguish positive and negative samples; Total number of samples in a batch; Index of the th sample in the batch, used to traverse all negative samples (including itself as a positive sample); Index of the th text sample in the batch. In the contrastive loss, represents the similarity between the current image embedding and the th text embedding. When corresponds to a positive sample (i.e., the text paired with ), this term is the positive sample similarity.

[0074] Through contrastive learning, similar image-text pairs are close in the -dimensional embedding space, and different pairs are far apart, achieving cross-modal alignment of image embedding vectors and text embedding vectors.

[0075] S4: After mapping and adding position embedding, the modal embedding vector obtains each modal encoding feature. Based on the modal missing mask, the missing modal is determined, and the available modal encoding feature is input into the prototype-based retrieval-augmented expert model to perform missing reconstruction (i.e., semantic reconstruction of the missing modal), to obtain a multi-modal joint representation, which is mapped to a task output space through a fully connected layer.

[0076] In this embodiment, the prototype-based retrieval-augmented expert model (PRAEM) is constructed as a dual-path architecture containing a "generative expert branch" and a "retrieval memory branch" to overcome the problem of "feature hallucination" in the severe missing scenario when relying solely on a generative expert network.

[0077] (I) Generative Expert Branch

[0078] The generative expert branch uses a modal knowledge expert hybrid model for missing reconstruction, which includes multiple modal knowledge experts, specifically visual experts and text experts.

[0079] In the present invention, missing modal representation reconstruction does not mean restoring the original pixel or text string, but generating an equivalent semantic representation of the missing modal in the shared semantic space. The core process is: when a certain modality (such as text) is identified as missing based on the modal missing mask, the system uses cross-modal knowledge transfer mechanism to input the existing feature of another modality (such as image) into the corresponding missing modal expert (text expert) model.

[0080] First, the pre-processing and unification of features are carried out. Before the mixed model of the modal knowledge expert is processed, the multi-modal embedding vector aligned across modes is processed to obtain the corresponding encoding features. The multi-modal embedding vector aligned across modes is denoted as , When , the image embedding vector is denoted as , and when , the text embedding vector is denoted as , both of which have a dimension of . In order to adapt to the subsequent processing requirements, the image embedding vector and the text vector are mapped to a fixed dimension by a projection matrix.

[0081] For single-modal features , they are mapped to a fixed dimension by a projection matrix:

[0082]

[0083] Among them, is the modal index, indicating the modal (e.g. = indicates the visual modal, and = indicates the text modal); is the time step or sequence position index, indicating the th feature vector in the modal sequence; is the length of the modal sequence, that is, each modal input is represented as a sequence containing time steps (or tokens) (by sampling / truncating each modal to align to the same sequence length ); is the original feature vector of modal at the th time step; is the embedding representation matrix of modal after linear projection, each row of which corresponds to an embedding vector of a time step; is a learnable projection matrix specific to modal .

[0084] is learnable and is optimized jointly with the downstream task. The projection matrix realizes the mapping from the general semantic space to the fixed dimension cross-domain spatial mapping, which enables the model to filter and reshape the most critical features for the current diagnosis or recognition task from the general-purpose embedding vectors.

[0085] Subsequently, by adding position embeddings , to the features, the sequence units of different modalities (such as words of text, spatial patches of images) are provided with position identification encoding in space or time:

[0086]

[0087] to ensure that the modal transformer expert network can perceive the structured distribution information of the data when processing the features. Among them, the encoded features, .

[0088] Then, a visual expert and a text expert are respectively constructed as generative branches, and both the visual expert and the text expert adopt a Transformer encoder architecture.

[0089] The core architecture of each expert adopts a modal transformer expert network, which is composed of Q identical transformer modules, each of which contains a multi-head self-attention (MSA) mechanism and a feed-forward neural network (FFN). The encoded features are input into the expert model, and the transformed features are output, and this process utilizes the discriminative knowledge of the modal itself.

[0090] When a modal is missing (for example, image missing), the existing modal (such as text) is input into the expert model of the other party (such as the visual expert), and the cross-modal generation features of the missing modal are generated by utilizing the cross-modal correlation ability of the other party expert, denoted as .

[0091] (II) Retrieval Memory Branch

[0092] In order to introduce external prior knowledge, the embodiment maintains a dual-modal prototype memory bank (Dual-Modal Prototype Memory Bank), denoted as . Among them, the image prototype memory bank stores high-level pathological semantic prototypes of the image modal (such as the feature cluster center of “ground glass shadow”, “nodule”), and the text prototype memory bank stores semantic prototypes of the text modal, is the number of prototypes, is the feature dimension.

[0093] When it is detected based on the modal missing mask that a modal (for example, the image modal) is missing, and another modal (for example, the text modal) is available, the system utilizes the text encoded features As Query, the image prototype memory bank is addressed for retrieval.

[0094] The similarity between Query and each prototype vector in the image prototype memory bank is calculated and converted into an addressing weight :

[0095]

[0096] where is the Query projection matrix, is the temperature coefficient, is the i-th prototype vector in the image prototype memory bank, is the i-th prototype vector in the image prototype memory bank. According to the addressing weight, the reference prototype feature is reconstructed from the memory bank :

[0097] The core of this step is no longer to generate missing information in the air, but to consult the most suitable standard pathology prototype according to the existing information.

[0098] In order to take into account the specific information of the input (provided by the generative expert branch) and the general prior knowledge (provided by the retrieval memory branch), the model adopts a gated residual mechanism for feature reconstruction, and the final missing modality reconstruction representation is calculated as follows:

[0099]

[0100]

[0101]

[0102] where is the Sigmoid activation function, is the learnable weight matrix of the gating mechanism, which is a parameter matrix that is constantly updated and optimized during the model training process, not a fixed value, and is randomly initialized; is the concatenation operation, is the bitwise multiplication, is the dynamic gating coefficient, according to which the model automatically determines whether to trust the generated details more or the retrieved standard prototype more.

[0103] In order to make full use of the real existing modality information and the reconstructed modality information, the high-level features of the available modalities after being encoded by their corresponding expert networks are denoted as ​​​​(source modality feature), splicing the source modality feature of the available modality with the missing modality reconstruction representation in the feature dimension to obtain the final multi-modal joint representation :

[0104]

[0105] Finally, the multi-modal joint representation is input into the full connection layer (task decoder) to finally obtain the label :

[0106]

[0107] wherein, is the full connection layer parameter (usually shared by all modalities) used to predict the label in the model training stage.

[0108] In the training stage, in order to ensure that the prototype memory bank can learn representative modality features, in addition to the conventional task loss (classification cross-entropy or regression mean square error) and cross-modal contrast loss , a prototype clustering loss is additionally introduced:

[0109]

[0110] wherein, is the real existing modality feature, indicates the stop gradient conduction, is a hyperparameter.

[0111] In order to realize end-to-end model optimization, the embodiment adopts a global joint loss function for training. The loss function is composed of three parts:

[0112] The global joint loss function includes NFD denoising loss (detail enhancement loss and consistency loss) for guiding the lightweight denoising network to remove modality-specific noise; CLIP cross-modal similarity loss (contrast loss ); PRAEM expert training loss for guiding the task loss ( ) and prototype clustering loss ( ) of the downstream task.

[0113] The global joint loss function is:

[0114]

[0115] wherein, The real modal features are close to the prototypes in the memory bank, and the prototypes are updated to match the data distribution, so that the retrieved reference prototype features have clear semantic representation; is a hyperparameter used to control the relative importance or contribution of the prototype clustering loss in the total loss .

[0116] The model parameters are optimized by the global joint loss to obtain a trained multi-modal medical image processing model. After the model is trained, the multi-modal data to be processed (containing noise / missing) is input, and the robust multi-modal processing result (such as segmentation result, emotion recognition result, cross-modal retrieval result) is output through the denoising, encoding and reconstruction process.

[0117] In summary, the present application does not require clean training data, has lightweight characteristics and high processing accuracy, and shows strong robustness in the case of missing modalities, and can be widely used in the fields of biomedical diagnosis, multi-modal emotion recognition, cross-modal retrieval and the like.

[0118] Table 1 is a simulation experiment of the method of the present application using the FMD data set (fluorescence microscope images). The evaluation index is PSNR, which is used to evaluate the overall performance of denoising. The inference time can show the lightweight of the present model. The method of the present application improves the performance of image denoising.

[0119] Table 1 Comparison of accuracy of the present application and other algorithms

[0120]

[0121] Table 2 is a simulation experiment of the method of the present application using the FMD data set (fluorescence microscope images). The evaluation index is WA / UA, which is used to evaluate the overall performance of the missing modality reconstruction. The method of the present application improves the performance of the missing modality reconstruction.

[0122] Table 2 Comparison of accuracy of the present application and other algorithms

[0123]

[0124] In summary, the present application proposes a multi-modal medical image adaptive denoising and missing reconstruction method, which has the working principle similar to the progressive processing chain of "purification-alignment-enhancement", and has the advantages of targeted modeling of multi-modal data noise characteristics and missing patterns. Specifically, the present application develops an NFD lightweight denoising module, which is adaptive to the modality-specific noise of medical images (such as quantum noise of CT and bias field artifacts of MRI), and eliminates interference while preserving lesion details (such as small nodules and gray-white matter boundaries); the CLIP cross-modal encoding module is introduced to map the denoised multi-modal data to a unified semantic space, and establish a high-level association between images and texts (such as the semantic alignment of "pulmonary nodule" image features and diagnostic text). In addition, the present application develops a prototype memory-based retrieval enhanced expert model module to realize robust feature reconstruction and fusion when the modal is missing. This method combines the detail preservation capability of lightweight denoising and the semantic association capability of cross-modal knowledge. Due to these innovative designs, experiments on challenging tasks such as chest CT / MRI multi-modal diagnosis and pathological section-report correlation analysis consistently demonstrate the excellent performance and generalization ability of the proposed model in noise suppression, missing robustness and diagnostic accuracy.

[0125] Embodiment two

[0126] The present embodiment discloses a multi-modal data adaptive denoising and missing reconstruction system, comprising:

[0127] The data acquisition module is configured to acquire the multi-modal original data to be processed and the modal missing mask;

[0128] The denoising module is configured to perform unsupervised denoising on the image data in the multi-modal original data to obtain denoised images, replace the image data in the multi-modal original data with the denoised images, and obtain multi-modal data;

[0129] The alignment encoding module is configured to input the multi-modal data into a double-flow encoder for processing to obtain a cross-modal aligned multi-modal embedding vector;

[0130] The missing reconstruction module is configured to obtain modal encoding features after mapping and adding position embedding to the modal embedding vector, determine the missing modal based on the modal missing mask, input the available modal encoding features into a prototype memory-based retrieval enhanced expert model for missing reconstruction, obtain a multi-modal joint representation, and map the multi-modal joint representation to a task output space through a full connection layer.

[0131] The system can effectively deal with the image noise interference and multi-modal data missing problems existing in the medical scene, realize high-quality and robust medical image restoration and intelligent analysis under the premise of no clean reference image and low computing resource consumption, and is suitable for various application scenarios such as clinical auxiliary diagnosis, remote medical treatment and edge deployment.

[0132] Embodiment three

[0133] The purpose of this embodiment is to provide a computing device, including a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method of embodiment one.

[0134] Embodiment four

[0135] The purpose of this embodiment is to provide a computer-readable storage medium, a computer-readable storage medium having a computer program stored thereon, wherein the program is executed by a processor to perform the steps of the method of embodiment one.

[0136] The steps and methods involved in the devices of embodiments three and four above correspond to embodiment one, and the specific embodiments can be referred to the relevant description part of embodiment one. The term "computer-readable storage medium" should be understood to include a single medium or multiple media of one or more instruction sets; it should also be understood to include any medium capable of storing, encoding or carrying instruction sets for execution by a processor and causing the processor to perform any of the methods in the present application.

[0137] Those skilled in the art should understand that the above-mentioned modules or steps of the present application can be realized by a general computer device, alternatively, they can be realized by program codes executable by a computing device, so that they can be stored in a storage device for execution by a computing device, or they can be made into individual integrated circuit modules, or a plurality of modules or steps among them can be made into a single integrated circuit module. The present application is not limited to any specific combination of hardware and software.

[0138] The above description is only the preferred embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

[0139] The above description of the specific embodiments of the present application in combination with the accompanying drawings is not a limitation on the protection scope of the present application, and those skilled in the art should understand that various modifications or changes made on the basis of the technical solutions of the present application without creative labor are still within the protection scope of the present application.

Claims

1. A method for multi-modal data adaptive denoising and missing reconstruction, the method comprising: The method comprises the following steps: acquire multi-modal original data to be processed and a modality missing mask; unsupervised denoising is performed on image data in the multi-modal original data to obtain denoised images, the image data in the multi-modal original data is replaced with the denoised images, and multi-modal data is obtained; the multi-modal data is input into a double-flow encoder for processing to obtain cross-modal aligned multi-modal embedding vectors; after the modality embedding vectors are mapped and position embedding is added, modality encoding features are obtained, a missing modality is determined based on the modality missing mask, available modality encoding features are input into a retrieval enhanced expert model based on prototype memory for missing reconstruction, and multi-modal joint representation is obtained, which is mapped to a task output space through a full connection layer; the retrieval enhanced expert model based on prototype memory comprises a generative expert branch and a retrieval memory branch, the generative expert branch comprises a visual expert and a text expert, and a representation of the missing modality is reconstructed by using a cross-modal knowledge transfer mechanism, specifically as follows: the available modality encoding features are input into multiple modality experts for representation, source modality features are generated after being processed by the available modality experts, and cross-modal generative features are generated after being processed by the missing modality experts; the retrieval memory branch performs addressing and retrieval from a missing modality prototype memory based on the available modality encoding features to reconstruct reference prototype features; based on the cross-modal generative features and the reference prototype features, feature reconstruction is performed through a gated residual mechanism to obtain missing modality reconstruction representation, and then the missing modality reconstruction representation is spliced with the source modality features to obtain the multi-modal joint representation.

2. The method of claim 1, wherein, The denoised images are obtained in the following manner: the image data is input into a lightweight denoising subnetwork, random noise of the image is filtered based on the lightweight denoising subnetwork to obtain a preliminary denoised image; a pixel rearrangement operation is performed on the preliminary denoised image to destroy noise spatial correlation, and a refined image is obtained; the preliminary denoised image and the refined image are fused to obtain the denoised image.

3. The method of adaptive denoising and missing reconstruction of multi-modal data as claimed in claim 2, wherein, The lightweight denoising subnetwork is trained through a Noise2Noise loss, a detail enhancement loss and a consistency loss to obtain a trained lightweight denoising subnetwork.

4. The method of adaptive denoising and missing reconstruction of multi-modal data as claimed in claim 1, wherein, The double-flow encoder adopts a CLIP model, comprising an image encoder and a text encoder, and the image-text pair is trained through contrastive learning to encode the image and the text respectively.

5. A multi-modal data adaptive denoising and missing reconstruction system, characterized in that, The method comprises the following steps: a data acquisition module configured to acquire multi-modal original data to be processed and a modality missing mask; a denoising module configured to perform unsupervised denoising on image data in the multi-modal original data to obtain denoised images, replace the image data in the multi-modal original data with the denoised images, and obtain multi-modal data; an alignment encoding module configured to input the multi-modal data into a double-flow encoder for processing to obtain cross-modal aligned multi-modal embedding vectors; a missing reconstruction module configured to obtain modality encoding features after mapping and adding position embedding to modality embedding vectors, determine a missing modality based on the modality missing mask, input available modality encoding features into a retrieval enhanced expert model based on prototype memory for missing reconstruction, obtain multi-modal joint representation, and map the multi-modal joint representation to a task output space through a full connection layer. The prototype memory-based retrieval enhanced expert model comprises a generative expert branch and a retrieval memory branch, the generative expert branch comprises a visual expert and a text expert, and a cross-modal knowledge transfer mechanism is used to reconstruct the representation of a missing modality, specifically: the available modality encoding features are respectively input into multiple modality experts for representation, the source modality features are generated after being processed by the available modality experts, and the cross-modal generated features are generated after being processed by the missing modality experts; The retrieval memory branch addresses and retrieves the reference prototype features from the missing modality prototype memory library based on the available modality encoding features; Based on the cross-modal generated features and the reference prototype features, the feature reconstruction is performed through a gating residual mechanism to obtain the missing modality reconstruction representation, and then the missing modality reconstruction representation is spliced with the source modality features to obtain the multi-modal joint representation.

6. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the steps of the multi-modal data adaptive denoising and missing reconstruction method in any one of claims 1-4.

7. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the steps of the multi-modal data adaptive denoising and missing reconstruction method in any one of claims 1-4.

Images