Medical image analysis method and system based on multi-modal image fusion

Abstract

The invention discloses a medical image analysis method and system based on multi-modal image fusion, and relates to the technical field of medical image processing, and the medical image analysis method comprises the steps: generating a training data set; the training data set is composed of a medical image RGB domain image and a RAW domain image pair which are subjected to pixel-level alignment and labeling; constructing a target detection network; the target detection network comprises a backbone network and a detection head, a bimodal feature enhancement module is introduced between the backbone network and the detection head to realize deep fusion of cross-modal features, and features of an RGB domain image and an RAW domain image are respectively enhanced in semantic and spatial dimensions by using a dual enhancement mechanism; training the target detection network by using the training data set to obtain a medical image detection model; and analyzing a medical image acquired in real time by using the medical image detection model so as to quickly and accurately realize detection and classification of a lesion area in the medical image.

Description

Technical Field

[0001] The present application relates to the technical field of medical image processing, and specifically to a medical image analysis method and system based on multimodal image fusion. Background Art

[0002] With the rapid development of medical imaging technology, imaging devices such as CT, MRI, and ultrasound have become core tools for clinical diagnosis and treatment. However, these devices usually store and display image information in different data formats (such as RAW domain, RGB domain, etc.). RAW domain images can retain the original information collected by the sensor, such as high dynamic range and grayscale details, while RGB domain images enhance the visual effect of the image through post-processing, making it easier for doctors to make intuitive diagnoses.

[0003] Existing medical image analysis systems usually rely on data from a single domain for processing, such as using only RAW domain images for lesion area detection, or only using RGB domain images for image classification. This single-modality processing approach has the following problems: Although RAW domain images retain the details of the original data, they lack visual enhancement and are difficult to use for rapid diagnosis.

[0004] Although RGB domain images are intuitive, some original information may be lost during the processing, such as subtle grayscale differences, which reduces the reliability and accuracy of detection. Summary of the invention

[0005] In order to overcome the limitations of single modality processing technology, this application proposes a medical image analysis method and system based on multimodal image fusion. This application adopts a multimodal medical image analysis method that can integrate the image characteristics of RAW domain and RGB domain to improve the accuracy and efficiency of lesion detection.

[0006] On the one hand, the present application is implemented through the following technical solutions: A medical image analysis method based on multimodal image fusion, the medical image analysis method comprising: Generate a training data set; the training data set consists of pixel-level aligned and annotated medical image RGB domain image and RAW domain image pairs, wherein the RAW domain images include directly acquired real RAW domain images and synthetic RAW domain images generated by preprocessing the RGB domain images; Constructing an object detection network; the object detection network includes a backbone network and a detection head, and introducing a dual-modal feature enhancement module between the backbone network and the detection head to achieve deep fusion of cross-modal features, and using a dual enhancement mechanism to enhance the features of RGB domain images and RAW domain images in semantic and spatial dimensions respectively; The target detection network is trained using the training data set to obtain a medical image detection model; wherein the RGB domain image and the RAW domain image are used as inputs of the target detection network, and the annotation information is used as outputs of the target detection network; The medical image detection model is used to analyze the medical images collected in real time, so as to quickly and accurately detect and classify the lesion areas in the medical images.

[0007] In some embodiments, the constructed target detection network includes two backbone networks, three fusion modules, a neck network and a detection head; RGB domain images and RAW domain images are respectively input into two backbone networks to extract 8x, 16x and 32x downsampled RGB features and RAW features, which are then input into three fusion modules to achieve deep fusion semantic enhancement of cross-modal features. They then enter the neck network for further feature fusion processing, and finally the detection results are output through the detection head.

[0008] In some embodiments, the fusion module includes: a dynamic fusion and channel enhancement module, and a dynamic spatial interaction and enhancement module; Among them, the dynamic fusion and channel enhancement module is used to realize the fusion enhancement and channel enhancement of RGB features and RAW features; in the fusion enhancement stage, after splicing the RGB features and RAW features in the channel dimension, multi-scale global and local features are extracted through dynamic convolution and inverse bottleneck layer; the dynamic convolution combines deep separation convolution and point-by-point convolution to capture the spatial interaction relationship between modalities; the inverse bottleneck layer realizes nonlinear optimization of features by expanding and compressing channel dimensions; in the channel enhancement stage, global average pooling and global maximum pooling are used to provide global statistical information of features and generate dynamic channel weights; the global statistical information is nonlinearly transformed through the fully connected layer to generate independent dynamic weights suitable for RGB features and RAW features; the dynamic weights are multiplied element by element with the fusion enhanced features, each modality is adaptively optimized, and the original feature information is retained through the residual path, and the enhanced modal features are output to the dynamic spatial interaction and enhancement module; The dynamic spatial interaction and enhancement module extracts multi-scale spatial features from the input RGB features and RAW features respectively; then, the multi-scale spatial features are compressed in the channel dimension by point-by-point convolution to generate single-channel RGB features and RAW feature representations; then, the RGB features and RAW features are concatenated in the channel dimension, and further compressed and fused into a single channel representation by point-by-point convolution; the fused features are dynamically enhanced by a channel attention mechanism, and then enhanced features suitable for RGB domain images and RAW domains are generated by an independent dynamic convolution module; finally, the optimized RGB features and RAW features are output.

[0009] In some embodiments, the dynamic spatial interaction and enhancement module uses a convolutional pyramid to extract multi-scale spatial features through depth-separated convolution kernels of different scales; The channel attention mechanism combines global average pooling and convolutional layers to generate dynamic channel weights to highlight the interactive relationship between salient feature regions and modalities.

[0010] In some embodiments, the training data set generation process includes: Perform pixel-level alignment and target detection and annotation on real RGB domain images and RAW domain image pairs; A synthetic RAW domain image is generated by preprocessing the real RGB domain image, and pixel-level alignment and object detection and annotation are performed on the real RGB domain image and the synthetic RAW domain image generated; The RGB domain image and RAW domain image pairs that are aligned and annotated at the pixel level are uniformly adjusted to the same input size, and a series of image pre-processing operations are performed to enhance data diversity and robustness, thereby generating the training data set.

[0011] In some implementations, generating a synthetic RAW domain image by preprocessing a real RGB domain image specifically includes: The RGB domain image is subjected to inverse global tone mapping, inverse gamma correction, inverse color correction, inverse white balance, Bayer arrangement, and demosaicing to generate a synthetic RAW domain image that matches the RAW data format collected by the sensor.

[0012] In some embodiments, during the training process of the target detection network, the loss function used is: a weighted sum of classification loss, bounding box regression loss, and alignment consistency loss; and the weights of the classification loss, bounding box regression loss, and alignment consistency loss are dynamically adjustable.

[0013] In some embodiments, the classification loss uses cross entropy loss or focal loss to handle the category imbalance problem, ensuring that the model can accurately predict the target category; the bounding box regression loss uses GIoU loss to measure the degree of overlap between the predicted box and the true box, and optimizes the positioning accuracy of the bounding box; the consistency loss uses L2 loss based on feature maps or loss based on pixel differences.

[0014] On the other hand, the present application also proposes a medical image analysis system based on multimodal image fusion, the medical image analysis system comprising: An image processing module, the image processing module is used to generate a training data set; the training data set is composed of pixel-level aligned and annotated medical image RGB domain image and RAW domain image pairs, wherein the RAW domain images include directly acquired real RAW domain images and synthetic RAW domain images generated by preprocessing RGB images; A model building module, wherein the model building module is used to build an object detection network, wherein the object detection network includes a backbone network and a detection head, and a dual-modal feature enhancement module is introduced between the backbone network and the detection head to achieve deep fusion of cross-modal features, and a dual enhancement mechanism is used to enhance the features of RGB domain images and RAW domain images in semantic and spatial dimensions respectively; A model training module, wherein the model training module trains the target detection network using the training data set to obtain a medical image detection model; wherein the RGB domain image and the RAW domain image are used as inputs of the target detection network, and the annotation information is used as outputs of the target detection network; And, an analysis module, which uses the medical image detection module to analyze the medical images collected in real time, so as to quickly and accurately detect and classify the lesion areas in the medical images.

[0015] In some embodiments, the image processing module includes: A preprocessing unit, the preprocessing unit is used to preprocess the RGB domain image to generate a synthetic RAW domain image; A pixel-level alignment unit, wherein the pixel-level alignment unit is used to achieve pixel-level alignment of an RGB domain image and a RAW domain image pair; A labeling unit, wherein the labeling unit is used to label the detection target on the RGB domain image and the RAW domain image; And, an image adjustment and expansion unit, which adjusts the pixel-level aligned and labeled RGB domain image and RAW domain image pairs to the same input size, and performs a series of image pre-processing operations to enhance data diversity and robustness, thereby generating the training data set.

[0016] The present application proposes a medical image analysis method and system based on multimodal image fusion. Based on the design of multimodal input, the complementary information of RAW domain and RGB domain images is fully utilized to overcome the limitations of single-modal analysis methods. At the same time, by introducing a dual feature enhancement module, the extraction and fusion capabilities of cross-modal features are significantly optimized, the important feature information between channels is effectively highlighted, and the significance of the target area is improved in the spatial dimension, thereby improving the detection accuracy and efficiency. The present application proposes a medical image analysis method and system based on multimodal image fusion. By constructing two feature extraction networks with the same structure, deep learning feature extraction is performed on RGB domain images and RAW domain images respectively, feature fusion between modalities is optimized, and semantic consistency of cross-modal alignment is ensured, thereby achieving efficient integration of multimodal features and reducing the impact of distribution differences between modalities on detection tasks, thereby achieving efficient target detection and being able to adapt to various needs in complex medical imaging scenarios. The medical image analysis method and system based on multimodal image fusion proposed in this application adopts a modularly designed model structure to improve the scalability and adaptability of the system. It not only supports real-time target detection, but also can flexibly adapt to the target detection needs of various medical imaging devices and different types of diseases, providing reliable technical support for diversified clinical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the embodiments of the present application, constitute a part of the present application, and do not constitute a limitation on the embodiments of the present application. In the drawings: Figure 1 A flow chart of the medical image analysis method proposed in the embodiment of the present application; Figure 2 Principle block diagram of the target detection network constructed for the embodiment of the present application; Figure 3 This is a schematic diagram of the fusion module structure of an embodiment of the present application; Figure 4 This is a schematic diagram of the structure of the dynamic fusion and channel enhancement module of an embodiment of the present application; Figure 5 This is a schematic diagram of the structure of the fusion enhancement part of an embodiment of the present application; Figure 6 This is a schematic diagram of the structure of the dynamic space interaction and enhancement module of an embodiment of the present application; Figure 7 This is a functional block diagram of the medical image analysis system proposed in an embodiment of the present application. DETAILED DESCRIPTION

[0018] In order to make the objectives, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with examples and drawings. The illustrative implementation scheme of the present application and its description are only used to explain the present application and are not intended to limit the present application.

[0019] Embodiment: The existing medical image analysis technology mainly relies on single-modality processing, which has the problems of reduced detection accuracy and low diagnostic efficiency. In view of this, this embodiment proposes a medical image analysis method based on multimodal image fusion. The medical image analysis method proposed in this embodiment combines the advantages of RAW domain and RGB domain images, optimizes the feature extraction and fusion process, and improves the accuracy and real-time performance of medical image analysis.

[0020] like Figure 1 As shown, the medical image analysis method proposed in this embodiment specifically includes the following steps: Step 1, generate a training data set; the training data set consists of pixel-level aligned and annotated medical image RGB domain image and RAW domain image pairs, wherein the RAW domain images include directly acquired real RAW domain images and synthetic RAW domain images generated by preprocessing RGB images.

[0021] In this step 1, the training data in the training data set is divided into two parts: One part is: directly using real RGB domain image and RAW domain image pairs, and performing pixel-level alignment and target detection annotation; this part of the data ensures the real modal characteristics.

[0022] The other part is: in order to increase the amount of data and simulate the characteristics of RAW domain images, some training data are generated by preprocessing RGB domain images to generate synthetic RAW domain images, thereby forming RGB domain image and RAW domain image pairs, and performing pixel-level alignment and object detection and annotation. This part enhances some modal features.

[0023] Specifically, in step 1, the above pixel-level aligned and annotated RGB domain image and RAW domain image pairs are uniformly adjusted to an input size of 640×640, and a series of image pre-processing operations (such as normalization, random cropping, horizontal flipping, brightness adjustment, Gaussian blurring, and noise addition) are performed to enhance data diversity and robustness, thereby generating a training dataset. Among them, the pixel-level aligned RGB domain images and RAW domain images are used as the input of the model, and the annotated targets are used as the output of the model.

[0024] Furthermore, a synthetic RAW domain image is generated by preprocessing the RGB image, wherein the preprocessing process specifically includes sequentially performing the following processes: global tone mapping inverse transform, gamma correction inverse transform, color correction inverse transform, inverse process of white balance, Bayer arrangement and de-mosaicing.

[0025] Among them, the global tone mapping inverse transformation: simulates the process of mapping the light intensity of the display device to the intensity of the sensor output signal, and uses a piecewise function to dynamically fit the characteristics of various devices to enhance the robustness to different scenes. This embodiment uses the function shown in the following formula to implement the tone mapping inverse transformation:

[0026] in, x is the input image, y For the output image.

[0027] Gamma correction inverse transform: simulates the nonlinear change process of the sensor converting the optical signal into the electrical signal. It is expressed as:

[0028] in, X is the input image, Y is the image after inverse transformation. γ is one of the parameters of supervised learning. To increase flexibility, the exponential parameter in the power function γ Can be dynamically predicted by the network to better adapt to device differences.

[0029] Color correction inverse transformation: Using the optimized 3×3 color correction matrix, the correction of the RGB primary colors is achieved through matrix inversion, eliminating deviations and simulating the color distribution of RAW images:

[0030] in, Represents the three pixel values ​​of red, green and blue respectively. Respectively represent the corrected red, green, and blue pixel values, It is a fixed parameter. Here, a random value is taken within the range of plus or minus 20% of the current value to increase the diversity of the data set.

[0031] The inverse process of white balance: White balance is to restore white objects in different ambient color temperatures to true white. The inverse process here is to obtain the original image taken by the camera lens:

[0032] in, is the parameter for supervised learning; Respectively represent the red, green and blue components of the input image; Represents the red, green and blue components of the output image respectively.

[0033] Bayer arrangement: Before light enters the camera sensor, it must pass through the Bayer filter to obtain a RAW domain image, and then the Bayer arrangement image is converted into an RGB image through a difference algorithm. The inverse process is to convert the RGB image into a Bayer arrangement RAW domain image. In essence, it is a reverse interpolation process, which restores the original Bayer array by estimating the lost color information (red, green, and blue components) of each pixel. This process usually relies on the color information of neighboring pixels and infers the missing color channels through an interpolation algorithm (such as bilinear interpolation).

[0034] De-mosaicing process: Aims to convert full-color RGB images into single-channel RAW data, simulating the original output of the sensor. Specifically, de-mosaicing generates an image that matches the RAW data format collected by the sensor by rearranging and sampling the pixels of the RGB image. This step ensures that subsequent processing can be performed on data consistent with the actual sensor output. Through the Bayer arrangement, the two green channels are averaged and then spliced ​​according to the RGB channel format to obtain a three-channel RAW domain image.

[0035] Step 2: Build a target detection network. The target detection network includes a backbone network and a detection head. A bimodal feature enhancement module is introduced between the backbone network and the detection head to achieve deep fusion of cross-modal features. The dual enhancement mechanism is used to enhance the features of RGB domain images and RAW domain images in the semantic and spatial dimensions respectively.

[0036] like Figure 2 As shown, the target detection network constructed in this embodiment mainly includes backbone network A, backbone network B, fusion module A, fusion module B, fusion module C, neck network and detection head.

[0037] Among them, RGB domain images and RAW domain images are respectively input into backbone network A and backbone network B to extract 8x, 16x and 32x downsampled RGB features and RAW features, and then input into fusion module A, fusion module B and fusion module C respectively to realize deep fusion and semantic enhancement of cross-modal features, and then enter the neck network (NECK) for further feature fusion processing, and finally output the detection results through the detection head (Dection Head).

[0038] Furthermore, if Figure 3As shown in FIG, the structures of fusion module A, fusion module B and fusion module C are the same, and mainly include: Dynamic Fusion and Channel Enhancement Module (DFCEM) and Dynamic Spatial Interaction and Enhancement Module (DSIEM).

[0039] like Figure 4 As shown in FIG, the dynamic fusion and channel enhancement module mainly includes a fusion enhancement part (i.e., splicing and fusion enhancement layer) and a channel enhancement part (global average pooling, global maximum pooling, etc.).

[0040] In the fusion enhancement stage, RAW features and RGB features are first concatenated in the channel dimension. Then, multi-scale global and local features are extracted through the fusion enhancement layer. Figure 5 As shown in , the fusion enhancement layer is mainly composed of dynamic convolution and inverse bottleneck layer, where dynamic convolution combines deep separation convolution (DW Conv) and point-by-point convolution (PT Conv) to capture the interaction between modalities in the spatial dimension. The inverse bottleneck layer performs nonlinear optimization on features by expanding and compressing channel dimensions, thereby improving the expressiveness of features; optionally, as Figure 5 As shown in Figure 1, the inverse bottleneck layer is mainly composed of a convolutional layer (Conv), a normalization layer (BN), and an activation layer (Act). Finally, the fused features are combined with the original features by direct addition, which retains the modality-specific information and improves the fusion effect.

[0041] In the channel enhancement stage, global average pooling (GAP) and global maximum pooling (GMP) are first used to extract the global statistics of the features and generate dynamic channel weights. Subsequently, the global statistics are nonlinearly transformed using a fully connected layer to generate independent dynamic channel weights for RAW features and RGB features. Finally, the generated dynamic channel weights are element-wise multiplied with the fused and enhanced features for adaptive optimization to enhance the features of each modality. In this stage, the enhanced features are combined with the original features by element-wise multiplication to ensure that the features of each modality are optimized and enhanced without losing the original key details. In this way, we can effectively retain the key information of the original features while improving the expressiveness of cross-modal features.

[0042] This embodiment introduces a dynamic fusion and channel enhancement module, which effectively enhances the balance between global semantics and local details and is suitable for feature fusion and optimization in multimodal tasks. Through dynamic modal fusion, precise channel enhancement and nonlinear optimization, the dynamic fusion and channel enhancement module provides significant performance improvement for multimodal feature representation and provides strong support for the design and implementation of multimodal tasks.

[0043] like Figure 6 As shown in the figure, the dynamic spatial interaction and enhancement module mainly includes convolutional pyramid, point-by-point convolution, channel attention mechanism, etc.

[0044] The dynamic spatial interaction and enhancement module first extracts multi-scale spatial features from the input RGB features and RAW features, and uses the convolution pyramid to extract multi-scale spatial feature information through depth-separated convolution kernels of different scales; then, the multi-scale features are compressed in the channel dimension through point-by-point convolution (PW Conv) to generate single-channel RGB features and RAW feature representations. This stage not only retains modality-specific information, but also provides rich spatial features for subsequent modality interaction and fusion.

[0045] In the modal interaction and fusion stage, RGB features and RAW features are concatenated (C) through the channel dimension and further compressed and fused into a single channel representation using point-by-point convolution. Then, the fused features are dynamically enhanced through the channel attention mechanism, which combines global average pooling (GPA) and convolutional layers (mainly including convolution + activation function Sigmoid) to generate dynamic channel weights to highlight the interaction between salient feature areas and modalities. After that, the fused features are generated through independent dynamic convolution modules to enhance features suitable for RGB and RAW. Finally, the optimized RGB features and RAW features are output to provide more expressive feature representations for multimodal tasks.

[0046] This embodiment introduces a dynamic spatial interaction and enhancement module and adopts a multi-scale convolutional pyramid design to capture the multi-scale spatial information of RGB features and RAW features, taking into account both global semantics and local details. It also integrates the attention mechanism to dynamically adjust channel weights and enhance the modal interaction capability, making it suitable for a variety of multimodal task scenarios, such as high dynamic range imaging, image denoising, and feature fusion.

[0047] Furthermore, the neck network is usually used to further fuse features and generate multi-scale feature maps to support detection tasks. Common neck networks include: FPN (Feature Pyramid Network), PANet (Path Aggregation Network), BiFPN (Bi-directional Feature Pyramid Network), etc. The detection head is mainly responsible for generating the final detection result based on the feature map output by the neck network. Common detection head structures include: YOLO series (YOLOv5 to YOLOv11).

[0048] Step 3: Use the training data set to train the target detection network to obtain a medical image detection model.

[0049] In step 3, the target detection network is iteratively trained using the training dataset. The number of iterations is 700, the batch size is 64, and the initial learning rate is 1×10 −2 , the final learning rate is 1×10 −4 , the optimizer is AdamW combined with cosine annealing learning rate schedule.

[0050] During the training process, the loss function is designed by combining classification loss, bounding box regression loss and alignment consistency loss, and the requirements of each task are dynamically balanced through weight adjustment. Classification loss optimizes the category prediction of the detection target, and bounding box regression loss determines the positioning accuracy.

[0051] Among them, the classification loss uses cross-entropy loss or focal loss to deal with the category imbalance problem to ensure that the model can accurately predict the target category, expressed as:

[0052] in, is the classification loss; and are the true category distribution and the predicted probability distribution respectively; N is the total number of samples, that is, the number of samples contained in the data set; K is the total number of categories, that is, the number of target categories in the classification task.

[0053] The bounding box regression loss uses GIoU (Generalized IoU) loss to measure the overlap between the predicted box and the true box and optimize the positioning accuracy of the bounding box, which is expressed as:

[0054] in, is the bounding box regression loss, A and B They are the predicted box and the real box respectively, and C refers to the minimum enclosing rectangle, which is the minimum rectangle that can completely surround the two given rectangular boxes A and B.

[0055] Consistency loss is used to measure the consistency of outputs between different modalities or stages. It can be calculated by similarity metrics between images, such as L2 loss based on feature maps or loss based on pixel differences.

[0056] in, is the consistency loss, and The model calculates the difference between the outputs of the corresponding feature maps obtained by downsampling the two backbone networks by 8 times, 16 times, and 32 times in different modalities (such as RGB and RAW images). By optimizing the consistency loss, the model can reduce the prediction differences across modalities or stages.

[0057] The classification loss, bounding box regression loss, and consistency loss are combined to balance the importance of the two with dynamic weights:

[0058] in, is the total loss, α , β , λ They are all hyperparameters used to adjust the relative weights of classification, positioning, and consistency tasks.

[0059] Step 4: Analyze the medical images collected in real time using the medical image detection model to quickly and accurately detect and classify the lesion areas in the medical images.

[0060] Specifically, the medical image detection model obtained through the above training is used to screen cancer images. The RAW domain grayscale image and RGB enhanced image of the CT scan are input into the above medical image detection model, which can quickly and accurately identify and mark the cancerous area.

[0061] The medical image detection model obtained through the above training can also be used to assist real-time surgical navigation. The real-time acquired ultrasound RAW domain images and preoperative RGB domain images are input into the above medical image detection model to generate a navigation path in real time and highlight key anatomical structures.

[0062] The medical image analysis method proposed in this embodiment is based on a multimodal input design, which enables this embodiment to fully utilize the complementary information of RAW domain and RGB domain images, overcome the limitations of single-modal analysis methods, and at the same time, by introducing a dual feature enhancement module, significantly optimizes the extraction and fusion capabilities of cross-modal features, effectively highlights important feature information between channels, and enhances the significance of the target area in the spatial dimension, thereby greatly improving detection accuracy and efficiency.

[0063] Based on the same technical concept as above, this embodiment also proposes a medical image analysis system based on multimodal image fusion, such as Figure 7 As shown, the medical image analysis system proposed in this embodiment includes: An image processing module is used to generate a training data set: the training data set consists of pixel-level aligned and annotated medical image RGB domain image and RAW domain image pairs, wherein the RAW domain images include directly acquired real RAW domain images and synthetic RAW domain images generated by preprocessing RGB images.

[0064] Model building module, which is used to build a target detection network, which includes a backbone network and a detection head, and introduces a dual-modal feature enhancement module between the backbone network and the detection head to achieve deep fusion of cross-modal features, and uses a dual enhancement mechanism to enhance the features of RGB domain images and RAW domain images in semantic and spatial dimensions respectively. The constructed target detection network structure is as described in step 1 above, and will not be repeated here.

[0065] The model training module uses the training data set to train the target detection network to obtain a medical image detection model.

[0066] And, an analysis module, which uses a medical image detection model to analyze medical images collected in real time to quickly and accurately detect and classify lesion areas in medical images.

[0067] Furthermore, the image processing module also includes: a preprocessing unit, which is used to preprocess the RGB domain image to generate a synthetic RAW domain image. The specific preprocessing process is as described in the above step 1 and will not be repeated here; a pixel-level alignment unit, which is used to achieve pixel-level alignment of the RGB domain image and the RAW domain image pair; a labeling unit, which is used to label the RGB domain image and the RAW domain image for detection targets; and an image adjustment and expansion unit, which adjusts the pixel-level aligned and labeled RGB domain image and RAW domain image pair to the same input size, and performs a series of image preprocessing operations (such as normalization, random cropping, horizontal flipping, brightness adjustment, Gaussian blurring and noise addition) to enhance data diversity and robustness, thereby generating a training data set.

[0068] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0069] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0070] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0071] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0072] The specific implementation methods described above further illustrate the purpose, technical solutions and beneficial effects of the present application in detail. It should be understood that the above description is only the specific implementation method of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A medical image analysis method based on multimodal image fusion, characterized in that: The medical image analysis method comprises: Generate a training data set; the training data set consists of pixel-level aligned and annotated medical image RGB domain image and RAW domain image pairs, wherein the RAW domain images include directly acquired real RAW domain images and synthetic RAW domain images generated by preprocessing the RGB domain images; Constructing an object detection network; the object detection network includes a backbone network and a detection head, and introducing a dual-modal feature enhancement module between the backbone network and the detection head to achieve deep fusion of cross-modal features, and using a dual enhancement mechanism to enhance the features of RGB domain images and RAW domain images in semantic and spatial dimensions respectively; The target detection network is trained using the training data set to obtain a medical image detection model; wherein the RGB domain image and the RAW domain image are used as inputs of the target detection network, and the annotation information is used as outputs of the target detection network; The medical image detection model is used to analyze the medical images collected in real time, so as to quickly and accurately detect and classify the lesion areas in the medical images.

2. The medical image analysis method based on multimodal image fusion according to claim 1, characterized in that: The constructed target detection network includes two backbone networks, three fusion modules, a neck network and a detection head; RGB domain images and RAW domain images are respectively input into two backbone networks to extract 8x, 16x and 32x downsampled RGB features and RAW features, which are then input into three fusion modules to achieve deep fusion semantic enhancement of cross-modal features. They then enter the neck network for further feature fusion processing, and finally the detection results are output through the detection head.

3. The medical image analysis method based on multimodal image fusion according to claim 2, characterized in that: The fusion module includes: a dynamic fusion and channel enhancement module, and a dynamic space interaction and enhancement module; Among them, the dynamic fusion and channel enhancement module is used to realize the fusion enhancement and channel enhancement of RGB features and RAW features; in the fusion enhancement stage, after splicing the RGB features and RAW features in the channel dimension, multi-scale global and local features are extracted through dynamic convolution and inverse bottleneck layer; the dynamic convolution combines deep separation convolution and point-by-point convolution to capture the spatial interaction relationship between modalities; the inverse bottleneck layer realizes nonlinear optimization of features by expanding and compressing channel dimensions; in the channel enhancement stage, global average pooling and global maximum pooling are used to provide global statistical information of features and generate dynamic channel weights; the global statistical information is nonlinearly transformed through the fully connected layer to generate independent dynamic weights suitable for RGB features and RAW features; the dynamic weights are multiplied element by element with the fusion enhanced features, each modality is adaptively optimized, and the original feature information is retained through the residual path, and the enhanced modal features are output to the dynamic spatial interaction and enhancement module; The dynamic spatial interaction and enhancement module extracts multi-scale spatial features from the input RGB features and RAW features respectively; then, the multi-scale spatial features are compressed in the channel dimension by point-by-point convolution to generate single-channel RGB features and RAW feature representations; then, the RGB features and RAW features are concatenated in the channel dimension, and further compressed and fused into a single channel representation by point-by-point convolution; the fused features are dynamically enhanced by a channel attention mechanism, and then enhanced features suitable for RGB domain images and RAW domains are generated by an independent dynamic convolution module; finally, the optimized RGB features and RAW features are output.

4. The medical image analysis method based on multimodal image fusion according to claim 3, characterized in that: The dynamic spatial interaction and enhancement module uses a convolutional pyramid to extract multi-scale spatial features through depth-separated convolution kernels of different scales; The channel attention mechanism combines global average pooling and convolutional layers to generate dynamic channel weights to highlight the interactive relationship between salient feature regions and modalities.

5. A medical image analysis method based on multimodal image fusion according to any one of claims 1 to 4, characterized in that: The training data set generation process includes: Perform pixel-level alignment and target detection and annotation on real RGB domain images and RAW domain image pairs; A synthetic RAW domain image is generated by preprocessing the real RGB domain image, and pixel-level alignment and object detection and annotation are performed on the real RGB domain image and the synthetic RAW domain image generated; The RGB domain image and RAW domain image pairs that are aligned and annotated at the pixel level are uniformly adjusted to the same input size, and a series of image pre-processing operations are performed to enhance data diversity and robustness, thereby generating the training data set.

6. The medical image analysis method based on multimodal image fusion according to claim 5, characterized in that: The method of generating a synthetic RAW domain image by preprocessing a real RGB domain image specifically includes: The RGB domain image is subjected to inverse global tone mapping, inverse gamma correction, inverse color correction, inverse white balance, Bayer arrangement, and demosaicing to generate a synthetic RAW domain image that matches the RAW data format collected by the sensor.

7. A medical image analysis method based on multimodal image fusion according to any one of claims 1 to 4, characterized in that: During the training process of the target detection network, the loss function used is: the weighted sum of classification loss, bounding box regression loss and alignment consistency loss; and the weights of the classification loss, bounding box regression loss and alignment consistency loss are dynamically adjustable.

8. The medical image analysis method based on multimodal image fusion according to claim 7, characterized in that: The classification loss uses cross entropy loss or focal loss to deal with the category imbalance problem, ensuring that the model can accurately predict the target category; the bounding box regression loss uses GIoU loss to measure the overlap between the predicted box and the real box, and optimizes the positioning accuracy of the bounding box; the consistency loss uses L2 loss based on feature maps or loss based on pixel differences.

9. A medical image analysis system based on multimodal image fusion, characterized in that: The medical image analysis system comprises: An image processing module, the image processing module is used to generate a training data set; the training data set is composed of pixel-level aligned and annotated medical image RGB domain image and RAW domain image pairs, wherein the RAW domain images include directly acquired real RAW domain images and synthetic RAW domain images generated by preprocessing RGB images; A model building module, wherein the model building module is used to build an object detection network, wherein the object detection network includes a backbone network and a detection head, and a dual-modal feature enhancement module is introduced between the backbone network and the detection head to achieve deep fusion of cross-modal features, and a dual enhancement mechanism is used to enhance the features of RGB domain images and RAW domain images in semantic and spatial dimensions respectively; A model training module, wherein the model training module trains the target detection network using the training data set to obtain a medical image detection model; wherein the RGB domain image and the RAW domain image are used as inputs of the target detection network, and the annotation information is used as outputs of the target detection network; And, an analysis module, which uses the medical image detection module to analyze the medical images collected in real time, so as to quickly and accurately detect and classify the lesion areas in the medical images.

10. The medical image analysis system based on multimodal image fusion according to claim 9, characterized in that: The image processing module comprises: A preprocessing unit, the preprocessing unit is used to preprocess the RGB domain image to generate a synthetic RAW domain image; A pixel-level alignment unit, wherein the pixel-level alignment unit is used to achieve pixel-level alignment of an RGB domain image and a RAW domain image pair; A labeling unit, wherein the labeling unit is used to label the detection target on the RGB domain image and the RAW domain image; And, an image adjustment and expansion unit, which adjusts the pixel-level aligned and labeled RGB domain image and RAW domain image pairs to the same input size, and performs a series of image pre-processing operations to enhance data diversity and robustness, thereby generating the training data set.

Images