Medical image processing method and device, medium and electronic equipment

Abstract

The invention provides a medical image processing method and device, a medium and electronic equipment, and the method comprises the steps: inputting a to-be-processed medical image into a feature extraction network, and obtaining a multi-scale feature map; wherein the feature extraction network comprises a plurality of feature extraction layers, and the feature extraction layers of different layers are used for extracting feature maps of different scales; inputting the multi-scale feature maps into a feature fusion network, performing internal fusion on the feature maps in each scale level by using intra-scale self-attention to obtain intra-scale fusion feature maps of each scale level, and performing cross-scale fusion on the intra-scale fusion feature maps of different scale levels by using cross-scale cross attention to obtain an intra-scale fusion feature map of each scale level; a multi-scale fusion feature map is obtained; and based on the multi-scale fusion feature map, using a classifier to obtain a processing result of the medical image. According to the invention, the detection sensitivity and the diagnosis accuracy of tiny focuses are improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of medical image intelligent analysis, in particular, to a medical image processing method and device, medium and electronic equipment. BACKGROUND

[0002] With the rapid development of medical imaging technology, intelligent diagnosis based on medical image analysis models (such as processing CT, MRI, etc.) has become an important support for clinical diagnosis and treatment.

[0003] In related technologies, medical image analysis models often rely on single-scale features for diagnosis; even if multi-scale features are extracted, only the extracted features are simply spliced or linearly weighted and fused. In this way, the medical image analysis model is difficult to effectively capture the complex dependency relationship between cross-scale features, and cannot fully tap their inherent complementary and collaborative potential. Since the pathological structure of the human body (from microscopic cell texture to macroscopic organ morphology) is essentially multi-scale, the above model is difficult to construct a comprehensive and robust feature representation, resulting in a serious limitation of the detection sensitivity of the diagnosis model for small lesions (≤5mm).

[0004] Therefore, it is urgent to develop a new model that can realize intelligent cross-scale fusion from microscopic texture to macroscopic organ features to improve the detection sensitivity and accuracy of small lesions. SUMMARY

[0005] The purpose of the present disclosure is to provide a medical image processing method, device, medium and electronic equipment, which can solve the technical problem of insufficient cross-scale feature fusion in medical image analysis, and improve the detection sensitivity and diagnostic accuracy of small lesions. The specific scheme is as follows:

[0006] According to the specific embodiment of the present disclosure, the present disclosure provides a medical image processing method, which comprises:

[0007] inputting a medical image to be processed into a feature extraction network to obtain a multi-scale feature map; wherein the feature extraction network comprises multiple layers of feature extraction layers, and different layers of feature extraction layers are used to extract different scale feature maps;

[0008] inputting the multi-scale feature map into a feature fusion network, using intra-scale self-attention to respectively perform internal fusion on the feature maps within each scale level to obtain intra-scale fusion feature maps of each scale level, and using cross-scale cross-attention to perform cross-scale fusion on the intra-scale fusion feature maps of different scale levels to obtain a multi-scale fusion feature map;

[0009] based on the multi-scale fusion feature map, using a classifier to obtain a processing result of the medical image.

[0010] In some embodiments, the inputting the medical image to be processed into the feature extraction network comprises:

[0011] The feature map of the i-th layer feature extraction layer of the feature extraction network is input into the (i+1)-th layer feature extraction layer to obtain an output feature map of the i-th layer feature extraction layer; wherein i is a positive integer from 1 to N-1, and N is the number of layers of the feature extraction network.

[0012] The input of the first layer feature extraction layer of the feature extraction network is the medical image.

[0013] In some embodiments, the inputting the medical image to be processed into the feature extraction network comprises:

[0014] The input of each feature extraction layer is preprocessed based on the object extraction scale corresponding to the feature extraction layer to obtain a plurality of non-overlapping feature map blocks; wherein the first layer feature extraction layer obtains a feature map block by segmenting and representing the medical image; and the (i+1)-th layer feature extraction layer obtains a feature map block by down-sampling and fusing the feature map output by the i-th layer.

[0015] The feature extraction network alternately uses local window intra-self-attention calculation and cross-window shift self-attention calculation to extract features from the feature map blocks to obtain an output feature map of the feature extraction layer; wherein the spatial dimension of the feature map output by each layer decreases layer by layer as the level deepens, and the channel dimension of the feature map output by each layer increases layer by layer as the level deepens.

[0016] In some embodiments, the inputting the medical image to be processed into the feature extraction network comprises:

[0017] The feature map block and the feature obtained by the local window intra-self-attention calculation are added to form a first residual feature; the first residual feature and the feature obtained by the cross-window shift self-attention calculation are added to form a second residual feature; and the second residual feature after the nonlinear transformation is connected in residual with the second residual feature before the nonlinear transformation to obtain an output feature map of the feature extraction layer.

[0018] The output feature map of the i-th layer feature extraction layer and the output feature map of the (i+1)-th layer feature extraction layer are added to obtain an enhanced output feature map of the (i+1)-th layer feature extraction layer.

[0019] Wherein, i+1 is a positive integer not greater than N.

[0020] In some embodiments, the inputting the medical image to be processed into the feature extraction network comprises:

[0021] adding the feature maps of each scale level output by the feature extraction network and the scale-intra fusion feature maps corresponding to the feature maps, to obtain enhanced scale-intra fusion feature maps;

[0022] adding the enhanced scale-intra fusion feature maps of each scale level after dimension adjustment to the cross-scale fusion feature maps obtained through cross-scale cross-attention processing; wherein the dimension adjustment includes down-sampling of higher scale level feature maps and up-sampling of lower scale level feature maps, so that the dimensions match the dimensions of the cross-scale fusion feature maps;

[0023] adding the multi-scale feature maps output by the feature extraction network after dimension adjustment to the multi-scale fusion feature maps processed through scale-intra self-attention and cross-scale cross-attention, to obtain the final multi-scale fusion feature maps.

[0024] In some embodiments, the method further comprises a training step, which comprises:

[0025] obtaining a training data set, wherein the training data set comprises a plurality of medical image samples and their corresponding classification labels;

[0026] using model parameters pre-trained on a large-scale visual data set as the initial parameters of the feature extraction network;

[0027] iteratively training the feature extraction network, the feature fusion network and the classifier using the medical image samples, wherein the iterative training comprises:

[0028] inputting the medical image samples into the feature extraction network, the feature fusion network and the classifier, and the classifier outputs the classification results of the medical image samples; calculating the cross-entropy loss between the classification results and the classification labels;

[0029] based on the cross-entropy loss, using an Adam optimizer to update the parameters of the feature extraction network, the feature fusion network and the classifier, wherein a cosine decay learning rate algorithm is used to dynamically adjust the learning rate of the Adam optimizer;

[0030] repeating the iterative training until the cross-entropy loss converges or reaches a preset training round.

[0031] In some embodiments, the training step further comprises:

[0032] dividing the training data set into a training subset and a validation subset;

[0033] After each preset verification interval, the feature extraction network, the feature fusion network and the classifier currently trained are verified by using the verification subset, to obtain a verification performance index;

[0034] When the verification performance index does not exceed the historical best value in the continuous preset number of verifications, the iterative training is terminated in advance.

[0035] According to the specific embodiments of the present disclosure, the present disclosure further provides a processing device for medical images, characterized in that the device comprises:

[0036] a feature extraction unit configured to input a medical image to be processed into a feature extraction network to obtain a multi-scale feature map; wherein the feature extraction network comprises a plurality of feature extraction layers, and different layers of the feature extraction layers are configured to extract different scale feature maps;

[0037] a feature fusion unit configured to input the multi-scale feature map into a feature fusion network, to perform internal fusion on the feature maps in each scale level respectively using intra-scale self-attention, to obtain intra-scale fused feature maps of each scale level, and to perform cross-scale fusion on the intra-scale fused feature maps of different scale levels using cross-scale cross-attention, to obtain a multi-scale fused feature map;

[0038] a result output unit configured to obtain a processing result of the medical image based on the multi-scale fused feature map using a classifier.

[0039] According to the specific embodiments of the present disclosure, the present disclosure further provides a computer readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method according to any one of the above.

[0040] According to the specific embodiments of the present disclosure, the present disclosure further provides an electronic device, comprising: one or more processors; a storage device configured to store one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method according to any one of the above.

[0041] Compared with the prior art, the above-mentioned scheme of the present disclosure has at least the following beneficial effects:

[0042] First, a feature extraction network is used to design a multi-layer feature extraction layer. Each feature extraction layer outputs feature maps of different scales. This process comprehensively captures the local microscopic details and global macroscopic structural information in medical images. Then, a feature fusion network is used to perform a two-stage fusion of feature maps of different scales. First, the intra-scale self-attention mechanism is applied to internally enhance the feature maps of each scale to strengthen the expression of key features at the same scale. Then, cross-scale cross-attention is used to achieve effective information interaction and complementary fusion between features at different scale levels. During the intra-scale fusion process, the intra-scale self-attention calculation is used to strengthen the expression of key features within the same scale. During the cross-scale fusion process, the cross-scale cross-attention calculation is used to effectively capture the spatial contextual relationship between feature maps at different scale levels, thereby enhancing the information interaction of feature maps at different scale levels. The resulting multi-scale fusion feature map retains the local details and global semantic information of the medical image, overcoming the limitation of traditional methods that it is difficult to pay attention to the details of tiny lesions and the overall structural semantics at the same time. The final multi-scale fusion feature map has both high-resolution details and global semantic information, which improves the detection sensitivity of tiny lesions (<5mm) in medical images, reduces the misdiagnosis rate under different imaging conditions, and provides the classifier with high-quality feature representation with both detail fidelity and semantic integrity, thereby greatly improving the overall accuracy of medical image analysis, and further enhancing the application value of medical image processing models in medical diagnosis clinics. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and those skilled in the art can derive other drawings based on these drawings without inventive effort. In the drawings:

[0044] Figure 1 A schematic diagram of the process of processing medical images provided in this application;

[0045] Figure 2 A flowchart of a feature extraction network for the medical image processing method provided in this application;

[0046] Figure 3 A flowchart of the medical image processing method provided in this application;

[0047] Figure 4 A schematic diagram of the structure of the medical image processing device provided in this application;

[0048] Figure 5 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0049] In order to make the purposes, technical solutions and advantages of the present disclosure clearer, the present disclosure will be further described in detail below with reference to the drawings. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present disclosure.

[0050] The terms used in the embodiments of the present disclosure are only for the purpose of describing particular embodiments and are not intended to limit the present disclosure. The singular forms "a", "an" and "the" used in the embodiments of the present disclosure and the appended claims are intended to include plural forms, unless the context clearly indicates otherwise. "Plural" generally includes at least two.

[0051] It should be understood that the term "and / or" used herein only describes an association relationship of associated objects, which means that there can be three relationships, for example, A and / or B can represent: A exists alone, A and B exist together, and B exists alone. In addition, the character " / " herein generally represents an "or" relationship between the front and rear associated objects.

[0052] It should be understood that although the terms first, second, third, etc. can be used in the embodiments of the present disclosure to describe, these descriptions should not be limited to these terms. These terms are only used to distinguish the description. For example, without departing from the scope of the embodiments of the present disclosure, the first can also be referred to as the second, and similarly, the second can also be referred to as the first.

[0053] It should also be noted that the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the goods or devices including a series of elements not only include those elements, but also include other elements not explicitly listed, or include elements inherent to such goods or devices. Without more limitations, the element defined by the sentence "including a" does not exclude the presence of other identical elements in the goods or devices including the element.

[0054] First, the terms involved in the present application are explained:

[0055] 1.Transformer model: A deep learning model architecture that has the core advantage of being able to process all elements in a sequence in parallel and effectively capture long-range dependencies between elements. In this application, the feature fusion network utilizes the encoder of Transformer to capture long-range dependencies between different scale feature maps through attention mechanism, learn and integrate their complementary information, and generate a feature-enhanced multi-scale fusion feature map. This fusion feature map contains rich information from microscopic tissue texture to macroscopic pathological semantics, which is used for final medical image classification.

[0056] 2.Swin Transformer model: A hierarchical visual Transformer designed specifically for computer vision tasks.

[0057] 3.Local window self-attention (W-MSA): Restricting self-attention computation within non-overlapping local windows in the image significantly reduces computational complexity. Cross-window shift self-attention (SW-MSA): Circularly shifting windows between consecutive Transformer blocks breaks the isolation between windows, enabling cross-window information exchange and capturing feature dependencies over a wider range. In this application, by alternately using W-MSA and SW-MSA modules, progressive feature capture from local details to global semantics is achieved. This design allows the model to focus on subtle features in medical images (such as the edge texture of a small lesion) and understand the context of these features in a larger range (such as the organ structure background), achieving a good balance between computational efficiency and model receptive field.

[0058] 4.Self-attention mechanism: It allows the model to dynamically evaluate the importance (i.e., "attention weight") of all other elements in the sequence (or feature set) to the current element when processing it, and aggregate the information of all elements according to these weights to get the new representation of the current element. This mechanism enables the model to focus on global information and capture long-range dependencies. In this application, in the scale-intra fusion stage, it helps the model understand the relationship between different regions within the same scale feature map.

[0059] 5. Cross-Attention: Similar to self-attention, but it deals with two or more different input sequences (or feature sets). One sequence (usually called Query) will focus on the information in another sequence (Key-Value). This allows the model to establish connections and integrate information between different modalities (scales) or different sources. In this application, the cross-attention mechanism enables the feature maps extracted from one scale to effectively "query" and integrate information from other scale feature maps, thus achieving effective complementation and enhancement between different levels of features (e.g., detailed texture and macro structure).

[0060] 6. Softmax activation function: Usually used in the output layer of multi-classification problems, it converts a vector containing arbitrary real numbers into a probability distribution vector, where each element value is between 0 and 1, and the sum of all elements is 1.

[0061] 7. Cross-Entropy Loss Function: Used to measure the difference between the model's predicted probability distribution and the true class label (usually one-hot encoded). It is a commonly used loss function for training classification models.

[0062] 8. Adam Optimizer: A commonly used gradient descent optimization algorithm that combines the advantages of AdaGrad and RMSProp algorithms, allowing for adaptive learning rates for different parameters.

[0063] 9. Cosine decay learning rate: A learning rate scheduling strategy that causes the learning rate to smoothly decay according to the shape of the cosine function during training, gradually reducing from the initial maximum learning rate to the minimum learning rate.

[0064] The optional embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0065] Figure 1 The flowchart of the medical image processing method provided in the present application is shown in FIG. 1, which includes the following steps: Figure 1

[0066] S102, input the medical image to be processed into a feature extraction network to obtain a multi-scale feature map; wherein the feature extraction network includes multiple layers of feature extraction layers, and different layers of feature extraction layers are used to extract different scale feature maps.

[0067] By way of example, the execution subject of the present embodiment can be an electronic device, or a terminal device, or a server, or a device or apparatus capable of executing the present embodiment, and the present embodiment is not limited in this regard.

[0068] ​The medical images can be two-dimensional images or three-dimensional images, including but not limited to computed tomography (CT) images, magnetic resonance imaging (MRI) sequences, X-ray films, ultrasound images, positron emission tomography (PET) images, pathological section microscopic images, and the like.

[0069] In the medical image analysis field, "multi-scale" in this embodiment refers to the ability to capture image features from different spatial resolutions and receptive field sizes. Each scale level corresponds to a different granularity of medical image feature representation, so that complete feature representation from micro to macro in the medical image can be captured at the same time. The high-resolution features extracted by the shallow feature extraction layer (the front end of the network) retain the fine texture information of the microstructure (such as the edge features of early lesions below 5 mm), and the semantic features extracted by the deep feature extraction layer express the overall anatomical structure and pathological distribution pattern. This multi-scale feature extraction scheme overcomes the limitations of single-scale feature extraction. For example, the traditional single-scale model has a high missed detection rate for lesions ≤5 mm, and by retaining high-resolution shallow features, the system improves the detection sensitivity for micro lesions (≤5 mm) that are clinically critical.

[0070] The following is an example of feature extraction at several scale levels of medical images:

[0071] Micro-scale extraction, definition: capture the finest local features in the medical image, usually corresponding to the structural details at the histological level. Resolution: highest, usually 1 / 2-1 / 4 of the original image, receptive field: smallest (such as a 7x7 or 9x9 pixel window), representation content: cell morphology, microvascular structure, tissue boundary texture, microcalcification points, etc.

[0072] Mesoscale extraction, definition: intermediate-level features between microscale and macroscale, capturing tissue structure patterns within local regions. Resolution: medium, usually 1 / 8-1 / 16 of the original image, receptive field: medium (covering local anatomical regions), representation content: organ substructure, medium-sized lesions (3-10 mm), local vascular network patterns, tissue density changes, etc.

[0073] Macro-scale extraction, definition: capture global features of large-scale anatomical structures and overall pathological distribution patterns. Resolution: lower, usually 1 / 32 or lower of the original image, receptive field: large (can cover the entire organ or anatomical region), representation content: organ morphology, large-scale lesion distribution, relative position relationship between anatomical structures, overall tissue density distribution, etc.

[0074] Global semantic scale extraction: In addition to the scale-specific features, the correlation between different scales is also concerned in the multi-scale analysis of medical images. The correlation includes: scale-specific feature complementarity, different scales capture complementary information, such as the accurate boundary of a small lesion (micro) and its location in the organ (macro); scale-specific consistency: the spatial structure and semantic consistency of features in the same scale; scale conversion dependence: the correlation between pathological features in different scales, such as how micro-texture changes affect macro-morphological features.

[0075] In some embodiments, in step S102, the feature extraction process of the feature extraction network on the medical image can include:

[0076] The feature map of the i-th layer feature extraction layer of the feature extraction network is input into the i+1-th layer feature extraction layer to obtain the output feature map of the i-th layer feature extraction layer; wherein i is a positive integer from 1 to N-1, and N is the number of layers of the feature extraction network.

[0077] The input of the first layer feature extraction layer of the feature extraction network is the medical image.

[0078] It can be understood that the multiple feature extraction layers included in the feature extraction network form a processing pipeline from the shallow layer to the deep layer, that is, the output of the previous layer feature extraction layer is the input of the next layer feature extraction layer. In this way, the cascaded structure of the feature extraction layers makes the feature extraction process highly parallel, each layer only processes the output of the previous layer, avoiding repeated calculation and improving the calculation efficiency of feature extraction.

[0079] In some embodiments, in step S102, the feature extraction process of each feature extraction layer can include:

[0080] Based on the object extraction scale corresponding to each feature extraction layer, the input of the feature extraction layer is preprocessed to obtain multiple non-overlapping feature map blocks; wherein the first layer feature extraction layer obtains the feature map block by segmenting and representing the medical image; the i+1-th layer feature extraction layer obtains the feature map block by downsampling and fusing the feature map output by the i-th layer.

[0081] The feature extraction layer alternately uses local window self-attention calculation and cross-window shift self-attention calculation to extract features from the feature map block to obtain the output feature map of the feature extraction layer; wherein the feature map output by each layer from the first layer to the N-th layer feature extraction layer decreases layer by layer in the spatial dimension and increases layer by layer in the channel dimension as the level deepens.

[0082] In this embodiment, the first layer feature extraction layer processes the medical image into multiple overlapping feature map blocks, i.e., divides the medical image into multiple non-overlapping small regions and performs representation transformation on each small region (transforms into a representation form that the feature extraction layer can process), and each feature map block is used to extract local details (such as the edges of a small lesion).

[0083] The subsequent feature extraction layers (the second layer to the Nth layer): reduce the size of the feature map output by the previous layer to generate a new feature map block. In this way, the high-level feature map block covers a larger image area and captures more macroscopic structures (such as the overall shape of an organ).

[0084] In the feature extraction process of each layer of the feature extraction layer on the feature map block, two self-attention mechanisms are alternately used for calculation:

[0085] Local window self-attention: calculate the correlation between pixels within each small block to extract local texture features (such as gray scale changes of a lesion).

[0086] Cross-window shift self-attention: move the window position to calculate the correlation between adjacent blocks, expand the feature extraction range, and capture the overall features across the region (such as the relationship between the lesion and the surrounding tissue).

[0087] From the first layer to the Nth layer, the feature map becomes smaller and smaller. For example, the first layer is 1 / 4 the size of the original image, and each subsequent layer continues to shrink, so that the high-level feature map can cover the global information of the entire image. At the same time, the channel dimension: the number of channels gradually increases at each layer. The more the number of channels, the more types of features are extracted (such as from simple edge features to multiple features including texture, shape, etc.).

[0088] In this embodiment, the local window self-attention calculation and the cross-window shift self-attention calculation are alternately used to process the feature map block, achieving an optimal balance between computational efficiency and receptive field range: the local window self-attention focuses on the detail enhancement of the features within the window, capturing the local features of a small lesion; and the cross-window shift self-attention breaks the window boundary restriction, establishes a wider spatial dependency relationship, and understands the context of the lesion. This alternating strategy not only maintains the controllability of the computational complexity, but also overcomes the problem of limited receptive field caused by the fixed window, and can simultaneously focus on local subtle features (such as lesion edge texture) and spatial context relationships in a larger range (such as surrounding tissue structure), thereby improving the detection sensitivity and classification accuracy of lesions of different sizes.

[0089] Exemplarily, Figure 2 The flowchart of the feature extraction network of the medical image processing method provided in the present application is as follows: Figure 2As shown, the feature extraction network is constructed based on a SwinTransformer model, and the feature extraction network includes four layers of feature extraction layers, i.e., stages 1 to 4, wherein the first layer of feature extraction layer at least includes a Patch Embeddings module and a plurality of Block modules, and the second to fourth layers of feature extraction layer at least include a Patch Merging module and a plurality of Block modules. The Patch Embeddings module of the first layer is located at the front end of the model, which is a preprocessing step of the medical image, divides the medical image into fixed-size patches, and converts each patch into a low-dimensional embedding vector through linear projection, while adding position encoding to preserve spatial information, outputs a patch sequence with uniform resolution as the initial input of the feature extraction network. The Patch Merging performs downsampling and feature fusion on the existing patch features (feature image blocks), reduces the resolution and increases the channel number by merging adjacent patches, and outputs hierarchical feature maps with different resolutions.

[0090] Figure 2 Among them, the multi-scale feature extraction process includes: input medical image→Patch Embeddings (1 time, only at the beginning of model calculation) Stage 1 (contains multiple SwinTransformer Blocks)→Patch Merging (first time to reduce resolution)→Stage 2 (contains multiple SwinTransformer Blocks)→Patch Merging (second time to reduce resolution)→Stage 3 (contains multiple SwinTransformer Blocks)→Patch Merging (third time to reduce resolution)→Stage 4 (contains multiple SwinTransformer Blocks)→the feature map output of the final (feature extraction network). The specific feature extraction is introduced as follows:

[0091] Stage 1: High-resolution feature extraction (original size 1 / 4), using a 7x7 local window to capture fine-grained features, and the output feature map maintains a high spatial resolution (H / 4xW / 4). This layer focuses on: microscopic texture features: edge response, gray gradient, tissue microstructure, local contrast features: lesion-normal tissue density difference, spatial frequency features: high-frequency detail information preservation. Especially suitable for <5mm micro-lesion detection, and the multi-head self-attention mechanism (8 heads) is used to establish pixel-level feature association.

[0092] Stage 2: Mid-level feature extraction (original size 1 / 8), 2x2 down-sampling is achieved through patch merging, and the channel dimension is expanded to 2 times. The features at this level have: organ topological structure representation: vascular branches, tissue boundaries, regional morphological features: lesion geometric properties (circularity, lobulation, etc.), cross-window feature interaction: adjacent region association is established through shifted window, and the window attention mechanism can cover an area of about 15x15 pixels at this stage, realizing sub-organ level feature modeling.

[0093] Stage 3: Macro feature extraction (original size 1 / 16), the receptive field is expanded to the entire region of interest (ROI), and the feature representation includes: global distribution pattern of lesions: spatial relationship of multiple lesions, tissue functional association: such as the topological connection between tumor and blood supply vessels, three-dimensional context information: implicitly modeled through axial attention, this stage adopts a cross-window information fusion strategy to improve the modeling ability of long-range dependence.

[0094] Stage 4: Global semantic extraction (original size 1 / 32), as the top of the feature pyramid, it realizes: high-level semantic compression: mapping image features to diagnostic semantic space, cross-modal association modeling: enhancing feature consistency across different sequences, domain adaptation: improving cross-device generalization through global statistics, and finally outputting a dual-channel representation containing global average pooling (GAP) feature vector and spatial feature map.

[0095] Of course, the feature extraction network can also be a feature extraction network based on a CNN model, as long as it can achieve multi-scale feature extraction of medical images.

[0096] In some embodiments, in step S102, the feature extraction network performs feature extraction based on a multi-level residual connection mechanism, and the feature extraction process can include:

[0097] Intra-scale residual connection: adding the feature map block to the feature calculated by local window intra-attention to form first residual feature;

[0098] Adding the first residual feature to the feature calculated by cross-window shift self-attention to form second residual feature; performing residual connection between the second residual feature after nonlinear transformation and the second residual feature before transformation to obtain the output feature map of the feature extraction layer;

[0099] Cross-scale residual connection: adding the output feature map of the i-th feature extraction layer to the output feature map of the i+1-th feature extraction layer to obtain the enhanced output feature map of the i+1-th feature extraction layer;

[0100] Wherein, i+1 is a positive integer not greater than N.

[0101] S104, input the multi-scale feature map into the feature fusion network, use intra-scale self-attention to respectively perform internal fusion on the feature map in each scale level to obtain intra-scale fused feature maps of each scale level, and use cross-scale cross-attention to perform cross-scale fusion on the intra-scale fused feature maps of different scale levels to obtain a multi-scale fused feature map.

[0102] Wherein, a two-stage feature fusion strategy of first intra-scale self-attention and then cross-scale cross-attention is adopted, the first stage performs internal enhancement on the feature map of each scale level through intra-scale self-attention, fully excavates the spatial correlation information within the same scale, and optimizes the feature expression of each scale; the second stage realizes intelligent interaction and complementary fusion between different scale features through cross-scale cross-attention, and establishes an information bridge between micro details and macro structures. In this way, the depth complementation and synergistic enhancement between features of the same scale and between features of different scales are realized, the representation ability of the model for complex medical images is improved, and the feature representation of the system for small lesions and boundary fuzzy lesions is particularly enhanced.

[0103] Exemplarily, the calculation process of applying intra-scale self-attention mechanism to each scale level of the multi-scale feature map and performing internal fusion on the feature map in each scale level to obtain the intra-scale fused feature map can include:

[0104] Generating query, key and value for the feature map of each scale level;

[0105] Calculating the similarity score of the query and the key in the same scale level;

[0106] Normalizing the similarity score by using a softmax function to obtain an attention weight;

[0107] Using the attention weight to perform weighted summation on the value to obtain the intra-scale fused feature map;

[0108] Inputting the intra-scale fused feature map into the cross-scale cross-attention module to perform cross-scale fusion on the feature map of different scale levels to obtain a multi-scale fused feature map can include:

[0109] Generating cross-scale query, cross-scale key and cross-scale value from the fused feature map of different scale levels, and these transformations can be represented by the formulas q = W q ·x, k = W k ·x, v = W v ·x, wherein x is the fused feature vector.

[0110] Calculating the similarity score of the query and the key between different scale levels, and the formula is

[0111] The cross-scale similarity scores are softmax normalized to obtain cross-scale attention weights;

[0112] The cross-scale values are weighted and summed using the cross-scale attention weights to obtain a multi-scale fusion feature map.

[0113] To efficiently integrate multi-scale information, in an example, the feature fusion network can be constructed based on a Transformer model, and the fusion mechanism based on a Transformer encoder includes:

[0114] Feature representation:

[0115] Let the scale feature of the s-th sequence be:

[0116]

[0117] wherein, represents the i-th layer feature of the s-th sequence.

[0118] Fusion process:

[0119] The features of different scales are fused by a Transformer encoder:

[0120]

[0121] The Transformer captures long-range dependencies between features of different scales through a self-attention mechanism, and combines residual connections to ensure the stability of gradient propagation, thereby realizing the complementary features across scales.

[0122] In some embodiments, Figure 3 The flowchart of the medical image processing method provided in the present application is shown in FIG. 1, wherein the feature fusion network further fuses multi-scale feature maps based on a residual connection mechanism, wherein the residual connection mechanism includes at least one of the following: Figure 3

[0123] Intra-scale residual connection: adding the feature maps of each scale level output by the feature extraction network to the corresponding intra-scale fusion feature maps to obtain enhanced intra-scale fusion feature maps;

[0124] Cross-scale residual connection: adding the enhanced intra-scale fusion feature maps of each scale level after dimension adjustment to the cross-scale fusion feature maps obtained by cross-scale cross-attention processing; wherein the dimension adjustment includes down-sampling of higher scale level feature maps and up-sampling of lower scale level feature maps, so that the dimensions match those of the cross-scale fusion feature maps;

[0125] ​Global residual connection: the multi-scale feature maps output by the feature extraction network are added to the multi-scale fusion feature maps processed by the intra-scale self-attention and the cross-scale cross-attention after dimension adjustment to obtain the final multi-scale fusion feature maps.

[0126] In this embodiment, the multi-level residual connection mechanism of this embodiment is used to strengthen the flow of feature information, retain important details of the original features, especially the feature information of small lesions in medical images, and effectively alleviate the gradient vanishing problem in the deep fusion network.

[0127] In some embodiments, as shown in Figure 3 The feature extraction layer also includes a multi-layer perceptron (MLP) module, which is arranged after each attention module. The MLP module is used to perform nonlinear transformation on the output features of the attention module. Nonlinearity is introduced through a GELU activation function to enhance the semantic differentiation ability of the features. In the feature map fusion process, the features after attention calculation are first subjected to MLP operation and then residual connection. Nonlinear transformation solves the problem of feature expression ability, and residual connection solves the problem of information loss.

[0128] S106, based on the multi-scale fusion feature map, using a classifier to obtain the processing result of the medical image.

[0129] The classification result of the medical image by the classifier can be a diagnosis result. For example, a CT image of a human lung is input, and the size, density and edge features of the lung nodule are analyzed to determine the benignity or malignancy of the nodule or the malignancy grade of the nodule.

[0130] By way of example, the last layer of the classifier is a Softmax activation function, which converts the output of the classifier into a probability distribution, suitable for multi-classification tasks. The Softmax cross-entropy loss function is defined as follows:

[0131]

[0132] where N represents the number of samples, C represents the number of classes, y i,c is the true label of sample i in class c (y i,c = 1 indicates that the sample belongs to class c, otherwise 0), is the predicted probability of sample i belonging to class c, calculated by the Softmax function:

[0133]

[0134] where z i,c is the logit value of the model output of class c.

[0135] The medical image processing method provided in the embodiments of the present application adopts a feature extraction network, a feature fusion network and a classifier to form an end-to-end medical image processing model,

[0136] The medical image processing method provided in the embodiments of the present application adopts a three-stage end-to-end design of "feature extraction-feature fusion-classification", realizes an integrated processing flow from medical image input to classification result (diagnosis result) output, avoids information loss of a separate model, facilitates the training strategy of transfer learning, can selectively retain the feature extraction capability of the feature extraction network pre-trained on a large-scale data set, and simultaneously trains the feature fusion network and the classifier, effectively solving the problem of scarcity of medical image annotation data. The medical image is processed layer by layer by the feature extraction network to realize ordered conversion from pixel-level representation to high-level semantic features, enhance the discriminability of the features, realize effective complementation and enhancement of feature information of different resolutions through the innovative double-level fusion stage, overcome the problem that details and semantic information are difficult to be considered in the traditional method, and through the joint action of multi-level residual connection and attention mechanism, the expression of key diagnostic features is enhanced, and the interference of irrelevant background and noise is inhibited, and the discrimination ability of feature signals and background noise is improved. In this way, the end-to-end medical image processing model improves the accuracy and robustness of medical image analysis through structural optimization and data processing innovation, and the detection capability of a small lesion which is clinically critical is greatly enhanced, thereby providing reliable technical support for precision medicine.

[0137] In some embodiments, the medical image processing method further includes a training step, which is a process of training an end-to-end medical image processing model including a feature extraction network, a feature fusion network and a classifier. The training step includes:

[0138] obtaining a training data set, wherein the training data set includes a plurality of medical image samples and corresponding classification labels;

[0139] using model parameters pre-trained on a large-scale visual data set as initial parameters of the feature extraction network;

[0140] iteratively training the feature extraction network, the feature fusion network and the classifier using the medical image samples, wherein the iterative training includes:

[0141] inputting the medical image samples into the feature extraction network, the feature fusion network and the classifier, and outputting classification results of the medical image samples by the classifier; and calculating cross-entropy loss between the classification results and the classification labels;

[0142] based on the cross-entropy loss, updating the parameters of the feature extraction network, the feature fusion network and the classifier by using an Adam optimizer, and dynamically adjusting the learning rate of the Adam optimizer by using a cosine decay learning rate algorithm.

[0143] The iterative training is repeatedly performed until the cross-entropy loss converges or a preset training round is reached.

[0144] wherein a SwinTransformer model or a CNN model pre-trained on a large-scale visual dataset (such as MSD, MedicalNet) is used as a basic model of the feature extraction network, the basic model is fine-tuned to obtain the trained feature extraction network. In this way, the pre-trained model has learned the general visual feature extraction capability, and the subsequent fine-tuning process can be shortened. Medical annotation data is extremely valuable and limited, and the transfer learning strategy greatly reduces the dependence on large-scale annotation data. At the same time, the pre-trained model has mastered the extraction capability of basic visual features (edges, textures, shapes, etc.), providing a high-quality starting point for medical feature learning. The Adam optimizer is used to update the model parameters because it can adaptively adjust the learning rate of each parameter, thereby speeding up the model convergence and improving the performance. At the same time, the cosine decay learning rate algorithm is used to balance the learning rate and model convergence, preventing the learning rate from being too large to cause unstable training, or too small to cause slow convergence.

[0145] Exemplarily, the training environment of the medical image processing model is deployed on a single A100 40GB GPU, the batch size is set to 16 to balance the calculation efficiency and memory occupation, and the initial learning rate is set to 0.001. Wherein, the calculation formula of the learning rate of the cosine decay learning rate algorithm is as follows:

[0146]

[0147] wherein:

[0148] is the minimum learning rate,

[0149] is the maximum learning rate,

[0150] T i is the maximum number of iterations,

[0151] T cur is the current number of iterations.

[0152] In some embodiments, the training step further comprises:

[0153] dividing the training dataset into a training subset and a validation subset;

[0154] after each preset validation interval, using the validation subset to verify the currently trained feature extraction network, feature fusion network and classifier to obtain a verification performance indicator (such as AUC, F1-score, accuracy, etc.);

[0155] When the validation performance index does not exceed the historical best value in the continuous preset number of verifications (such as 10 consecutive validation AUC improvements <0.001), the iterative training is terminated in advance.

[0156] In this embodiment, the validation performance is monitored in real time through the early stopping mechanism, and the training is automatically terminated when the performance no longer improves, effectively preventing overfitting.

[0157] In this embodiment, through the multi-task learning framework design, the model can share the bottom layer feature representation while optimizing the two key tasks of accurate feature extraction and differential diagnosis of medical images, and the mutual enhancement effect between tasks improves the overall diagnostic performance and clinical practical value of the model.

[0158] Figure 4 The structural schematic diagram of the medical image processing device provided in the present application is shown in Figure 4 The present disclosure provides a medical image processing device, which comprises:

[0159] The feature extraction unit 402 is configured to input the medical image to be processed into a feature extraction network to obtain a multi-scale feature map; wherein the feature extraction network comprises a plurality of feature extraction layers, and different layers of the feature extraction layers are configured to extract different scale feature maps.

[0160] The feature fusion unit 404 is configured to input the multi-scale feature map into a feature fusion network, perform internal fusion on the feature maps in each scale level using intra-scale self-attention to obtain intra-scale fusion feature maps of each scale level, and perform cross-scale fusion on the intra-scale fusion feature maps of different scale levels using cross-scale cross-attention to obtain a multi-scale fusion feature map.

[0161] The result output unit 406 is configured to obtain a processing result of the medical image based on the multi-scale fusion feature map using a classifier.

[0162] In a possible implementation, the feature extraction unit 402 is configured to input the feature map of the i-th layer feature extraction layer of the feature extraction network into the (i+1)-th layer feature extraction layer to obtain an output feature map of the i-th layer feature extraction layer; wherein i is a positive integer from 1 to N-1, and N is the number of layers of the feature extraction network; and the input of the first layer feature extraction layer of the feature extraction network is the medical image.

[0163] In a possible implementation, the feature extraction unit 402 is configured to perform preprocessing on an input of a feature extraction layer based on an object extraction scale corresponding to the feature extraction layer, to obtain a plurality of non-overlapping feature map blocks; the first feature extraction layer obtains the feature map blocks by performing segmentation and representation conversion on the medical image; the (i+1)th feature extraction layer obtains the feature map blocks by performing down-sampling and fusion on the feature map output by the ith feature extraction layer; the feature map blocks are subjected to feature extraction by alternately using local window intra-self-attention calculation and cross-window shift self-attention calculation, to obtain an output feature map of the feature extraction layer; wherein the feature map output by each layer from the first feature extraction layer to the Nth feature extraction layer is gradually reduced in a spatial dimension and gradually increased in a channel dimension with the deepening of a level.

[0164] In a possible implementation, the feature extraction unit 402 is configured to add the feature map blocks and features obtained by the local window intra-self-attention calculation to form first residual features; add the first residual features and features obtained by the cross-window shift self-attention calculation to form second residual features; perform residual connection between the second residual features after nonlinear transformation and the second residual features before the nonlinear transformation, to obtain the output feature map of the feature extraction layer; add the output feature map of the ith feature extraction layer and the output feature map of the (i+1)th feature extraction layer, to obtain an enhanced output feature map of the (i+1)th feature extraction layer; wherein i+1 is a positive integer not greater than N.

[0165] In a possible implementation, the feature fusion unit 404 is configured to add each scale level feature map output by the feature extraction network and a scale-intra fusion feature map corresponding to the feature map, to obtain an enhanced scale-intra fusion feature map; after dimension adjustment of the enhanced scale-intra fusion feature maps of each scale level, add the cross-scale fusion feature map obtained by the cross-scale cross-attention processing; wherein the dimension adjustment includes down-sampling of a higher scale level feature map and up-sampling of a lower scale level feature map, so that the dimensions match the dimensions of the cross-scale fusion feature map; after dimension adjustment of the multi-scale feature maps output by the feature extraction network, add the multi-scale fusion feature maps processed by the scale-intra self-attention and the cross-scale cross-attention, to obtain a final multi-scale fusion feature map.

[0166] The medical image processing apparatus provided in this embodiment can execute the method provided in the method embodiment, and is used to implement the method steps of the above embodiments, has the same technical effects as the above embodiments based on the same explanation of the meanings of the same names, and will not be described here.

[0167] As Figure 5As shown, the embodiment provides an electronic device, the electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method steps of the above embodiment.

[0168] The embodiment of the present disclosure provides a non-volatile computer storage medium, and the computer storage medium stores computer executable instructions, and the computer executable instructions are executable to perform the method steps of the above embodiment.

[0169] Reference will now be made to the following description Figure 5 , which shows a structural schematic diagram of an electronic device suitable for implementing the embodiment of the present disclosure. The terminal device in the embodiment of the present disclosure can include, but is not limited to, a mobile terminal such as a mobile phone, a notebook computer, a digital broadcast receiver, a PDA (Personal Digital Assistant), a PAD (Tablet Personal Computer), a PMP (Portable Multimedia Player), a vehicle terminal (for example, a vehicle navigation terminal), and the like, and a fixed terminal such as a digital TV, a desktop computer, and the like. Figure 5 The electronic device shown is merely an example, and should not bring any limitation to the function and use range of the embodiment of the present disclosure.

[0170] As shown, Figure 5 , the electronic device can include a processing device (for example, a central processing unit, a graphics processing unit, and the like) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the electronic device are also stored. The processing device 501, the ROM 502, and the RAM 503 are connected to each other through a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0171] Generally, the following devices can be connected to the I / O interface 505: an input device 506 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, and the like; an output device 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, and the like; a storage device 508 including, for example, a magnetic tape, a hard disk, and the like; and a communication device 509. The communication device 509 can allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 5 The electronic device with various devices is shown, but it should be understood that it is not required to implement or have all the devices shown. More or less devices can be alternatively implemented or provided.

[0172] In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer readable medium, the computer program comprising program code for performing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via the communication device 509, or installed from the storage device 508, or installed from the ROM 502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the methods of the embodiments of the present disclosure are performed.

[0173] It should be noted that the computer readable medium described above in the present disclosure can be a computer readable signal medium or a computer readable storage medium or any combination thereof. The computer readable storage medium may, for example, be, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, the computer readable storage medium can be any tangible medium that contains or stores a program used or used in conjunction with an instruction execution system, device, or apparatus. In the present disclosure, the computer readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, which carries computer readable program code. Such a propagated data signal can take on many forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium can also be any computer readable medium that is not a computer readable storage medium and that can transmit, propagate, or transport program for use by or in connection with an instruction execution system, device, or apparatus. The program code contained on the computer readable medium can be transmitted by any suitable medium, including but not limited to a wire, cable, RF (radio frequency), or the like, or any suitable combination of the above.

[0174] The above-mentioned computer readable medium can be included in the above-mentioned electronic device; or can exist separately and not be assembled into the electronic device.

[0175] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0176] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0177] The units involved in the embodiments described in this disclosure may be implemented in software or hardware, wherein the name of a unit does not necessarily limit the unit itself.

Claims

1. A method for processing medical images, characterized in that: The method comprises: Inputting the medical image to be processed into a feature extraction network to obtain a multi-scale feature map; wherein the feature extraction network includes multiple layers of feature extraction layers, and different layers of feature extraction layers are used to extract feature maps of different scales; Inputting the multi-scale feature map into the feature fusion network, using intra-scale self-attention to perform internal fusion on the feature maps in each scale level to obtain intra-scale fused feature maps at each scale level, and using cross-scale cross-attention to perform cross-scale fusion on the intra-scale fused feature maps of different scale levels to obtain a multi-scale fused feature map; Based on the multi-scale fusion feature map, a classifier is used to obtain a processing result of the medical image.

2. The method according to claim 1, characterized in that The step of inputting the medical image to be processed into a feature extraction network to obtain a multi-scale feature map includes: Inputting the feature map of the i-th feature extraction layer of the feature extraction network into the i+1-th feature extraction layer to obtain the output feature map of the i-th feature extraction layer; wherein i is a positive integer from 1 to N-1, and N is the number of layers of the feature extraction network; The input of the first feature extraction layer of the feature extraction network is the medical image.

3. The method according to claim 2, characterized in that The step of inputting the medical image to be processed into a feature extraction network to obtain a multi-scale feature map includes: Based on the object extraction scale corresponding to each feature extraction layer, the input of the feature extraction layer is preprocessed to obtain multiple non-overlapping feature blocks; wherein the first feature extraction layer obtains the feature blocks by segmenting and representing the medical image; the i+1th feature extraction layer obtains the feature blocks by downsampling and fusing the feature maps output by the i-th layer; Alternating between local window self-attention calculation and cross-window shifted self-attention calculation, feature extraction is performed on the feature map block to obtain the output feature map of the feature extraction layer; wherein, from the 1st layer to the Nth layer of feature extraction layer, the feature map output by each layer decreases layer by layer in the spatial dimension as the layer deepens, and increases layer by layer in the channel dimension as the layer deepens.

4. The method according to claim 3, characterized in that The step of inputting the medical image to be processed into a feature extraction network to obtain a multi-scale feature map further includes: Adding the feature block to the feature calculated by self-attention within the local window to form a first residual feature; adding the first residual feature to the feature calculated by self-attention across the window shift to form a second residual feature; performing a residual connection on the second residual feature after nonlinear transformation and the second residual feature before transformation to obtain the output feature map of the feature extraction layer; Add the output feature map of the i-th feature extraction layer and the output feature map of the i+1-th feature extraction layer to obtain an enhanced output feature map of the i+1-th feature extraction layer; Wherein, i+1 is a positive integer not greater than N.

5. The method according to claim 1, wherein Inputting the multi-scale feature map into a feature fusion network to obtain a multi-scale fusion feature map further includes: Adding the feature maps of each scale level output by the feature extraction network to the intra-scale fusion feature map corresponding to the feature map to obtain an enhanced intra-scale fusion feature map; The enhanced intra-scale fusion feature maps at each scale level are dimensionally adjusted and added to the cross-scale fusion feature map obtained by cross-scale cross-attention processing; the dimension adjustment includes downsampling the feature maps at higher scale levels and upsampling the feature maps at lower scale levels to match the dimensions of the cross-scale fusion feature map; The multi-scale feature map output by the feature extraction network is dimensionally adjusted and then added to the multi-scale fusion feature map processed by intra-scale self-attention and cross-scale cross-attention to obtain the final multi-scale fusion feature map.

6. The method according to any one of claims 1 to 5, characterized in that The method further comprises a training step, wherein the training step comprises: Acquire a training dataset, wherein the training dataset includes a plurality of medical image samples and their corresponding classification labels; Using model parameters pre-trained on a large-scale visual dataset as initial parameters of the feature extraction network; The feature extraction network, the feature fusion network, and the classifier are iteratively trained using the medical image samples, wherein the iterative training includes: Inputting the medical image sample into the feature extraction network, the feature fusion network and the classifier, wherein the classifier outputs a classification result of the medical image sample; calculating the cross entropy loss between the classification result and the classification label; Based on the cross entropy loss, the Adam optimizer is used to perform gradient updates on the parameters of the feature extraction network, the feature fusion network, and the classifier, wherein the learning rate of the Adam optimizer is dynamically adjusted using a cosine decay learning rate algorithm; The iterative training is repeatedly performed until the cross entropy loss converges or reaches a preset training round.

7. The method according to claim 6, characterized in that The training step further comprises: Dividing the training data set into a training subset and a validation subset; After each preset verification interval, the verification subset is used to verify the currently trained feature extraction network, the feature fusion network, and the classifier to obtain a verification performance indicator; When the verification performance index does not exceed the historical best value in the consecutive preset number of verifications, the iterative training is terminated early.

8. A medical image processing device, characterized in that: The device comprises: A feature extraction unit is used to input the medical image to be processed into a feature extraction network to obtain a multi-scale feature map; wherein the feature extraction network includes multiple layers of feature extraction layers, and different layers of feature extraction layers are used to extract feature maps of different scales; a feature fusion unit, configured to input the multi-scale feature map into a feature fusion network, perform internal fusion on the feature maps within each scale level using intra-scale self-attention to obtain intra-scale fused feature maps of each scale level, and perform cross-scale fusion on the intra-scale fused feature maps of different scale levels using cross-scale cross-attention to obtain a multi-scale fused feature map; A result output unit is used to obtain the processing result of the medical image using a classifier based on the multi-scale fusion feature map.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, causes the one or more processors to implement the method according to any one of claims 1 to 7.

Images