Image segmentation and classification method and device based on cross-scale attention mechanism

Abstract

The invention provides an image segmentation and classification method based on a cross-scale attention mechanism, and the method comprises the steps: segmenting an input image through a UNet model, and recognizing a focus region in the input image; wherein the UNet model comprises an encoder and a decoder, the encoder comprises a plurality of feature extraction modules which are connected in sequence, the decoder comprises a plurality of restoration modules which are connected in sequence, and the feature extraction modules and the restoration modules are symmetrically arranged; performing feature weighting, interactive processing and splicing on the encoder features extracted by the encoder and the decoder features extracted by the decoder to form cross-scale fusion features; and sending the cross-scale fusion features into a full connection layer, and outputting a progress prediction result through nonlinear transformation and a classifier. The method and device provided by the invention can improve the utilization efficiency and prediction accuracy of image features.

Description

Technical Field

[0001] The present disclosure relates to the technical field of medical image processing, and in particular to an image segmentation and classification method and device based on a cross-scale attention mechanism. Background Art

[0002] In the field of medical image processing, accurate medical image segmentation is crucial for subsequent diagnosis and treatment planning. However, since medical images are usually high-dimensional and diverse, traditional image segmentation algorithms face significant challenges in processing these data.

[0003] Specifically, in terms of segmentation, traditional algorithms often lack the ability to process high-dimensional and diverse data, making it difficult to precisely locate the boundaries of tumors. In addition, the large amount of pixel-level annotation required for segmentation is not only time-consuming and expert-dependent, but also increases the time cost of research. In terms of classification, although traditional classification models can quickly process large amounts of data and provide global judgments, they cannot provide detailed location and morphological information of tumors, limiting their clinical applications. In addition, classification models often ignore local features, affecting the accuracy of predictions.

[0004] In view of the above problems, there is an urgent need for an improved medical image segmentation method and device to improve the utilization efficiency and prediction accuracy of image features and provide efficient support for clinical decision-making. Summary of the invention

[0005] Some embodiments of the present disclosure provide an image segmentation and classification method based on a cross-scale attention mechanism, comprising the following steps:

[0006] The input image is segmented using a UNet model to identify the lesion area in the input image; wherein the UNet model includes an encoder and a decoder, the encoder includes a plurality of sequentially connected feature extraction modules for extracting features from the input image or the previous layer of features, and the decoder includes a plurality of sequentially connected restoration modules, and the feature extraction module and the restoration module are symmetrically arranged;

[0007] The encoder features feature extracted by the encoder en And the decoder feature feature extracted by the decoder de Perform feature weighting and interaction processing to generate encoder feature features based on decoder feature attention prepost1 And the decoder feature feature based on the encoder feature attention prepost2 ;

[0008] The encoder feature feature based on the decoder feature attention prepost1And the decoder feature feature based on the encoder feature attention prepost2 Splicing to form cross-scale fusion features fusion ;

[0009] The cross-scale fusion feature fusion It is sent to the fully connected layer, and through nonlinear transformation and classifier, the progress prediction result is output.

[0010] Optionally, the encoder feature feature extracted by the encoder en And the decoder feature feature extracted by the decoder de Perform feature weighting and interaction processing to generate encoder feature features based on decoder feature attention prepost1 And the decoder feature feature based on the encoder feature attention prepost2 In the step, the encoder extracts the encoder feature feature en At least two, obtained from different feature extraction modules, the decoder features extracted by the decoder feature de The encoder feature feature extracted by the encoder en The number is the same and the position is symmetrical.

[0011] Optionally, the encoder gradually encodes the input image into a low-resolution, high-dimensional feature representation through multiple layers of convolution operations and pooling operations. The encoder includes a plurality of feature extraction modules connected in sequence, each of which performs a set of convolution operations, and pooling operations are performed between adjacent feature extraction modules.

[0012] Optionally, the encoder feature feature extracted by the encoder en And the decoder feature feature extracted by the decoder de Perform feature weighting and interaction processing to generate encoder feature features based on decoder feature attention prepost1 And the decoder feature feature based on the encoder feature attention prepost2 The steps include: transforming the encoder feature feature en And decoder features feature de Input a cross-attention model, which is trained by a joint network based on a multi-layer perceptron and an attention mechanism. The cross-attention model spatially maps features of different modalities through a multi-layer perceptron, and forms attention weights for the spatially mapped features using a softmax function.

[0013] Optionally, further comprising: using a multi-task loss function to improve the prediction ability of the image segmentation classification method, and the calculation formula of the multi-task loss function L is:

[0014] L=λL CE +(1-λ)L Dice

[0015] Among them, L CE is the cross entropy loss, L Dice is the Dice loss, and λ is the weight coefficient.

[0016] Optionally, the cross entropy loss L CE The calculation formula is:

[0017]

[0018] The Dice loss L Dice The calculation formula is:

[0019]

[0020] Among them, y i is the true label, which takes the value of 0 or 1; p i is the classification probability predicted by the model; N is the number of samples.

[0021] Optionally, when the image segmentation and classification method is used to mark lesions, the value range of λ is 0.2-0.4; when the image segmentation and classification method is used to predict disease stages, the value range of λ is 0.6-0.8.

[0022] The present disclosure provides an image segmentation and classification device based on a cross-scale attention mechanism, comprising:

[0023] A segmentation module, used to segment the input image using a UNet model and identify the lesion area in the input image; wherein the UNet model includes an encoder and a decoder, the encoder includes a plurality of sequentially connected feature extraction modules, used to extract features from the input image or the previous layer of features, the decoder includes a plurality of sequentially connected restoration modules, and the feature extraction module and the restoration module are symmetrically arranged;

[0024] The cross-scale attention fusion module is used to use the cross-scale attention fusion module to fusion the encoder features extracted by the encoder. en And the decoder features extracted by the decoder feature de Perform feature weighting and interaction processing to generate encoder feature features based on decoder feature attention prepost1 And the decoder feature feature based on the encoder feature attentionprepost2 ;

[0025] A fusion module is used to integrate the encoder feature feature based on the decoder feature attention prepost1 And the decoder feature feature based on the encoder feature attention prepost2 Splicing to form cross-scale fusion features fusion ;

[0026] Prediction module, used to fusion the cross-scale feature fusion It is sent to the fully connected layer, and through nonlinear transformation and classifier, the progress prediction result is output.

[0027] An embodiment of the present disclosure provides an electronic device, including:

[0028] Processor; and

[0029] A memory, configured to store executable instructions of the processor;

[0030] The processor is configured to perform the method by executing the executable instructions.

[0031] An embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method described above is implemented.

[0032] Compared with the prior art, the image segmentation method based on the cross-scale attention mechanism provided by the embodiment of the present disclosure constructs a comprehensive progression prediction model through an attention-based cross-fusion algorithm, which can make full use of the complementarity of Encoder-Decoder to provide more accurate and comprehensive guidance for clinical decision-making. Compared with traditional post-result fusion methods, such as voting, it can more finely utilize the complex relationship between data, thereby improving the accuracy and effect of prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0034] Figure 1 A flow chart of an image segmentation and classification method based on a cross-scale attention mechanism provided for some embodiments of the present disclosure;

[0035] Figure 2A schematic diagram of the overall structure of a segmentation and classification system for an image segmentation and classification method based on a cross-scale attention mechanism provided in some embodiments of the present disclosure;

[0036] Figure 3 A schematic diagram of the local structure of a segmentation and classification system of an image segmentation and classification method based on a cross-scale attention mechanism provided in some embodiments of the present disclosure;

[0037] Figure 4 An algorithm flow chart of a weighted attention module involved in some embodiments of the present disclosure;

[0038] Figure 5 A schematic diagram of the structure of an image segmentation and classification device based on a cross-scale attention mechanism provided in some embodiments of the present disclosure;

[0039] Figure 6 A schematic diagram of the connection structure of electronic devices provided in some embodiments of the present disclosure. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solutions and advantages of the present disclosure clearer, the present disclosure will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0041] The terms used in the embodiments of the present disclosure are only for the purpose of describing specific embodiments, and are not intended to limit the present disclosure. The singular forms "a", "said" and "the" used in the embodiments of the present disclosure and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings, and "multiple" generally includes at least two.

[0042] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0043] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a product or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such product or device. In the absence of more restrictions, the elements defined by the sentence "comprises a" do not exclude the presence of other identical elements in the product or device including the elements.

[0044] As mentioned in the background technology, medical imaging data usually has the characteristics of high dimensionality, multimodality, and diversity, while the existing segmentation and classification models adopt a separate design, usually performed through independent models, which may lead to information isolation, feature loss, and insufficient learning ability for complex image features. For example, one model focuses on the boundary segmentation of lesions, and the other model focuses on the classification of global data. This separation design will cause the model to be unable to fully utilize the potential connection between the two tasks when performing tasks. The two independent models find it difficult to capture the complex relationship between global features and local features in the Encoder-Decoder structure. The separation and loss of features may lead to the model's insufficient understanding of key features, thereby reducing the accuracy of the prediction.

[0045] To this end, the present disclosure provides an image segmentation and classification method based on a cross-scale attention mechanism, comprising the following steps:

[0046] S110, segmenting the input image using a UNet model to identify the lesion area in the input image; wherein the UNet model includes an encoder and a decoder, the encoder includes a plurality of sequentially connected feature extraction modules for extracting features from the input image or a previous layer of features, the decoder includes a plurality of sequentially connected restoration modules, the feature extraction module and the restoration module are symmetrically arranged;

[0047] S120, using the cross-scale attention fusion module to fuzzy the encoder features extracted by the encoder en And the decoder features extracted by the decoder feature de Perform feature weighting and interaction processing to generate encoder feature features based on decoder feature attention prepost1 And the decoder feature feature based on the encoder feature attention prepost2 ;

[0048] S130, the encoder feature feature based on the decoder feature attention prepost1 And the decoder feature feature based on the encoder feature attention prepost2 Splicing to form cross-scale fusion features fusion ;

[0049] S140, the cross-scale fusion feature fusion It is sent to the fully connected layer, and through nonlinear transformation and classifier, the progress prediction result is output.

[0050] The image segmentation and classification method based on the cross-scale attention mechanism provided by the embodiment of the present disclosure integrates the segmentation and classification tasks into a unified network, so that the global features extracted by the encoder and the local features decoded by the decoder can interact with each other, capture the complex relationship between the above features, thereby improving the comprehensive understanding of medical images and enhancing the accuracy of prediction and clinical applicability.

[0051] The optional embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0052] Example 1

[0053] refer to Figures 1 to 4 The disclosed embodiment provides an image segmentation and classification method based on a cross-scale attention mechanism, which can be used for accurate segmentation of complex lesion areas and disease classification prediction. For example, in image analysis scenarios of diseases such as acute pancreatitis, it can achieve high-precision segmentation of lesion areas and combine classification networks to predict the patient's disease prognosis, providing important support for clinical decision-making. Specifically, the method includes the following steps:

[0054] S110, segmenting the input image using a UNet model to identify the lesion area in the input image; wherein the UNet model includes an encoder and a decoder, the encoder includes a plurality of sequentially connected feature extraction modules for extracting features from the input image or a previous layer of features, the decoder includes a plurality of sequentially connected restoration modules, the feature extraction module and the restoration module are symmetrically arranged;

[0055] S120, using the cross-scale attention fusion module to fuzzy the encoder features extracted by the encoder en And the decoder features extracted by the decoder feature de Perform feature weighting and interaction processing to generate encoder feature features based on decoder feature attention prepost1 And the decoder feature feature based on the encoder feature attention prepost2 ;

[0056] S130, the encoder feature feature based on the decoder feature attention prepost1 And the decoder feature feature based on the encoder feature attention prepost2 Splicing to form cross-scale fusion features fusion ;

[0057] S140, the cross-scale fusion feature fusion It is sent to the fully connected layer, and through nonlinear transformation and classifier, the progress prediction result is output.

[0058] Figure 2 Specifically, an end-to-end medical image segmentation and classification system based on the UNet model and the cross-scale attention mechanism is shown. The leftmost side of the system is an input module for inputting the original medical image. Specifically, in this embodiment, the input image is a cross-sectional CT image of the abdomen before treatment, in which the lesion area is located in the upper left quadrant of the abdomen (as shown by the arrow), indicating that the pancreas and surrounding tissues are abnormal. The rightmost side of the system is a segmentation result output module for generating a mask marking the specific location of the lesion area. The lower side (bottom) of the system is a classification result output module, and the different color (grayscale) blocks at the bottom represent the results of classification or progression prediction.

[0059] After the original medical image enters the input module, the segmentation and classification system extracts features and gradually generates segmentation results and classification predictions. According to the above segmentation and classification system, the lesion area can be accurately segmented, the scope of pancreatic necrosis, the accumulation of surrounding exudate, and the relationship with adjacent organs can be clarified. In addition, the severity of acute pancreatitis and the clinical prognosis of patients can be predicted based on the segmented inflammatory area, the scope of pancreatic necrosis, and the accumulation of fluid. The following is a detailed introduction to each process of the above medical image segmentation and classification system.

[0060] The right side of the input module is connected to the Encoder area, which further contains multiple feature extraction modules (BLUEEN). The Encoder area is used to extract high-level semantic features of the input image and gradually encode the image into a low-resolution but high-dimensional feature representation through multi-layer convolution operations and pooling operations. The main function of the Encoder area is to learn global information, such as the overall position and size of the lesion area. Among them, each BLUEEN module can represent a set of convolution operations, which are used to extract features from the input image or the previous layer of features. The convolution operation scans the image (or feature map) through a filter to extract local features (such as edges, textures, etc.). As the depth of the network increases, the convolution layers in the BLUEEN module can gradually learn higher-level semantic features. The arrow on the right side of the BLUEEN module can represent a pooling operation, which is usually a maximum pooling operation, which is used to downsample the feature map according to a certain window size and stride. The pooling operation can reduce the resolution of the feature map while retaining important feature information and enabling subsequent convolution layers to capture greater context information.

[0061] The left side of the output module is connected to the Decoder area, which contains multiple restoration modules (BLUEDE). The Decoder area is responsible for gradually restoring the deep features extracted by the Encoder into high-resolution feature maps for generating segmentation results. Each BLUEDE module can represent a set of convolution operations, and the arrows on the left side of the BLUEDE module can represent the corresponding upsampling operations, which are used to restore the spatial resolution of the feature map and restore the low-resolution feature map to a resolution close to that of the original image. After each BLUEDE module, the resolution of the feature is improved, the image size becomes larger, and finally restored to the same size as the input image.

[0062] In this embodiment, a cross-scale attention mechanism is further used to connect the features of the symmetrical positions of the encoder and decoder to achieve dynamic weighted fusion of features. Specifically, the above process can be implemented through one or more cross-scale attention fusion modules. As a specific example, Figure 3 A schematic diagram of a cross-scale attention fusion module is shown in Figure 2 There are two of them.

[0063] The Encoder feature module stores the Encoder feature extracted from the Encoder containing global information. en The decoder feature module stores the decoder feature extracted from the decoder containing local detail information. de .

[0064] Based on the cross attention mechanism, the Encoder feature en And Decoder feature de Perform feature weighting and interaction processing to generate Encoder feature features based on Decoder feature attention prepost1 And the Decoder feature feature based on the Encoder feature attention prepost2 .

[0065] This process can achieve weighting and interaction between features through the cross attention model. Specifically, the Encoder feature en And Decoder feature de Input the pre-trained cross attention model to generate the Encoder feature feature based on the Decoder feature attention prepost1 And the Decoder feature feature based on the Encoder feature attentionprepost2 The cross attention model is obtained by training a joint network based on a multi-layer perceptron and an attention mechanism.

[0066] The cross-attention model uses a multi-layer perceptron (MLP) to spatially map features of different modalities, and the dimensions of the biological information features and imaging features obtained through spatial mapping are consistent. Specifically, the Encoder feature feature en And Decoder feature de Dimension mapping is performed to make the feature dimensions of the two consistent. Then, the Encoder features and Decoder features obtained by spatial mapping are used to form attention weights using the softmax function.

[0067] As a specific example, see Figure 4 , with Encoder feature en Forming attention and weighted Decoder features de For example, the specific algorithm flow of the cross attention mechanism is as follows:

[0068] The features of different modalities are spatially mapped through a multi-layer perceptron (MLP): Encoder feature en First, the multi-layer perceptron (MLP) model is used for spatial mapping to make its feature dimension consistent with the Decoder feature de same, so as to be able to interact and weight.

[0069] Generate attention weights: Use the Softmax function on the mapped features. The specific calculation formula for generating attention weights is as follows:

[0070]

[0071] Among them, atten en2de It is the Decoder feature de The attention weight of i is feature de The feature weight of the i-th feature in , K is feature de feature dimension.

[0072] Using Decoder feature de The attention weight atten en2de Encoder feature feature en Perform bitwise weighting to form weighted features. The calculation formula is as follows:

[0073]

[0074] Among them, ⊙ represents the bitwise product, Indicates feature concatenation. Finally, the Decoder feature feature based on the Encoder attention mechanism is formed. PrePost1 .

[0075] Similarly, the Encoder feature feature can be en The attention weight is weighted to the Decoder feature feature de On top of that, the encoder weighted features based on the decoder attention mechanism are generated, denoted as feature PrePost2 .

[0076] The cross-scale attention fusion module further converts the weighted Encode r Feature PrePost2 and Decoder feature PrePost1 Specifically, in this embodiment, the above features can be spliced ​​by pre-fusion to obtain the final cross-attention feature feature. fusion , and finally pass through the fully connected layer to output the progress prediction result. The specific fusion formula is as follows:

[0077]

[0078] in, Represents the feature concatenation operation. The weighted Encoder feature feature PrePost2 and Decoder feature PrePost1 Merging in the feature dimension can make the fused features contain both global information and local detail information.

[0079] The number of the cross-scale attention fusion modules can be multiple, such as at least two. Figure 3 As shown, the encoder feature feature extracted by the encoder en There are two decoder features, which are obtained from different feature extraction modules. de The encoder feature feature extracted by the encoder en The number is the same and the position is symmetrical. Each module generates a cross-attention feature In this embodiment, the number is 2. The fusion method of different cross-scale modules i, j is as follows:

[0080]

[0081] Among them, module i generates cross-attention features Module j generates cross-attention features Represents the feature concatenation operation, through which the features of multiple modules are merged to generate global cross features.

[0082] Cross-attention feature fusion is a high-dimensional vector that contains global and local information extracted from multiple scales. The above fused features are fed into the fully connected layer, which outputs the progression prediction results through nonlinear transformation and classifier. The output of the fully connected layer is the predicted progression results, such as: the classification results of lesions, such as early and late stages; the development trend of the disease, such as stable and worsening conditions, etc.

[0083] Through operations such as feature weighting, pre-fusion, and multi-scale fusion, the final cross-attention feature is generated. fusion ,This feature contains global and local multi-scale information, and after being processed by the fully connected layer, it can be used for disease classification or progression prediction, greatly improving the accuracy and practicality of the model.

[0084] Furthermore, the embodiment of the present application uses a multi-task loss function to improve the prediction ability of the model. Specifically, the multi-task loss function combines the classification task and the segmentation task. The classification task uses the cross entropy loss, and the segmentation task is based on the Dice loss. The formula of the loss function L is:

[0085] L=λL CE +(1-λ)L Dice

[0086] Among them, L CE is the cross entropy loss,

[0087]

[0088] In the above formula, y i is the true label, which takes the value of 0 or 1; p i is the classification probability predicted by the model; N is the number of samples. Cross entropy loss is used for classification tasks to measure the difference between the probability distribution predicted by the model and the true label distribution, and can effectively handle binary and multi-classification problems.

[0089] Among them, L Dice is the Dice loss,

[0090]

[0091] In the above formula, y i is the true label, which takes the value of 0 or 1; p iis the classification probability predicted by the model; N is the number of samples. Dice loss is highly robust to class imbalance problems, such as small lesion areas in medical images.

[0092] Where λ is a weight coefficient used to balance the relative importance of the classification task and the segmentation task in the loss function L. The choice of λ has a key impact on the model performance. When λ is large, for example, when λ approaches 1, L≈L CE , the cross entropy loss of the classification task dominates, and the model mainly optimizes the classification task. When λ is small, for example, when λ approaches 0, L≈L Dice , the Dice loss of the segmentation task is dominant, and the model mainly optimizes the segmentation task. In actual training, if the main goal of the task is segmentation, such as accurately marking lesions in medical images, a smaller λ needs to be selected, such as 0.2-0.4; if the main goal of the task is classification, such as disease stage prediction, a larger λ needs to be selected, such as 0.6-0.8; if the importance of the two tasks is equal, λ = 0.5 can be selected.

[0093] In order to improve the generalization ability and training efficiency of the model, the embodiment of the present application adopts the cosine decay learning rate algorithm. By dynamically adjusting the learning rate, the model can gradually reduce the learning rate during the training process, avoid the oscillation during the training process caused by the excessive learning rate, and ensure better convergence effect in the later stage of training. The adjustment formula of the learning rate is as follows:

[0094]

[0095] Among them, ηt is the learning rate of the tth round; is the minimum learning rate, which is set to 0 in this embodiment; is the maximum learning rate, which is set to 0.01 in this embodiment; T cur is the current training round; T i is the number of iteration cycles, which is set to 50 in this embodiment.

[0096] Through the above training strategy, the accuracy and robustness of the multimodal data fusion method based on the cross-attention mechanism provided in the embodiment of the present application can be improved to provide a higher prediction accuracy.

[0097] Compared with the prior art, the image segmentation method based on the cross-scale attention mechanism provided by the embodiment of the present disclosure constructs a comprehensive progression prediction model through an attention-based cross-fusion algorithm, which can make full use of the complementarity of the Encoder-Decoder habitat features to provide more accurate and comprehensive guidance for clinical decision-making. Compared with traditional post-result fusion methods, such as voting, this method can more finely utilize the complex relationship between data, thereby improving the accuracy and effect of prediction.

[0098] Example 2

[0099] See also Figure 5 According to the specific implementation of the present disclosure, in the second aspect, the present disclosure provides an image segmentation and classification device 200 based on a cross-scale attention mechanism, including: a segmentation module 210, a cross-scale attention fusion module 220, a fusion module 230, and a prediction module 240.

[0100] The segmentation module 210 is used to segment the input image using the UNet model to identify the lesion area in the input image; the UNet model includes an encoder and a decoder, the encoder includes a plurality of sequentially connected feature extraction modules for extracting features from the input image or the previous layer of features, the decoder includes a plurality of sequentially connected restoration modules, and the feature extraction module and the restoration module are symmetrically arranged;

[0101] The cross-scale attention fusion module 220 is used to use the cross-scale attention fusion module to fusion the encoder feature feαture extracted by the encoder en And the decoder features extracted by the decoder feature de Perform feature weighting and interaction processing to generate encoder feature features based on decoder feature attention prepost1 And the decoder feature feature based on the encoder feature attention prepost2 ;

[0102] The fusion module 230 is used to integrate the encoder feature feature based on the decoder feature attention prepost1 And the decoder feature feature based on the encoder feature attention prepost2 Splicing to form cross-scale fusion features fusion ;

[0103] The prediction module 240 is used to transform the cross-scale fusion feature fusion It is sent to the fully connected layer, and through nonlinear transformation and classifier, the progress prediction result is output.

[0104] Example 3

[0105] like Figure 6 As shown, this embodiment provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method steps described in the above embodiment.

[0106] Example 4

[0107] An embodiment of the present disclosure provides a non-volatile computer storage medium, wherein the computer storage medium stores computer executable instructions, and the computer executable instructions can execute the method steps described in the above embodiment.

[0108] Example 5

[0109] Reference below Figure 6 , which shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present disclosure. The terminal device in the embodiments of the present disclosure may include but is not limited to mobile terminals such as mobile phones, notebook computers, vehicle-mounted terminals, etc. and fixed terminals such as desktop computers, etc. Figure 6 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0110] like Figure 6 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage device 408 into a random access memory (RAM) 403. In RAM 403, various programs and data required for the operation of the electronic device are also stored. Processing device 401, ROM 402, and RAM 403 are connected to each other via a bus 405. An input / output (I / O) interface 405 is also connected to bus 405.

[0111] Typically, the following devices may be connected to the I / O interface 405: input devices 406 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 405 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 408 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 405. The communication devices 405 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0112] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 405, or installed from the storage device 408, or installed from the ROM 402. When the computer program is executed by the processing device 401, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0113] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0114] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0115] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0116] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0117] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of the unit does not, in some cases, limit the unit itself.

[0118] Finally, it should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other. For the system or device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0119] The above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than to limit them. Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present disclosure.

Claims

1. An image segmentation and classification method based on a cross-scale attention mechanism, characterized in that: The following steps are involved: The input image is segmented using a UNet model to identify the lesion area in the input image; wherein the UNet model includes an encoder and a decoder, the encoder includes a plurality of sequentially connected feature extraction modules for extracting features from the input image or the previous layer of features, and the decoder includes a plurality of sequentially connected restoration modules, and the feature extraction module and the restoration module are symmetrically arranged; The encoder features feature extracted by the encoder en And the decoder feature feature extracted by the decoder de Perform feature weighting and interaction processing to generate encoder feature features based on decoder feature attention prepost1 And the decoder feature feature based on the encoder feature attention prepost2 ; The encoder feature feature based on the decoder feature attention prepost1 And the decoder feature feature based on the encoder feature attention prepost2 Splicing to form cross-scale fusion features fusion ;as well as The cross-scale fusion feature fusion It is sent to the fully connected layer, and through nonlinear transformation and classifier, the progress prediction result is output.

2. The image segmentation and classification method based on the cross-scale attention mechanism according to claim 1 is characterized in that: The encoder feature feature extracted by the encoder en And the decoder feature feature extracted by the decoder de Perform feature weighting and interaction processing to generate encoder feature features based on decoder feature attention prepost1 And the decoder feature feature based on the encoder feature attention prepost2 In the steps, The encoder feature feature extracted by the encoder en At least two, obtained from different feature extraction modules, the decoder features extracted by the decoder feature de The encoder feature feature extracted by the encoder en The number is the same and the position is symmetrical.

3. The image segmentation and classification method based on the cross-scale attention mechanism according to claim 1 is characterized in that: The encoder gradually encodes the input image into a low-resolution and high-dimensional feature representation through multi-layer convolution operations and pooling operations. The encoder includes a plurality of feature extraction modules connected in sequence, each of which performs a group of convolution operations, and pooling operations are performed between adjacent feature extraction modules.

4. The image segmentation and classification method based on the cross-scale attention mechanism according to claim 1, characterized in that: The encoder feature feature extracted by the encoder en And the decoder feature feature extracted by the decoder de Perform feature weighting and interaction processing to generate encoder feature features based on decoder feature attention prepost1 And the decoder feature feature based on the encoder feature attention prepost2 The steps include: The encoder feature feature en And decoder features feature de Input a cross-attention model, which is trained by a joint network based on a multi-layer perceptron and an attention mechanism. The cross-attention model spatially maps features of different modalities through a multi-layer perceptron, and forms attention weights for the spatially mapped features using a softmax function.

5. The image segmentation and classification method based on the cross-scale attention mechanism according to claim 1, characterized in that: Further including: A multi-task loss function is used to improve the prediction ability of the image segmentation classification method. The calculation formula of the multi-task loss function L is: L=λL CE +(1-λ)L Dice Among them, L CE is the cross entropy loss, L Dice is the Dice loss, and λ is the weight coefficient.

6. The image segmentation and classification method based on the cross-scale attention mechanism according to claim 5, characterized in that: The cross entropy loss L CE The calculation formula is: The Dice loss L Dice The calculation formula is: Among them, y i is the true label, which takes the value of 0 or 1; p i is the classification probability predicted by the model; N is the number of samples.

7. The image segmentation and classification method based on the cross-scale attention mechanism according to claim 5 or 6, characterized in that: When the image segmentation and classification method is used to mark lesions, the value range of λ is 0.2-0.4; when the image segmentation and classification method is used to predict disease stages, the value range of λ is 0.6-0.

8.

8. An image segmentation and classification device based on a cross-scale attention mechanism, characterized in that: include: A segmentation module, used to segment the input image using a UNet model and identify the lesion area in the input image; wherein the UNet model includes an encoder and a decoder, the encoder includes a plurality of sequentially connected feature extraction modules, used to extract features from the input image or the previous layer of features, the decoder includes a plurality of sequentially connected restoration modules, and the feature extraction module and the restoration module are symmetrically arranged; The cross-scale attention fusion module is used to use the cross-scale attention fusion module to fusion the encoder features extracted by the encoder. en And the decoder features extracted by the decoder feature de Perform feature weighting and interaction processing to generate encoder feature features based on decoder feature attention prepost1 And the decoder feature feature based on the encoder feature attention prepost2 ; A fusion module is used to integrate the encoder feature feature based on the decoder feature attention prepost1 And the decoder feature feature based on the encoder feature attention prepost2 Splicing to form cross-scale fusion features fusion ; Prediction module, used to integrate the cross-scale fusion features fusion It is sent to the fully connected layer, and through nonlinear transformation and classifier, the progress prediction result is output.

9. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 7 by executing the executable instructions.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images