Polyp image segmentation method and system, electronic equipment and storage medium

Abstract

The invention provides a polyp image segmentation method and system, electronic equipment and a storage medium, and belongs to the technical field of image segmentation, and the method comprises the steps: carrying out the pairwise combination of a plurality of target image features, and obtaining a first feature, a second feature and a third feature, the plurality of target image features include a first target image feature, a second target image feature and a third target image feature, and the plurality of target image features are features acquired from the plurality of image features; fusing the first target image feature and the first feature to obtain a first global feature; performing Hadder code product on the second feature and the third feature to obtain a fourth feature; fusing the first global feature and the fourth feature to obtain a second global feature; a segmentation map is generated based on the second global feature. According to the polyp image segmentation method and system, the electronic equipment and the storage medium provided by the invention, the image segmentation precision can be improved.

Description

Technical Field

[0001] The present disclosure belongs to the technical field of image segmentation, and more specifically, relates to a polyp image segmentation method and system, electronic equipment, and storage medium. Background Art

[0002] In the field of medical image processing, polyp image segmentation is a crucial task. As an abnormal proliferation in human tissue, accurate segmentation of polyps is of great significance for early detection and treatment of diseases. However, traditional polyp image segmentation methods often have problems such as low segmentation accuracy and insufficient extraction of detailed information, which greatly limits their application in medical diagnosis. Summary of the invention

[0003] The purpose of the present disclosure is to provide a polyp image segmentation method and system, electronic equipment, and storage medium to improve the accuracy of image segmentation.

[0004] A first aspect of the embodiments of the present disclosure provides a polyp image segmentation method, comprising: Combining a plurality of target image features in pairs to obtain a first feature, a second feature, and a third feature, wherein the plurality of target image features include the first target image feature, the second target image feature, and the third target image feature, and the plurality of target image features are features obtained from the plurality of image features; fusing the first target image feature and the first feature to obtain a first global feature; Performing a Hardy code product on the second feature and the third feature to obtain a fourth feature; fusing the first global feature and the fourth feature to obtain a second global feature; A segmentation prediction map is generated based on the second global feature.

[0005] A second aspect of the embodiments of the present disclosure provides a polyp image segmentation device, comprising: a feature combination module, used for combining a plurality of target image features in pairs to obtain a first feature, a second feature and a third feature, wherein the plurality of target image features include a first target image feature, a second target image feature and a third target image feature, and the plurality of target image features are features obtained from a plurality of image features; A first fusion module, used for fusing the first target image feature and the first feature to obtain a first global feature; A feature product module, used for performing a Hardy code product on the second feature and the third feature to obtain a fourth feature; A second fusion module, used for fusing the first global feature and the fourth feature to obtain a second global feature; The segmentation module is used to generate a segmentation prediction map based on the second global feature.

[0006] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements the steps of the above-mentioned polyp image segmentation method when executing the computer program.

[0007] According to a fourth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned polyp image segmentation method are implemented.

[0008] The beneficial effects of a polyp image segmentation method and system, electronic device, and storage medium provided by the embodiments of the present disclosure are: the embodiments of the present disclosure can fully explore the relationship between features at different levels by combining multiple target image features in pairs, thereby obtaining more representative and discriminative first features, second features, and third features, avoiding the limitations of a single feature. Secondly, the first target image feature is fused with the first feature to form a first global feature, and the second feature and the third feature are multiplied by the Hardy code to obtain the fourth feature, and further fused to obtain the second global feature, which can effectively integrate different types of feature information, making the features richer and more comprehensive, and is conducive to highlighting the key features of polyps. Finally, generating a segmentation map based on the second global feature helps to improve the accuracy of segmentation, especially the accurate extraction of detailed information such as the edge of the polyp, providing a reliable basis for medical diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0010] Figure 1 A schematic flow chart of a polyp image segmentation method provided in one embodiment of the present disclosure; Figure 2 A schematic diagram of extracting boundary information from a Fourier transform image provided by an embodiment of the present disclosure; Figure 3 A schematic flow chart of another polyp image segmentation method provided by an embodiment of the present disclosure; Figure 4 A schematic diagram of the structure of a denoising unit provided in one embodiment of the present disclosure; Figure 5A schematic diagram of the structure of a multi-scale feature denoising decoder provided in an embodiment of the present disclosure; Figure 6 A schematic diagram of a colorectal polyp image plane provided by an embodiment of the present disclosure; Figure 7 A result diagram of medical image segmentation provided by an embodiment of the present disclosure; Figure 8 A visualization result diagram of an ablation experiment provided by an embodiment of the present disclosure; Fig. 9 A structural block diagram of a polyp image segmentation device provided by an embodiment of the present disclosure; Fig.10 A schematic block diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0011] In the following description, specific details such as specific system structures and technologies are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present disclosure. However, it should be clear to those skilled in the art that the present disclosure may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obstructing the description of the present disclosure with unnecessary details.

[0012] In order to make the purpose, technical solutions and advantages of the present disclosure more clear, specific embodiments will be described below in conjunction with the accompanying drawings.

[0013] Please refer to Figure 1 , Figure 1 A schematic flow chart of a polyp image segmentation method provided in one embodiment of the present disclosure, the method comprising: S101: Combine multiple target image features in pairs to obtain a first feature, a second feature, and a third feature, where the multiple target image features include the first target image feature, the second target image feature, and the third target image feature, and the multiple target image features are features obtained from multiple image features.

[0014] In this embodiment, feature extraction is performed on the common polyp image to obtain multiple image features.

[0015] Public polyp data can be obtained from the datasets Kvasir, CVC-ClinicDB, CVC-300, CVC-ColonDB, and ETIS.

[0016] Constructing a convolutional neural network (CNN). The CNN in this embodiment may include a CNN encoder composed of a five-layer ConvNeXt convolutional coding network; The CNN encoder is used to extract features from the acquired public polyp images to obtain five layers of features (i.e., multiple image features). Figure 3 As shown, multiple image features are the first layer shallow features in the CNN encoder , the second shallow layer features , the third layer of deep features , the fourth layer of shallow features 、The fifth layer of deep features .

[0017] Among them, the first layer of shallow features , the second shallow layer features , the third layer of deep features , the fourth layer of shallow features 、The fifth layer of deep features are the target image features respectively.

[0018] like Figure 3 As shown, the fifth layer of deep features That is the first target image feature, the fourth shallow feature That is the second target image feature, the third layer deep feature That is the third target image feature.

[0019] In this embodiment, reference Figure 5 , Figure 5 A flowchart of a multi-scale feature denoising decoder provided in an embodiment of the present disclosure. The third layer deep features extracted by the CNN encoder are , the fourth layer of shallow features 、The fifth layer of deep features Combine in pairs.

[0020] You can , They are respectively sent to three denoising units, refer to Figure 4 , Figure 4 A schematic diagram of the process of the denoising unit provided in one embodiment of the present disclosure. After multiple target image features are combined in pairs and enter the denoising unit, the deep features and shallow features are respectively convolved by convolution kernels of sizes , , convolution for learning; subtract the deep features and shallow features of corresponding sizes element by element, and then add the three layers of features after subtraction; finally, get the first feature, the second feature and the third feature.

[0021] The first, second and third features can be expressed as:

[0022] in, Represented as deep features, Represented as shallow features.

[0023] S102: Fusing the first target image feature and the first feature to obtain a first global feature.

[0024] In this embodiment, first, the first target image feature ( ) through the convolution kernel size The convolution learning of the first target image captures more global information. Then, an upsampling operation is performed to restore the image size of the first target image feature.

[0025] In this embodiment, the first target image feature ( ) is combined with the first feature obtained after denoising to perform channel-level feature fusion to increase the richness of the first target image feature, thereby obtaining the first global feature.

[0026] S103: Perform a Hardy code product on the second feature and the third feature to obtain a fourth feature.

[0027] In this embodiment, for the shallow subtraction features (i.e., the second feature and the third feature obtained after denoising), the Haddon code product method is used to multiply each pixel in the feature element by element, highlighting the features useful for the segmentation task and reducing the influence of the background features. After the Haddon code product processing, the fourth feature is obtained.

[0028] S104: Fusing the first global feature and the fourth feature to obtain a second global feature.

[0029] In this embodiment, the first global feature is convolved with a convolution kernel size of After convolution learning, more global information is captured; then an upsampling operation is performed to restore the image size of the first global feature; finally, the first global feature and the fourth feature are fused at the channel level to increase the richness of the feature, thereby obtaining the second global feature.

[0030] S105: Generate a segmentation prediction map based on the second global feature.

[0031] In this embodiment, the feature value of the second global feature is relocated between 0 and 1 through a Sigmoid operation to generate a high-resolution segmentation prediction map.

[0032] In this embodiment, the generated segmentation prediction map can be deeply supervised through the Mask label, which can better guide the model to better extract the edge information in the medical image.

[0033] From the above, it can be concluded that this embodiment can fully explore the relationship between features at different levels by combining multiple target image features in pairs, so as to obtain more representative and discriminative first features, second features, and third features, avoiding the limitations of single features. Secondly, the first target image feature is fused with the first feature to form the first global feature, and the second feature and the third feature are multiplied by the Hardy code to obtain the fourth feature, and further fused to obtain the second global feature, which can effectively integrate different types of feature information, making the features richer and more comprehensive, and is conducive to highlighting the key features of polyps. Finally, generating a segmentation map based on the second global feature helps to improve the accuracy of segmentation, especially the accurate extraction of detailed information such as the edge of the polyp, providing a reliable basis for medical diagnosis.

[0034] In one embodiment of the present disclosure, it also includes: Perform Fourier transform on the public polyp image to obtain the polyp image features; Perform feature extraction on the public polyp image to obtain multiple image features; fusing the polyp image feature and the first target image feature to obtain a first polyp image feature; Processing the multiple image features to obtain multiple second polyp image features; The first polyp image feature and the plurality of second polyp image features are fused to generate a segmentation map.

[0035] In this embodiment, the public polyp image is Fourier transformed to obtain the polyp image features. The semantic information of the polyp can be enhanced through the spatial domain branch of the most original public polyp image features through two identical Fourier Transform Modules (FTM). Through the frequency domain branch, the frequency domain branch converts the image from the spatial domain to the frequency domain information, and extracts the key content such as the edge texture details contained in the high-frequency information. In medical images, the edges of polyps and their own unique texture details can help depict the specific morphology and boundary conditions of polyps. After such Fourier transform processing, the polyp image features are obtained.

[0036] In this embodiment, feature extraction is performed on the public polyp image to obtain multiple image features. The feature extraction of the acquired public polyp image can be performed through a CNN encoder to obtain five layers of features (i.e., multiple image features). Figure 3 As shown, multiple image features are the first layer shallow features in the CNN encoder , the second shallow layer features , the third layer of deep features , the fourth layer of shallow features 、The fifth layer of deep features Among them, the first layer of shallow features , the second shallow layer features , the third layer of deep features , the fourth layer of shallow features 、The fifth layer of deep features are the target image features respectively.

[0037] Each layer of features contains information of different levels and dimensions of the image. Shallow features can reflect more local and basic appearance features of the image, while deep features can capture more abstract and semantic information. Multiple layers of features together constitute multiple image features, providing a rich data foundation for subsequent further processing.

[0038] In this embodiment, the first target image feature ( ) as a deep feature contains more advanced semantic information, which can form a good complement with the polyp image features that focus on details extracted by Fourier transform. The polyp image features and the first target image features are fused to obtain the first polyp image features. The polyp image features obtained by Fourier transform can be combined with the first target image features ( ) for fusion.

[0039] The polyp image features obtained by Fourier transform and the first target image features ( ) are fused so that the fused first polyp image features not only have the advantages of details such as edge texture obtained from Fourier transform, but also incorporate the high-level semantic information contained in the deep features, thereby more comprehensively describing the characteristics of the polyp.

[0040] In this embodiment, if Figure 3 As shown, the multiple image features (i.e., the first layer of shallow features) obtained by the CNN encoder can be , the second shallow layer features , the third layer of deep features , the fourth layer of shallow features ) are processed by convolution, batch normalization and activation function (Relu) in sequence to obtain multiple second polyp image features. Convolution is used to further mine its feature information, batch normalization is used to stabilize data distribution and speed up training, and Relu is used to introduce nonlinear factors to enhance its expression ability and put it in a more appropriate state.

[0041] In this embodiment, the first polyp image feature and the plurality of second polyp image features are fused to generate a segmentation map.

[0042] First, the first polyp image features are processed by convolution, batch normalization and activation function (ReLU) in sequence. A multi-layer feature fusion decoder is constructed. The decoder includes four layers of convolution operations, which combine the features of each layer of encoder in the upstream task (i.e., the first layer of shallow features) , the second shallow layer features , the third layer of deep features , the fourth layer of shallow features ) and the previous layer features for multi-scale feature fusion. Since the features of different layers differ in scale, for example, the shallow feature scale is relatively closer to the original image, with rich detail information but low semantic abstraction, while the deep features undergo multiple downsampling operations, with rich semantics but possible loss of details. Through multi-scale feature fusion, the features of different layers can complement each other, thereby making up for the differences between feature scales and integrating a more comprehensive and discriminative feature representation.

[0043] After the above preprocessing of the first polyp image features and the generation and processing of multiple second polyp image features, the two are fused. This fusion combines the advantages of the carefully processed features of both aspects, including the unique information carried by the first polyp image features after Fourier transform and related processing, and covers the multi-scale feature content of multiple second polyp image features extracted and optimized from different layers of the CNN encoder.

[0044] In this embodiment, in this embodiment, the generated segmentation prediction map can be deeply supervised by the Mask label. The segmentation prediction map can guide the CNN encoder to learn better, thereby generating a more accurate segmentation map and better extracting the edge information in the medical image.

[0045] From the above, it can be concluded that in this embodiment, the polyp image features obtained by Fourier transform are fused with the fifth-layer deep features of the CNN encoder, combining details and semantic information to make the first polyp image features more comprehensive. Secondly, the second polyp image features are obtained by processing some features of the CNN encoder, and these features are mined and optimized. Furthermore, the multi-layer feature fusion decoder fuses features of different layers at multiple scales to compensate for scale differences and integrate strong discriminative features. The final fusion generates a segmentation map that can accurately depict the morphology and boundaries of polyps, improve segmentation accuracy, provide a more reliable basis for medical diagnosis, and assist doctors in accurately judging the condition of polyps.

[0046] like Figure 2 As shown, in one embodiment of the present disclosure, Fourier transform is performed on the public polyp image to obtain polyp image features, including: Dividing the public polyp image to obtain local polyp image information and global polyp image information; Extracting spatial information from local polyp image information and global polyp image information to obtain target local features; Extracting frequency domain information from local polyp image information and global polyp image information to obtain target global features; The local features of the target and the global features of the target are fused to obtain the polyp image features.

[0047] In this embodiment, the public polyp image is divided into local polyp image information and global polyp image information , local information focuses on reflecting the details of the local area of ​​the polyp in the image, such as the texture of the polyp surface, the local edge contour, etc., while the global information reflects the relationship between the polyp and the surrounding tissue, the entire image scene, the position layout, etc. Through this diversion, the characteristics of the polyp in the image can be grasped more carefully and comprehensively.

[0048] Local polyp image information and global polyp image information The signals are fed into the frequency domain branch and the space domain branch together to extract the space and frequency domain information.

[0049] In the spatial domain, for local polyp image information, its local detail features can be further enhanced through spatial information extraction, so that it can better highlight the subtle texture of the polyp, the shape characteristics in the small area, etc.; for global polyp image information, its macro-level features such as layout and relative position in the overall image space can be extracted. The two are fused after spatial information extraction to obtain the target local features, integrating the key characteristics of the local and global in the spatial dimension.

[0050] In the frequency domain, the image is converted into representations of different frequency components. By extracting frequency domain information, high-frequency key details such as polyp edge texture can be extracted from local polyp image information, and low-frequency features that reflect the overall structure can be sorted out from the global polyp image information. The two are fused to form the target global features, integrating important local and global information in the frequency domain.

[0051] In this embodiment, two stages of the same Fourier transform module are included, and the above-mentioned target local features and target global features are input into another Fourier transform module to obtain the final target local features and target global features.

[0052] Finally, the final target local features and target global features are fused, and all the key features of the polyp image mined in the spatial domain and frequency domain are gathered together, which includes both the local and global morphology and layout characteristics observed from the spatial perspective, and the details and overall structural information reflected by the different frequency components analyzed from the frequency domain perspective. The final polyp image features can fully and multi-levelly characterize the specific situation of the polyp in the image.

[0053] From the above, it can be concluded that the present embodiment firstly diverts the public polyp images to distinguish between local and global information, which is conducive to a detailed and comprehensive grasp of the characteristics of polyps. Secondly, information is extracted in the spatial domain and frequency domain respectively and fused into target local and global features, which can enhance the details and macro features. Furthermore, a two-level Fourier transform module is set to further optimize the features and make them more accurate. Finally, the final target local and global features are fused to obtain the polyp image features, which comprehensively and multi-levelly characterizes the polyp situation, provides a rich and high-quality feature basis for subsequent polyp image analysis, segmentation and other operations, and improves processing accuracy.

[0054] In one embodiment of the present disclosure, spatial information is extracted from local polyp image information and global polyp image information to obtain target local features, including: Extracting features of the global polyp image information to obtain first global polyp image information features; Extracting features of local polyp image information to obtain first local polyp image information features; Performing a Hardy code product on the first global polyp image information feature and the first local polyp image information feature to obtain a local feature; The target local features are obtained based on the local features.

[0055] In this embodiment, when extracting spatial information, local polyp image information can be and global polyp image information is fed into the spatial branch, local polyp image information Through a convolution kernel to extract features, the details of the polyp in the local area are highlighted, such as the texture of the polyp surface, the local edge contour and other subtle details, and then the first local polyp image information features are obtained; the global polyp image information By extracting features through a convolution kernel, the macro-level features such as the layout and relative position of polyps in the overall image scene are excavated, thereby obtaining the first global polyp image information feature.

[0056] The first global polyp image information feature and the first local polyp image information feature are multiplied at the pixel level using the Hardy code convolution, that is, the features useful for the segmentation task are retained and the influence of background noise on the segmentation is reduced. For example, if a certain pixel position is related to the key performance of the polyp in both the local feature and the global feature (such as a detail feature that reflects the edge of the polyp), this feature will be highlighted after multiplication; and those parts that are more of background noise or have little relevance to the segmentation task will be relatively weakened after such multiplication, thereby achieving the effect of reducing the influence of background noise on the segmentation, and finally obtaining the local feature.

[0057] Finally, the local features are processed by convolution, normalization and ReLu activation function to obtain the target local features. The convolution operation can further explore the potential information in the local features, making the feature expression richer and more accurate; normalization helps to stabilize the data distribution, making the feature data more standardized in subsequent processing, fusion and other operations; and the ReLu activation function can limit each element in the local feature to between [0, 1]. This not only introduces nonlinear factors and enhances the expression ability of the model, but also makes the feature value within a reasonable range, and finally obtains the final target local feature.

[0058] From the above, it can be concluded that this embodiment can accurately mine high-quality target local features by extracting global and local image information features respectively, and then highlighting useful features and weakening background noise through Hardy code product, and finally using convolution, normalization and ReLu activation function optimization, so as to provide a reliable basis for polyp image segmentation and improve segmentation accuracy.

[0059] In one embodiment of the present disclosure, frequency domain information is extracted from local polyp image information and global polyp image information to obtain target global features, including: Extracting features of the global polyp image information to obtain second global polyp image information features; Performing frequency domain Fourier transform on the local polyp image information to obtain second local polyp image information features; Obtaining a global feature based on a second global polyp image information feature and a second local polyp image information feature; The target global features are obtained based on the global features.

[0060] In this embodiment, when extracting frequency domain information, local polyp image information is By performing frequency domain Fourier transform, local image information can be converted from the spatial domain to the frequency domain. In the frequency domain, the image is represented by different frequency components. Through this transformation, high-frequency key details such as the edge texture of the polyp can be extracted, and then the second local polyp image information features can be obtained. , feature learning is performed through convolution operation. The convolution kernel slides on the global image information, which can extract relevant features such as the layout and macroscopic morphology of polyps in the overall image at different positions and scales, thereby obtaining the second global polyp image information feature.

[0061] The second local polyp image information feature is added to the second global polyp image information feature to obtain the global feature; the detail information reflected in the frequency domain of the local feature and the macroscopic layout and other information learned from the spatial convolution of the global feature complement each other. After adding, the advantages of both aspects can be integrated to integrate the features of polyps in different dimensions and different perspectives, thereby obtaining a more comprehensive global feature. After obtaining the global feature, the global feature is processed through convolution, normalization, and ReLu activation function to obtain the target global feature (i.e., global edge information).

[0062] From the above, it can be concluded that this embodiment can extract high-frequency details such as polyp edge texture by performing frequency domain Fourier transform on local polyp image information, and extract features such as macroscopic morphology by convolution on global polyp image information, mining features from different dimensions. Secondly, the corresponding features of the two are added and fused, integrating the advantages of multiple perspectives to make the global features more comprehensive. Finally, after convolution, normalization, and ReLu activation function processing, potential information is further mined, data is stabilized, and expression capabilities are enhanced to obtain high-quality target global features, which provide accurate and reliable basis for subsequent tasks such as polyp image segmentation and improve processing accuracy.

[0063] In one embodiment of the present disclosure, the local polyp image information is subjected to frequency domain Fourier transform to obtain the second local polyp image information feature, including: Segmenting local polyp image information to obtain a first local feature and a second local feature; Perform feature extraction on the second local feature to obtain a third local feature; Based on the first local feature and the third local feature, a second local polyp image information feature is obtained.

[0064] In this embodiment, local polyp image information Preprocessing is first performed on the local polyp image information Perform convolution, normalization, and ReLu activation function processing. The convolution operation can mine local feature patterns in the image and enhance the image's detail information; normalization can stabilize data distribution, making the subsequent processed data more standardized; the ReLu activation function introduces nonlinear factors to enhance the model's ability to express complex image feature relationships.

[0065] Then, the preprocessed image information is sliced ​​according to the feature and The ratio of local polyp image information Divided into the first local feature (proportion ) and the second local feature (proportion ). For example, when When the proportion of (the second local feature) is 0.4, The first local feature accounts for 0.6. We mine valuable information for the final feature from different parts and perform differentiated processing on features of different proportions in a targeted manner.

[0066] The second local feature ( ) to extract edge features. In polyp images, edge features contain key detail information, such as the boundary contour of the polyp and the boundary with the surrounding tissue. By extracting edge features, the morphological characteristics of the polyp in the local area can be captured more accurately, and then the third local feature can be obtained.

[0067] Then the first local feature (1- ) is added to the third local feature, the features obtained from different emphasis angles are merged, and the local information about the polyp contained in the two parts of the features is integrated to complement each other and jointly improve the description of the local characteristics of the polyp.

[0068] Finally, the added features are convolved, normalized, and ReLu activated to obtain the second local polyp image information features, enriching the expression of polyp semantic information so that it can more comprehensively and accurately reflect the various characteristics of polyps in the local area.

[0069] From the above, it can be concluded that this embodiment optimizes the data features by preprocessing the local polyp image information, making it more conducive to subsequent operations. Secondly, the feature slices are divided in proportion, which can specifically mine the valuable information of different parts and realize differentiated processing. Furthermore, by extracting edge features, the local morphological characteristics of polyps can be accurately captured, and the features of different focus angles are integrated to improve the local feature description. Finally, the second local polyp image information features obtained by further processing enrich the semantic information, so that it can more comprehensively and accurately reflect the characteristics of polyps, and provide strong support for subsequent tasks such as polyp image analysis.

[0070] In one embodiment of the present disclosure, feature extraction is performed on the second local feature to obtain a third local feature, including: Edge feature extraction is performed on the second local feature to obtain boundary information of the second local feature, and the boundary information is used as the third local feature.

[0071] In this embodiment, the second local feature ( ) Input the edge feature extraction module to extract the boundary information in the polyp image.

[0072] First, the second local feature is subjected to fast Fourier transform to obtain the frequency domain feature.

[0073] The second local feature can be regarded as feature X. Feature X is first transformed from the spatial domain to the frequency domain by fast Fourier transform. The specific operation can be expressed as:

[0074] Among them, u and v represent the real and imaginary parameters after conversion to the frequency domain, respectively, and H and W represent the height and width of feature X, respectively. represents the ordinate of feature X, Represents the horizontal axis of feature X.

[0075] Secondly, the low-frequency information in the frequency domain features is moved to the middle of the image.

[0076] After the fast Fourier transform, all low-frequency information including contours and textures are moved to the middle of the image. The process can be expressed as:

[0077] in, It means to move the low-frequency components in the frequency domain features to the middle of the image.

[0078] In the frequency domain, low-frequency information is related to the overall structural features of the image, such as contour and texture. Concentrating it in the middle of the image facilitates subsequent unified processing and screening, and can operate on the low-frequency information in a more targeted manner.

[0079] Again, remove the low-frequency information in the frequency domain features to obtain high-frequency information.

[0080] By multiplying the low-frequency information within a certain range in the frequency domain features with a full 0 matrix of the same size to remove the low-frequency information, the process can be expressed as:

[0081] in, It represents the high-frequency information after the low-frequency components are blocked. Represents the range of blocking low-frequency information. It is represented by the operation of blocking the middle low-frequency information, specifically multiplying the middle low-frequency information by a An all-zero matrix of size .

[0082] and Multiplying the two can set the low-frequency information in the specified range to zero, thereby extracting high-frequency information. In polyp images, high-frequency information usually contains detailed features such as edges. By removing low-frequency information and retaining high-frequency information, it is more conducive to focusing on extracting edge details and highlighting information that is useful for depicting the polyp boundary contour.

[0083] Again, perform a Stack operation on the high-frequency information.

[0084] By performing a Stack operation on the features extracted from the high-frequency information in the frequency domain and adding the real and imaginary parameters, the features of different dimensions (real and imaginary parts) of the high-frequency information in the frequency domain can be integrated, making its representation more concise and easier to learn in subsequent convolution operations, further enhancing the extraction effect of edge features.

[0085] Again, convolution is performed on the features after the Stack operation.

[0086] Three convolutions with different receptive fields are used for feature learning, with sizes of , , .

[0087] Convolution kernels with different receptive fields can capture feature information at different scales. Convolution can be used to adjust the number of channels, fuse features, etc. and Convolution can extract features in a larger local range. By learning the high-frequency information after stacking through these three convolution operations, the edge features contained in the high-frequency information can be mined from multiple angles and scales, making the extraction of edge features more comprehensive and accurate.

[0088] Finally, the convolutional features are subjected to inverse fast Fourier transform.

[0089] Because subsequent operations (such as fusion with other spatial domain features, participation in the generation of the final local polyp image information features, etc.) are mostly performed in the spatial domain, the high-frequency edge detail information extracted in the frequency domain is transformed back to the spatial domain, and the key edge information extracted in the frequency domain is converted back to the spatial domain, which is convenient for collaboration with other spatial domain features and further integration and utilization of polyp image features in the entire process, thereby providing strong support for accurately depicting the morphological characteristics of polyps in local areas.

[0090] From the above, it can be concluded that this embodiment extracts edge features by performing fast Fourier transform and subsequent series of operations on the second local features, mining information from the frequency domain perspective, and can accurately focus on key details such as the polyp boundary contour. Secondly, operations such as shifting and removing low-frequency information are conducive to highlighting high-frequency detail features and strengthening the characterization of edges. Thirdly, the Stack operation makes the feature representation more concise and conducive to convolution learning, and convolutions with different receptive fields comprehensively and accurately extract edge features. Finally, the inverse fast Fourier transform is convenient for coordination with spatial domain features, providing strong support for accurately characterizing polyp morphology and improving the quality of polyp image analysis.

[0091] For example, Figure 6-8 As shown, the network framework of the embodiment is implemented in PyTorch and trained on an NVIDIA RTX3090 GPU with 24GB memory. The input image is resized to 224×224, the batch size is 32, and the CNN encoder is initialized with pre-trained weights using ConvNeXt on ImageNet. Our model is optimized using the SGD optimizer with an initial learning rate of 0.01, a momentum of 0.9, and a weight decay of 1e-4. During training, there is no overlap between the training data and the test data, and data enhancement methods such as flipping and rotation are used to enhance image quality.

[0092] The five specific embodiments include public polyp training data Kvasir, CVC-ClinicDB, CVC-300, CVC-ColonDB and ETIS, wherein the training data set includes 900 data sets from Kvasir and 550 data sets from CVC-ClinicDB, and the remaining data sets are used as tests.

[0093] In order to verify the effectiveness of each module of the model, this embodiment uses the above data set to conduct experiments with 15 different models in the fifth specific embodiment to compare the segmentation results, and an ablation experiment is performed to prove the effectiveness of the model; Table 2 shows the comparison of the results with other methods in the first Kvasir and the second CVC-ClinicDB implementation, including UNet, UNet++, Swin-UNet, UTNet, UNeXt, PraNet, FANet, TGANet, MSnet, TMF-Net, CMUNet, CMUNeXt, PRCNet, GCtx-UNet, and CFANet. It can be seen from the table that the comprehensive indicators of Dice similarity score and MAE of this method are better than those of other methods.

[0094] Table 2 Comparison of results with other methods in the first Kvasir and second CVC-ClinicDB implementations

[0095] Table 3 compares the results of this implementation in the second embodiment with other methods, including UNet, UNet++, Swin-UNet, UTNet, UNeXt, PraNet, FANet, TGANet, MSnet, TMF-Net, CMUNet, CMUNeXt, PRCNet, GCtx-UNet, and CFANet. It can be seen from the table that this method is superior to other methods in terms of IOU similarity score and MAE indicators.

[0096] Table 3 Comparison of results with other methods in the third CVC-300, fourth CVC-ColonDB and fifth ETIS embodiments

[0097] The purpose of this embodiment is to obtain the global receptive field from the multi-scale feature representation and to obtain the edge information contained in the high-frequency information in the frequency domain. Therefore, when this embodiment extracts important edge information, the image boundary features are extracted from the high-frequency domain according to the Fourier transform of the input features, and they are combined with the spatial domain features to reduce background noise and irrelevant information. In addition, this embodiment proposes a multi-scale feature denoising decoder based on the problem that the original decoder has insufficient anti-interference ability to information such as background noise. The multi-scale subtraction method is used to subtract the features in the deep and shallow information element by element. On the one hand, the noise influence in the information of different scales is reduced. On the other hand, the problem of information loss in the upsampling high layer is alleviated through the Hadard code product, and the accuracy of polyp segmentation is further improved. A large number of experiments conducted on CVC-300, CVC-ClinicDB, CVC-ColonDB, Kvasir and ETIS datasets show that this embodiment improves the accuracy of polyp segmentation.

[0098] This embodiment uses the above-mentioned polyp image segmentation method to first construct an encoder using the ConvNeXt convolutional layer to extract semantic information of local features and global features of different scales; then, an image boundary information extraction module based on Fourier transform is constructed to obtain the global receptive field in the multi-scale feature representation and the edge information contained in the high-frequency information in the frequency domain; this embodiment constructs a frequency domain global receptive field in FTM, thereby extracting image boundary features from the high-frequency domain according to the Fourier transform of the input features, and combining them with the spatial domain features to reduce background noise and irrelevant information. Finally, this embodiment constructs a multi-scale feature denoising decoder, and designs a denoising module in the cross-order multi-scale fusion decoder to effectively reduce the impact of background noise on the segmentation task and retain the useful part of the features for segmentation. On the one hand, it reduces the noise impact in the information of different scales, and on the other hand, it alleviates the problem of information loss in the upsampling high layer through the Hadard code product, further improving the accuracy of polyp segmentation. A large number of experiments conducted on CVC-300, CVC-ClinicDB, CVC-ColonDB, Kvasir and ETIS datasets show that this embodiment improves the accuracy of multi-organ segmentation and heart segmentation.

[0099] Corresponding to a polyp image segmentation method in the above embodiment, Fig. 9 This is a structural block diagram of a polyp image segmentation device provided by an embodiment of the present disclosure. For ease of explanation, only the parts related to the embodiment of the present disclosure are shown. Fig. 9 The polyp image segmentation device 20 includes: a feature combination module 21, a first fusion module 22, a feature product module 23, a second fusion module 24, and a segmentation module 25.

[0100] The feature combination module 21 is used to combine multiple target image features in pairs to obtain a first feature, a second feature and a third feature, the multiple target image features include the first target image feature, the second target image feature and the third target image feature, and the multiple target image features are features obtained from the multiple image features; A first fusion module 22, used for fusing the first target image feature and the first feature to obtain a first global feature; A feature product module 23, used for performing a Hardy code product on the second feature and the third feature to obtain a fourth feature; A second fusion module 24, used for fusing the first global feature and the fourth feature to obtain a second global feature; The segmentation module 25 is used to generate a segmentation prediction map based on the second global feature.

[0101] In one embodiment of the present disclosure, a polyp image segmentation device 20 further includes: an image segmentation module; the image segmentation module is specifically used to: Perform Fourier transform on the public polyp image to obtain the polyp image features; Perform feature extraction on the public polyp image to obtain multiple image features; fusing the polyp image feature and the first target image feature to obtain a first polyp image feature; Processing the multiple image features to obtain multiple second polyp image features; The first polyp image feature and the plurality of second polyp image features are fused to generate a segmentation map.

[0102] In one embodiment of the present disclosure, the image segmentation module is further used to: Dividing the public polyp image to obtain local polyp image information and global polyp image information; Extracting spatial information from local polyp image information and global polyp image information to obtain target local features; Extracting frequency domain information from local polyp image information and global polyp image information to obtain target global features; The local features of the target and the global features of the target are fused to obtain the polyp image features.

[0103] In one embodiment of the present disclosure, the image segmentation module is further used to: Extracting features of the global polyp image information to obtain first global polyp image information features; Extracting features of local polyp image information to obtain first local polyp image information features; Performing a Hardy code product on the first global polyp image information feature and the first local polyp image information feature to obtain a local feature; The target local features are obtained based on the local features.

[0104] In one embodiment of the present disclosure, the image segmentation module is further used to: Extracting features of the global polyp image information to obtain second global polyp image information features; Performing frequency domain Fourier transform on the local polyp image information to obtain second local polyp image information features; Obtaining a global feature based on a second global polyp image information feature and a second local polyp image information feature; The target global features are obtained based on the global features.

[0105] In one embodiment of the present disclosure, the image segmentation module is further used to: Segmenting local polyp image information to obtain a first local feature and a second local feature; Perform feature extraction on the second local feature to obtain a third local feature; Based on the first local feature and the third local feature, a second local polyp image information feature is obtained.

[0106] In one embodiment of the present disclosure, the image segmentation module is further used to: Edge feature extraction is performed on the second local feature to obtain boundary information of the second local feature, and the boundary information is used as the third local feature.

[0107] See also Fig.10 , Fig.10 A schematic block diagram of an electronic device provided by an embodiment of the present disclosure. Fig.10 The electronic device 300 in the embodiment shown may include: one or more processors 301, one or more input devices 302, one or more output devices 303 and one or more memories 304. The processors 301, input devices 302, output devices 303 and memories 304 communicate with each other via a communication bus 305. The memory 304 is used to store computer programs, which include program instructions. The processor 301 is used to execute the program instructions stored in the memory 304. The processor 301 is configured to call the program instructions to execute the functions of the modules in the above-mentioned device embodiments, such as Fig. 9 The functions of modules 21 to 25 are shown.

[0108] It should be understood that in the embodiment of the present disclosure, the processor 301 may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0109] The input device 302 may include a touch panel, a fingerprint collection sensor (for collecting the user's fingerprint information and fingerprint direction information), a microphone, etc., and the output device 303 may include a display (LCD, etc.), a speaker, etc.

[0110] The memory 304 may include a read-only memory and a random access memory, and provide instructions and data to the processor 301. A portion of the memory 304 may also include a non-volatile random access memory. For example, the memory 304 may also store information about the device type.

[0111] In a specific implementation, the processor 301, input device 302, and output device 303 described in the embodiments of the present disclosure can execute the implementation methods described in the first and second embodiments of a polyp image segmentation method provided in the embodiments of the present disclosure, and can also execute the implementation methods of the electronic device described in the embodiments of the present disclosure, which will not be repeated here.

[0112] In another embodiment of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by the processor, all or part of the processes in the above-mentioned embodiment method are implemented, and the computer program can also be completed by instructing the relevant hardware through the computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, the steps of each of the above-mentioned method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0113] The computer-readable storage medium may be an internal storage unit of the electronic device of any of the aforementioned embodiments, such as a hard disk or memory of the electronic device. The computer-readable storage medium may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Further, the computer-readable storage medium may also include both an internal storage unit of the electronic device and an external storage device. The computer-readable storage medium is used to store computer programs and other programs and data required by the electronic device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0114] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this disclosure.

[0115] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the electronic devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0116] In the several embodiments provided in the present application, it should be understood that the disclosed electronic devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces or units, or it can be an electrical, mechanical or other form of connection.

[0117] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present disclosure.

[0118] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0119] The above are only specific embodiments of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present disclosure, and these modifications or replacements should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A polyp image segmentation method, characterized in that: include: Combining a plurality of target image features in pairs to obtain a first feature, a second feature, and a third feature, wherein the plurality of target image features include the first target image feature, the second target image feature, and the third target image feature, and the plurality of target image features are features obtained from the plurality of image features; fusing the first target image feature and the first feature to obtain a first global feature; Performing a Hardy code product on the second feature and the third feature to obtain a fourth feature; fusing the first global feature and the fourth feature to obtain a second global feature; A segmentation prediction map is generated based on the second global feature.

2. A polyp image segmentation method as claimed in claim 1, characterized in that: Also includes: Perform Fourier transform on the public polyp image to obtain the polyp image features; Perform feature extraction on the public polyp image to obtain multiple image features; fusing the polyp image feature and the first target image feature to obtain a first polyp image feature; Processing the multiple image features to obtain multiple second polyp image features; The first polyp image feature and the plurality of second polyp image features are fused to generate a segmentation map.

3. A polyp image segmentation method as claimed in claim 2, characterized in that: The method of performing Fourier transform on the public polyp image to obtain the polyp image features includes: Dividing the public polyp image to obtain local polyp image information and global polyp image information; Extracting spatial information from the local polyp image information and the global polyp image information to obtain target local features; Extracting frequency domain information from the local polyp image information and the global polyp image information to obtain target global features; The target local features and the target global features are fused to obtain polyp image features.

4. A polyp image segmentation method as claimed in claim 3, characterized in that: The extracting spatial information from the local polyp image information and the global polyp image information to obtain target local features includes: Extracting features from the global polyp image information to obtain first global polyp image information features; Extracting features from the local polyp image information to obtain first local polyp image information features; Performing a Hardy code product on the first global polyp image information feature and the first local polyp image information feature to obtain a local feature; A target local feature is obtained based on the local feature.

5. A polyp image segmentation method as claimed in claim 3, characterized in that: The extracting frequency domain information from the local polyp image information and the global polyp image information to obtain target global features includes: Extracting features from the global polyp image information to obtain second global polyp image information features; Performing frequency domain Fourier transform on the local polyp image information to obtain second local polyp image information features; Obtaining a global feature based on the second global polyp image information feature and the second local polyp image information feature; A target global feature is obtained based on the global feature.

6. A polyp image segmentation method as claimed in claim 5, characterized in that: The performing frequency domain Fourier transform on the local polyp image information to obtain a second local polyp image information feature includes: Segmenting the local polyp image information to obtain a first local feature and a second local feature; Performing feature extraction on the second local feature to obtain a third local feature; Based on the first local feature and the third local feature, the second local polyp image information feature is obtained.

7. A polyp image segmentation method as claimed in claim 6, characterized in that: The extracting the second local feature to obtain the third local feature includes: Perform edge feature extraction on the second local feature to obtain boundary information of the second local feature, and use the boundary information as the third local feature.

8. A polyp image segmentation device, characterized in that: include: a feature combination module, used for combining a plurality of target image features in pairs to obtain a first feature, a second feature and a third feature, wherein the plurality of target image features include a first target image feature, a second target image feature and a third target image feature, and the plurality of target image features are features obtained from a plurality of image features; A first fusion module, used for fusing the first target image feature and the first feature to obtain a first global feature; A feature product module, used for performing a Hardy code product on the second feature and the third feature to obtain a fourth feature; A second fusion module, used for fusing the first global feature and the fourth feature to obtain a second global feature; The segmentation module is used to generate a segmentation prediction map based on the second global feature.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images