Lesion identification and classification method and system for endoscopic mucosal image, apparatus, and medium

Abstract

Disclosed in the present invention are a lesion identification and classification method and system for an endoscopic mucosal image, an apparatus, and a medium. The method comprises: reading a video signal outputted by an endoscopic device to obtain mucosal images; calculating a plurality of quality characteristic values of the mucosal images, and on this basis, determining the image quality of the mucosal images; predicting a normal mucosal region, an unclear mucosal region, an abnormal mucosal region, and a lesion classification result for the abnormal mucosal region in a mucosal image meeting a preset image quality standard; and integrating the prediction results into one image to generate an identification and classification map. In the present invention, the unclear images are screened out, and then clear regions in the clear images are identified, thereby improving the accuracy of abnormal mucosal region identification and classification, and enhancing the overall accuracy and clinical utility of the lesion identification and classification method for the endoscopic mucosal image for assistive analysis. By means of classification prediction, the analysis of abnormal mucosal regions is realized, and classification is performed on the basis of clear image regions, so that the classification is accurate.

Description

Methods, systems, devices, and media for lesion identification and classification in endoscopic mucosal images

[0001] Cross-references

[0002] This application claims priority to Chinese Patent Application No. 202411214655.9, filed with the Chinese Patent Office no earlier than September 2, 2024, entitled “Method, System, Apparatus and Medium for Lesion Identification and Classification of Endoscopic Mucosal Images”, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This invention relates to the field of image recognition technology, specifically to a method, system, device, and medium for lesion identification and classification in endoscopic mucosal images. Background Technology

[0004] Digestive endoscopy is a crucial tool for identifying, classifying, and treating digestive system diseases. In recent years, advancements in endoscopic equipment and artificial intelligence (AI) technology have significantly improved the accuracy and efficiency of endoscopic examinations. Currently, digestive endoscopy techniques include traditional endoscopy, capsule endoscopy, and AI-assisted endoscopy. Among these, the application of AI technology is particularly noteworthy, as it can significantly improve the accuracy of lesion detection and classification.

[0005] Existing endoscopic examination techniques mainly rely on the experience and skills of doctors, which are subject to certain subjectivity and errors in lesion identification and classification. Although AI technology has improved the accuracy of lesion identification and classification in some aspects, endoscopic videos cannot guarantee that every frame and every area in every frame is clear. Therefore, the identification and classification of AI technology often has inaccurate problems, resulting in low overall accuracy.

[0006] In addition, there are many classification standards for gastrointestinal mucosal lesions, such as V / S, JNET, JES, and NICE classifications. Few experts can fully master all the classification standards, so it is necessary to design a method that can classify lesions.

[0007] Therefore, there is an urgent need to develop a method and system for identifying and classifying lesions in endoscopic mucosal images that is accurate and capable of classifying lesion types. Summary of the Invention

[0008] In view of the above problems, the present invention provides a method, system, device and medium for lesion identification and classification of endoscopic mucosal images.

[0009] The technical solution adopted by this invention to solve the technical problem is as follows:

[0010] In a first aspect, the present invention provides a method for lesion identification and classification in endoscopic mucosal images, comprising:

[0011] S1. Read the video signal output by the endoscope to obtain the mucosal image;

[0012] S2. Calculate multiple quality feature values ​​of the mucosal image and determine the image quality of the mucosal image accordingly;

[0013] S3. Predict the classification results of normal mucosal regions, unclear mucosal regions, abnormal mucosal regions, and abnormal mucosal regions in mucosal images that meet the preset image quality standards.

[0014] S4. The predicted results are merged into a single image to generate a recognition classification map.

[0015] In a preferred embodiment, the calculation of multiple quality feature values ​​of the mucosal image specifically involves: a first image quality detection module calculating a first quality feature value and a second quality feature value of the mucosal image; a second image quality detection module calculating a third quality feature value of the mucosal image; and a third image quality detection module calculating a fourth quality feature value and a fifth quality feature value of the mucosal image.

[0016] The first quality feature is the gradient mean, the second quality feature is the gradient variance, the third quality feature is the high-frequency energy, the fourth quality feature is the mean of the local variance, and the fifth quality feature is the variance of the local variance.

[0017] In a preferred embodiment, the step of calculating multiple quality feature values ​​of the mucosal image includes:

[0018] The first image quality detection module uses the Sobel gradient operator to calculate the gradient magnitudes in the horizontal and vertical directions of the mucosal image, synthesizes a gradient magnitude map, and calculates the mean and variance of the gradient magnitudes to obtain the gradient mean and gradient variance.

[0019] The second image quality detection module performs Fourier transform on the mucosal image to obtain the frequency domain transformed image, and calculates the energy of the high-frequency components of the frequency domain transformed image to obtain the high-frequency energy.

[0020] The third image quality detection module divides the mucosal image into multiple sub-image blocks, calculates the pixel value variance of each sub-image block, calculates the pixel value variance of each sub-image block as the local variance, and calculates the mean of the variances of all sub-image blocks and the variance of the sub-image block variances to obtain the mean of the local variances and the variance of the local variances.

[0021] In a preferred embodiment, S3 specifically involves: predicting the classification results of normal mucosal regions, unclear mucosal regions, abnormal mucosal regions, and abnormal mucosal regions in a mucosal image that meets a preset image quality standard through a multi-task cascaded network.

[0022] In a preferred embodiment, the multi-task cascaded network includes a first network module for generating a backbone feature map, a second network module for generating coarse-grained candidate target regions, and a third network module for refining the edge features of the candidate target regions to obtain classification results and segmentation masks. The third network module includes a first cascaded location prediction network module, a first cascaded mask prediction network module, a second cascaded location prediction network module, a second cascaded mask prediction network module, a third cascaded location prediction network module, and a third cascaded mask prediction network module arranged sequentially. The segmentation mask serves as the prediction result for predicting normal mucosal regions, unclear mucosal regions, and abnormal mucosal regions in a mucosal image that meets a preset image quality standard, and the classification result serves as the prediction result for predicting the subtyping result of abnormal mucosal regions.

[0023] In a preferred embodiment, S3 specifically includes:

[0024] The first network module generates multiple backbone feature maps;

[0025] The second network module performs convolution calculations on the backbone feature map obtained by the first network module to obtain the intermediate feature map.

[0026] The second network module calculates the foreground or background classification probability of the candidate region by convolution on the intermediate feature map, and calculates the bounding box regression parameters of the candidate region by convolution on the intermediate feature map.

[0027] The second network module integrates the foreground or background classification probability of the candidate region and the bounding box regression parameters of the candidate region to obtain a coarse-grained candidate target region.

[0028] The first-level location prediction network module generates preliminary classification results and preliminary bounding boxes based on coarse-grained candidate target regions and all backbone feature maps;

[0029] The first-level mask prediction network module generates a preliminary segmentation mask by performing convolution and deconvolution calculations based on the preliminary classification results, preliminary bounding boxes, and all backbone feature maps.

[0030] The second-level location prediction network module generates refined classification results and refined bounding boxes based on the preliminary classification results, preliminary bounding boxes, preliminary segmentation masks, and all backbone feature maps.

[0031] The second-level cascaded mask prediction network module generates a fine segmentation mask by convolution and deconvolution based on the refined classification results, refined bounding boxes, and all backbone feature maps.

[0032] The third-level location prediction network module generates the final classification result and the final bounding box through convolution and full convolution calculations based on the refined classification results, refined bounding boxes, fine segmentation masks and all backbone feature maps.

[0033] The third-level concatenated mask prediction network module generates the final segmentation mask based on the final bounding box through deconvolution.

[0034] In a preferred embodiment, S4 specifically includes:

[0035] The final segmentation mask for each channel is then smoothed at the edges.

[0036] Edge detection is performed on the final segmentation mask after edge contour smoothing.

[0037] Using the edge detection results, extract the contour of each channel;

[0038] The contours of all channels and the final classification results are merged into a single image to generate a recognition classification map.

[0039] Secondly, the present invention provides a lesion identification and classification system for endoscopic mucosal images, comprising:

[0040] The acquisition module is used to read the video signal output by the endoscope to obtain mucosal images;

[0041] The calculation and judgment module is used to calculate multiple quality feature values ​​of the mucosal image and determine the image quality of the mucosal image accordingly.

[0042] The prediction module is used to predict the classification results of normal mucosal regions, unclear mucosal regions, abnormal mucosal regions, and abnormal mucosal regions in mucosal images that meet preset image quality standards.

[0043] The fusion module is used to combine the prediction results of the prediction module into a single image to generate a recognition classification map.

[0044] Thirdly, the present invention provides an electronic device, characterized in that it comprises: a memory; one or more processors; one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising instructions for performing any of the methods for lesion identification and classification of endoscopic mucosal images according to the first aspect.

[0045] Fourthly, the present invention provides that the computer-readable storage medium includes instructions that, when executed on a computer, cause the computer to perform the lesion identification and classification method for endoscopic mucosal images as described in the first aspect.

[0046] This invention discloses a method, system, device, and medium for lesion identification and classification of endoscopic mucosal images. It uses multiple quality feature values ​​to analyze and eliminate mucosal images that do not meet preset image quality standards. Based on clear images, it further identifies unclear mucosal regions, performs classification prediction for abnormal mucosal regions, and integrates the results onto a single image to generate a classification map. This invention improves the accuracy of abnormal mucosal regions and classification by filtering out unclear images and identifying clear regions within clear images, thus enhancing the accuracy and reference value of endoscopic mucosal image lesion identification and classification methods. Through classification prediction, this invention achieves the analysis of abnormal mucosal regions, and classification based on clear image regions ensures accurate classification. Attached Figure Description

[0047] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 is a flowchart of the lesion identification and classification method for endoscopic mucosal images provided in Example 1.

[0049] Figure 2(a) is a flowchart of S2 in the lesion identification and classification method of endoscopic mucosal images provided in Example 2.

[0050] Figure 2(b) is a flowchart of S3 in the lesion identification and classification method of endoscopic mucosal images provided in Example 2.

[0051] Figure 2(c) is a flowchart of S4 in the lesion identification and classification method of endoscopic mucosal images provided in Example 2.

[0052] Figure 3 is a flowchart of the lesion identification and classification method for endoscopic mucosal images provided in Example 3.

[0053] Figure 4 is a flowchart of the lesion identification and classification method for endoscopic mucosal images provided in Example 4.

[0054] Figure 5 is a framework diagram of the lesion identification and classification system for endoscopic mucosal images provided in Example 5.

[0055] Figure 6 is a frame diagram of the electronic device provided in Embodiment 6. Detailed Implementation

[0056] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention, and not all of the structures.

[0057] Existing endoscopic techniques rely heavily on physician experience and skill, inherently containing subjectivity and potential errors in lesion identification and classification. While AI technology improves accuracy in some aspects, occasional image blurring can lead to inaccurate AI classification, potentially influencing physician judgment. Furthermore, numerous classification standards exist for gastrointestinal mucosal lesions, such as V / S, JNET, JES, and NICE, making it difficult for most experts to fully grasp all standards. Therefore, a method for lesion classification is needed to assist physicians. To address this, this invention provides a method, system, device, and medium for lesion identification and classification in endoscopic mucosal images.

[0058] The various embodiments of the present invention described below can be performed individually or in combination with each other, and the embodiments of the present invention do not impose specific limitations in this regard. Various components are described using terms such as "first" and "second" in the embodiments of the present invention, but these components should not be limited by these terms. These terms are only used to distinguish one component from another. The term "and / or" as used in the present invention refers to any and all combinations including one or more of the related listed items.

[0059] The following describes the methods, systems, devices, and media for lesion identification and classification in endoscopic mucosal images, as well as their technical effects.

[0060] Example 1

[0061] This embodiment provides a lesion identification and classification method for endoscopic mucosal images. Figure 1 is a schematic diagram of the lesion identification and classification method for endoscopic mucosal images provided in this embodiment. The method includes:

[0062] S1. Read the video signal output by the endoscope to obtain the mucosal image;

[0063] S2. Calculate multiple quality feature values ​​of the mucosal image and determine the image quality of the mucosal image accordingly;

[0064] S3. Predict the normal mucosal region, unclear mucosal region, abnormal mucosal region, and predict the abnormal mucosal region classification results in mucosal images that meet the preset image quality standards.

[0065] S4. The predicted results are merged into a single image to generate a recognition classification map.

[0066] Understandably, the mucosal images in S3 that meet the preset image quality standards contain at least one of the following: normal mucosal areas, unclear mucosal areas, and abnormal mucosal areas.

[0067] The prediction result of the mucosal image that meets the preset image quality standard for predicting normal mucosal areas, unclear mucosal areas and abnormal mucosal areas may be that it does not have at least one of unclear mucosal areas, normal mucosal areas and abnormal mucosal areas.

[0068] The unclear mucosal area may be an area with poor image quality, or an area that does not belong to the normal mucosal area and cannot be identified as an abnormal mucosal area.

[0069] In this embodiment, the abnormal mucosal region and the abnormal mucosal region classification result are predicted simultaneously. In other embodiments, the normal mucosal region, the unclear mucosal region and the abnormal mucosal region can be predicted first. If the abnormal mucosal region exists, the abnormal mucosal region classification result is predicted then.

[0070] Typically, in S4, unclear or abnormal mucosal areas are marked on the recognition and classification map, while unmarked areas are considered normal mucosal areas. These normal mucosal areas are also specially marked. This method of not marking normal mucosal areas ensures image clarity and reduces the need for fusion annotation operations.

[0071] Here, it is not limited to fusing the prediction results of several mucosal images into one image in S4. For example, it can be fusing the prediction results of one mucosal image into one image, or it can be fusing the prediction results of several relatively close frames of mucosal images from several videos into one image. In this embodiment, the prediction result of each mucosal image corresponds to a recognition classification image.

[0072] This embodiment of the endoscopic mucosal image lesion identification and classification method integrates image quality assessment. It uses multiple quality feature values ​​to analyze and eliminate mucosal images that do not meet preset image quality standards. Based on clear images, it further identifies unclear mucosal regions, performs classification prediction for abnormal mucosal regions, and integrates the results onto a single image to generate a classification map. This embodiment improves the accuracy of abnormal mucosal regions and classification by filtering out unclear images and identifying clear regions within clear images as normal or abnormal. This enhances the accuracy and reference value of the endoscopic mucosal image lesion identification and classification method. This embodiment achieves the analysis of abnormal mucosal regions through classification prediction, and the classification based on clear image regions ensures accurate classification.

[0073] Example 2

[0074] The existing AI technologies used in gastrointestinal endoscopy have the following main drawbacks:

[0075] Regarding the accuracy of lesion identification and classification: Although AI technology has improved the detection rate of some lesions, it still has shortcomings in the classification and precise segmentation of complex mucosal lesions. The boundaries of some lesion areas are blurred and difficult to identify, and the wide variety of lesions increases the difficulty of lesion identification and classification, resulting in a high error rate.

[0076] Regarding image quality: In actual operation, the clarity of the image can be affected by factors such as endoscopic operation and patient movement, resulting in blurred images in some areas, which is not conducive to accurate lesion identification and classification.

[0077] In terms of time efficiency: Current endoscopic examinations rely on the experience and skills of doctors, and the speed of lesion identification and classification is slow and easily affected by subjective factors. Although the application of AI can improve the speed, it still needs further optimization to achieve real-time and efficient lesion identification and classification.

[0078] Regarding classification: There are many classification standards for gastrointestinal mucosal lesions, such as V / S, JNET, JES, and NICE classifications. Few experts can fully master all the classification standards, so it is necessary to design a method that can classify lesions.

[0079] Based on the above problems, compared with Embodiment 1, this embodiment provides a detailed method for lesion identification and classification of endoscopic mucosal images, including:

[0080] S1. Using a video capture card, transmit digital / analog video signals such as HDMI, DVI, SDI, and S-Video to the computer. Use OpenCV to read the video signals output by the endoscope and convert them frame by frame into RGB image format to obtain video frame images, which in turn yield several mucosal images.

[0081] S2. Calculate the quality feature value of the mucosal image using the image quality detection module, and classify the mucosal image based on the quality feature value using a binary classification model to obtain the image quality judgment result of the mucosal image.

[0082] It can be understood that the results of judging the image quality of the mucosal image are divided into two categories: one is a clear image, which is a mucosal image that meets the preset image quality standard, and the other is an unclear image, which is a mucosal image that does not meet the preset image quality standard.

[0083] The image quality detection module includes a first image quality detection module, a second image quality detection module, and a third image quality detection module. The first image quality detection module calculates the first quality feature value (gradient mean) and the second quality feature value (gradient variance) of the mucosal image. The second image quality detection module calculates the third quality feature value (high frequency energy) of the mucosal image. The third image quality detection module calculates the fourth quality feature value (mean of local variance) and the fifth quality feature value (variance of local variance) of the mucosal image.

[0084] Specifically, the first and second quality characteristics are gradient features; the first quality characteristic is the gradient mean, and the second quality characteristic is the gradient variance. The third quality characteristic is a frequency domain feature, specifically high-frequency energy. The fourth and fifth quality characteristics are local variance features; the fourth quality characteristic is the mean of the local variance, and the fifth quality characteristic is the variance of the local variance.

[0085] Referring to Figure 2(a), the gradient of the mucosal image is calculated using the first image quality detection module. Specifically, the Sobel gradient operator is used to calculate the gradient magnitudes in the horizontal and vertical directions, respectively. Then, a gradient magnitude map is synthesized, and the mean and variance of the gradient magnitudes are calculated. The mean and variance of the gradient magnitudes are used to quantify edge sharpness, obtaining the first quality feature value (gradient mean) and the second quality feature value (gradient variance). The gradient magnitude in the horizontal direction is represented by G. x The gradient magnitude in the vertical direction is represented by G. y express,

[0086] Where I represents the mucosal image, x represents the horizontal coordinate of the mucosal image, and y represents the horizontal coordinate of the mucosal image. That is, the mucosal image can be represented as I(x,y).

[0087] The high-frequency energy of the mucosal image is calculated using the second image quality detection module, specifically as follows:

[0088] Performing a Fourier transform on the mucosal image yields the frequency domain transformed image, as shown in the following formula:

[0089] Where F(u,v) represents the image after frequency domain transformation, u represents the x-coordinate of the image after frequency domain transformation, v represents the y-coordinate of the image after frequency domain transformation, M represents the width of the mucosal image, N represents the height of the mucosal image, I'(x,y) represents the grayscale mucosal image in the spatial domain, and j represents the imaginary number.

[0090] Calculate the energy of the high-frequency components of the image after frequency domain transformation, i.e., high-frequency energy, which is used to quantify the image's detail information. The high-frequency energy is the third quality eigenvalue, E. highThe calculation formula is as follows:

[0091] High_Frequency represents the high-frequency domain.

[0092] The fourth and fifth quality feature values ​​of the mucosal image are calculated using the third image quality detection module, including:

[0093] The mucosal image is divided into multiple sub-image blocks, for example, 64 sub-image blocks in an 8×8 format. The variance of pixel values ​​in each sub-image block (also known as local variance or sub-image block variance) is calculated using the following formula:

[0094] Where, σ 2 I represents the variance of pixel values ​​in a sub-image patch, P represents the number of pixels in the sub-image patch, and I represents the variance of pixel values ​​in the sub-image patch. i represents the i-th pixel value of the sub-image block, and μ represents the average pixel value of the sub-image block;

[0095] The mean and variance of the variances of all sub-image patches are calculated to quantify the sharpness of local regions (sub-image patches), resulting in the fourth quality feature value (mean of local variance) and the fifth quality feature value (variance of local variance).

[0096] Using the first fusion judgment module, the image quality is determined based on the first, second, third, fourth, and fifth quality feature values ​​to determine whether it meets the preset image quality standard, thus obtaining the image quality result (clear / unclear). Specifically,

[0097] The first fusion judgment module standardizes the extracted gradient features, frequency domain features, and local variance features. The standardized first quality feature value, second quality feature value, third quality feature value, fourth quality feature value, and fifth quality feature value are then input into the SVM model. The SVM model outputs the image quality classification result, which is either the image meets the preset image quality standard (clear image) or does not meet the preset image quality standard (unclear image).

[0098] S3. Mucosal images that meet the preset image quality standards are used as input to a pre-trained multi-task cascaded network. The output of the multi-task cascaded network is the abnormal mucosal region and its classification result, the normal mucosal region and / or unclear mucosal region. That is, the prediction result is obtained through the prediction of the multi-task cascaded network.

[0099] S3 includes constructing a multi-task cascaded network and training the multi-task cascaded network to predict the mucosal image in S2 that meets the preset image quality standard, and obtain the prediction result. The prediction result corresponds to the optimized final segmentation mask and the final classification result below. The final classification result is used as the classification result of the multi-task cascaded network, and the final segmentation mask is used as the segmentation result of the multi-task cascaded network.

[0100] Specifically, the multi-task cascaded network includes: a first network module for feature extraction and generating a backbone feature map; a second network module for generating coarse-grained candidate target regions; and a third network module for refining the edge features of the candidate target regions to obtain classification results (category labels of the target regions) and segmentation masks. The third network module includes three location prediction network modules and three mask prediction network modules, which are interspersed and sequentially connected. The segmentation mask output by the third network module serves as the prediction result for normal mucosal regions, unclear mucosal regions, and abnormal mucosal regions in a mucosal image that meets a preset image quality standard. The classification result obtained by the third network module serves as the prediction result for the abnormal mucosal region classification. The first, second, and third network modules are connected sequentially, and the output of the first network module is connected to the third network module. See Figure 2(b) for details.

[0101] The first network module, used for feature extraction, is, in one specific embodiment, composed of four sets of residual network modules with attention mechanisms. A mucosal image conforming to a preset image quality standard is input to the first network module. The four sets of residual network modules with attention mechanisms respectively obtain a first backbone feature map, a second backbone feature map, a third backbone feature map, and a fourth backbone feature map. It can be understood that the second backbone feature map is obtained based on the first backbone feature map, the third backbone feature map is obtained based on the second backbone feature map, and the fourth backbone feature map is obtained based on the third backbone feature map. Here, the fourth backbone feature map is used as the final backbone feature map obtained by the first network module.

[0102] The second network module receives the fourth backbone feature map output by the first network module. This second network module uses 3x3 convolutional layers to generate intermediate feature maps. The second network module also includes two sets of branch networks: a first 1x1 convolutional layer and a second 1x1 convolutional layer. The intermediate feature map is input into the first 1x1 convolutional layer to generate foreground / background classification probabilities for candidate regions, outputting a first two-dimensional tensor (shape H, W, 2H, W, 2H, W, 2, where H and W are the feature map height and width). The intermediate feature map is then input into the second 1x1 convolutional layer to obtain the bounding box regression parameters for the candidate regions, outputting a second two-dimensional tensor (shape H, W, 4H, W, 4H, W, 4, where H and W are the feature map height and width, and the final 4 represents the four coordinates of the bounding box). The outputs of the two sets of branch networks are fused to obtain a list of candidate regions, i.e., coarse-grained candidate target regions. These coarse-grained candidate target regions correspond to coarse-grained normal mucosal regions, coarse-grained unclear mucosal regions, and coarse-grained abnormal mucosal regions.

[0103] The third network module comprises three cascaded location prediction network modules and three cascaded mask prediction network modules, interconnected and interleaved. Each cascaded location prediction network module is followed by a cascaded mask prediction network module. Therefore, the third network module includes, in sequence, a first cascaded location prediction network module, a first cascaded mask prediction network module, a second cascaded location prediction network module, a second cascaded mask prediction network module, a third cascaded location prediction network module, and a third cascaded mask prediction network module. The third network module refines the edge features of candidate target regions to obtain classification results and segmentation masks. The third network module includes several sequentially arranged convolutional layers, fully convolutional layers, and deconvolutional layers. The fully convolutional layers output the classification results, and the deconvolutional layers output the segmentation masks.

[0104] In the third network module, the segmentation operation corresponds to segmenting normal mucosal regions, unclear mucosal regions, and abnormal mucosal regions, while the classification operation in the third network module corresponds to classifying abnormal mucosal regions.

[0105] The system employs a first-cascaded location prediction network module and a first-cascaded mask prediction network module. The first-cascaded location prediction network module takes as input a candidate region list and multiple (e.g., four) backbone feature maps generated by the first network module. Within this module, corresponding feature regions are extracted from the backbone feature maps based on the candidate region list output by the second network module. These feature regions are then processed through multiple convolutional and pooling layers to generate preliminary classification results and preliminary bounding boxes, outputting preliminary classification results (category labels) and preliminary bounding boxes. The first-cascaded mask prediction network module takes as input the output of the first-cascaded location prediction network module (preliminary classification results and preliminary bounding boxes) and four backbone feature maps. Within this module, the preliminary classification results and the four backbone feature maps are processed through multiple convolutional and deconvolutional layers to generate a preliminary segmentation mask, outputting a preliminary segmentation mask.

[0106] The second-level cascaded location prediction network module and the second-level cascaded mask prediction network module are used. The input to the second-level cascaded location prediction network module includes the preliminary classification results, preliminary bounding boxes, and preliminary segmentation masks from the first-level cascaded location prediction network module, as well as four backbone feature maps from the first network module. Similar to the structure of the first-level cascaded location prediction network module, the second-level cascaded location prediction network module further processes and refines the preliminary results, generating refined classification results and refined bounding boxes. The output is the refined classification results and refined bounding boxes. The input to the second-level cascaded mask prediction network module includes the output of the second-level cascaded location prediction network module (refined classification results and refined bounding boxes) and the four backbone feature maps. In the second-level cascaded mask prediction network module, the refined classification results and the four backbone feature maps are processed through multiple convolutional and deconvolutional layers to generate a more refined segmentation mask. The output is the refined segmentation mask.

[0107] The system employs a third-level concatenated location prediction network module and a third-level concatenated mask prediction network module. The input to the third-level concatenated location prediction network module includes the refined classification results and refined bounding boxes from the second-level concatenated location prediction network module, the fine segmentation mask from the second-level concatenated mask prediction network module, and four backbone feature maps. Within the third-level concatenated location prediction network module, the results are further processed and refined through multiple convolutional layers and fully convolutional layers to generate the final classification results and final bounding boxes. The output of the third-level concatenated mask prediction network module includes the output of the third-level concatenated location prediction network module (the final bounding box). Within the third-level concatenated mask prediction network module, the final bounding box is processed through deconvolutional layers to generate the final segmentation mask. The output of the final segmentation mask is the final segmentation mask. The final segmentation mask serves as the prediction result for normal mucosal regions, unclear mucosal regions, and abnormal mucosal regions in a mucosal image that meets a preset image quality standard. The final classification result serves as the prediction result for the classification of abnormal mucosal regions.

[0108] A multi-task cascaded network is used as a mucosal partitioning prediction network model. For example, for mucosal images from gastroscopy, the mucosal partitioning prediction network model outputs V / S classification results, and for mucosal images from colonoscopy, the mucosal partitioning prediction network model outputs JNET classification results. The specific model depends on the preset training data and is not limited in this invention.

[0109] For the training of the multi-task cascaded network, specifically, a large number of endoscopic images are collected and labeled to ensure that each image contains detailed segmentation and classification information of the mucosal region. Here, target images can be selected according to actual needs and labeled with specified classification categories. For example, magnified images of gastric and esophageal mucosa can be labeled, and the labeling categories can be labeled according to VS classification and JES classification.

[0110] S4. Smooth the edge contours of the final segmentation mask, then perform edge detection, extract the contours using the edge detection results, and fuse the extracted contours and regional target category labels into a single image to generate a recognition classification map.

[0111] Referring to Figure 2(c), S4 specifically refers to:

[0112] S4.1. Perform edge contour smoothing on the final segmentation mask of each channel, that is, perform erosion + dilation on the multi-channel mask image to smooth the edge contour.

[0113] S4.2 Perform edge detection on the final segmentation mask after edge contour smoothing, specifically using the Sobel operator or the Canny edge detection algorithm;

[0114] S4.3. Using the edge detection results, extract the contour of each channel using the findContours function in OpenCV;

[0115] S4.4. Merge the results from all channels (contours and final classification results) into a single image to generate a recognition classification map.

[0116] The lesion identification and classification method for endoscopic mucosal images in this embodiment improves the segmentation and classification accuracy of mucosal regions in endoscopic images through a multi-stage deep learning network. In particular, the identification effect in complex lesion regions is far superior to that of existing technologies, thereby achieving high-precision lesion identification and classification.

[0117] The lesion identification and classification method for endoscopic mucosal images in this embodiment can control the image quality used for prediction. It integrates image quality assessment by analyzing multiple quality feature values ​​to eliminate mucosal images that do not meet the preset image quality standards. Based on clear images, it further determines unclear mucosal areas, and double-screens unclear areas. For clear areas, it determines abnormal mucosal areas and abnormal mucosal area classification results, which improves prediction accuracy and enhances the reliability of intelligent lesion identification and classification.

[0118] This embodiment achieves the analysis of abnormal mucosal regions through classification prediction, and the classification is accurate based on clear image regions.

[0119] This embodiment generates an identification classification map, which includes the identification and labeling of normal mucosal areas, unclear mucosal areas, and abnormal mucosal areas, as well as the classification results of abnormal mucosal areas. The highly accurate identification classification map facilitates doctors' diagnosis of diseases and improves their work efficiency.

[0120] Example 3

[0121] Please refer to Figure 3. Compared to Embodiment 2, S3 in this embodiment differs from that in Embodiment 2. Here, S3 specifically includes:

[0122] Segmentation algorithms such as U-Net or Mask R-CNN are used to segment targets in mucosal images that meet preset image quality standards to obtain bounding boxes;

[0123] Based on the bounding boxes, classification networks such as ResNet or DenseNet are used to predict the classification results of normal mucosal regions, unclear mucosal regions, abnormal mucosal regions, and abnormal mucosal regions in mucosal images that meet the preset image quality standards.

[0124] The lesion identification and classification method for endoscopic mucosal images in this embodiment can perform lesion classification and has high accuracy in identifying normal mucosal areas, unclear mucosal areas, abnormal mucosal areas, and classifications. Based on this, it can improve the efficiency and accuracy of doctors' lesion identification and classification.

[0125] Example 4

[0126] Compared to Example 2, this example provides detailed processing for images that do not meet the preset image quality standards, as shown in Figure 4, specifically including:

[0127] Mucosal images that do not meet the preset image quality standards are classified into two categories: mucosal images that can be optimized and mucosal images that are difficult to optimize. For the mucosal images that can be optimized, intelligent image clarity optimization is performed. For the optimized images, it can be determined again whether they meet the preset image quality standards. If they do, steps S3 and S4 are performed.

[0128] As an example, the intelligent optimization method could be:

[0129] The image of the mucosa can be sharpened by optimizing the local contrast of the image region where the suspended impurities are located, thereby generating a sharpened image.

[0130] An optimized image is generated based on the visual scene enhancement processing of the sharpened image, that is, the feature information of the visual scene features is enhanced.

[0131] As another example, the intelligent optimization method could be:

[0132] For the deblurring of the mucosal image that can be optimized, the specific image is divided into multiple image blocks; each image block is analyzed and marked, the original clear image block and the image block to be optimized are marked, and the blurred areas in each image block to be optimized are removed by conventional local image deblurring method.

[0133] In some embodiments, for mucosal images that do not meet preset image quality standards or optimized images, normal mucosal regions, unclear mucosal regions, and abnormal mucosal regions in the image are predicted separately. If there is an abnormal mucosal region, the abnormal mucosal region classification result is predicted.

[0134] The lesion identification and classification method of endoscopic mucosal images in this embodiment can make full use of the mucosal images in the video signal, and can perform lesion classification. It has high accuracy in identifying normal mucosal areas, unclear mucosal areas, abnormal mucosal areas, and classification, thereby improving the efficiency and accuracy of lesion identification and classification for doctors.

[0135] Example 5

[0136] Please refer to Figure 5. This embodiment provides a lesion identification and classification system for endoscopic mucosal images, including:

[0137] The acquisition module 10 is used to read the video signal output by the endoscope device to obtain a mucosal image;

[0138] The calculation and judgment module 20 is used to calculate multiple quality feature values ​​of the mucosal image and determine the image quality of the mucosal image accordingly.

[0139] Prediction module 30 is used to predict the normal mucosal region, unclear mucosal region, abnormal mucosal region, and abnormal mucosal region classification results in mucosal images that meet preset image quality standards.

[0140] The fusion module 40 is used to fuse the prediction results of the prediction module 30 into a single image to generate a recognition classification map.

[0141] In this embodiment, the calculation and judgment module 20 is used to calculate the first quality feature value and the second quality feature value of the mucosal image using the first image quality detection module, to calculate the third quality feature value of the mucosal image using the second image quality detection module, to calculate the fourth quality feature value and the fifth quality feature value of the mucosal image using the third image quality detection module, and to determine the image quality of the mucosal image based on the first quality feature value, the second quality feature value, the third quality feature value, the fourth quality feature value and the fifth quality feature value.

[0142] The first quality feature is the gradient mean, the second quality feature is the gradient variance, the third quality feature is the high-frequency energy, the fourth quality feature is the mean of the local variance, and the fifth quality feature is the variance of the local variance.

[0143] Understandably, the calculation and judgment module 20 includes a first image quality detection module, a second image quality detection module, a third image quality detection module, and a judgment unit;

[0144] The first image quality detection module is used to calculate the gradient magnitudes in the horizontal and vertical directions of the mucosal image using the Sobel gradient operator, then synthesize the gradient magnitude map, and calculate the mean and variance of the gradient magnitudes to obtain the gradient mean and gradient variance.

[0145] The second image quality detection module is used to perform Fourier transform on the mucosal image to obtain the frequency domain transformed image, and calculate the energy of the high frequency components of the frequency domain transformed image to obtain the high frequency energy.

[0146] The third image quality detection module is used to segment the mucosal image into multiple sub-image blocks, calculate the pixel value variance of each sub-image block, calculate the pixel value variance of each sub-image block as the local variance, and calculate the mean of the variances of all sub-image blocks and the variance of the sub-image block variances to obtain the mean of the local variances and the variance of the local variances.

[0147] The judgment unit is used to determine the image quality of the mucosal image based on the gradient mean, gradient variance, high-frequency energy, mean of local variance, and variance of local variance.

[0148] The prediction module 30 includes a multi-task cascaded network unit. The multi-task cascaded network unit carries a multi-task cascaded network and is used to predict the classification results of normal mucosal regions, unclear mucosal regions, abnormal mucosal regions, and abnormal mucosal regions in a mucosal image that meets preset image quality standards.

[0149] The multi-task cascaded network includes a first network module for generating backbone feature maps, a second network module for generating coarse-grained candidate target regions, and a third network module for refining edge features of the candidate target regions to obtain classification results and segmentation masks. The third network module includes a first cascaded location prediction network module, a first cascaded mask prediction network module, a second cascaded location prediction network module, a second cascaded mask prediction network module, a third cascaded location prediction network module, and a third cascaded mask prediction network module arranged sequentially. The segmentation mask output by the third network module serves as the prediction result for predicting normal mucosal regions, unclear mucosal regions, and abnormal mucosal regions in a mucosal image that meets a preset image quality standard. The classification result output by the third network module serves as the prediction result for the abnormal mucosal region classification.

[0150] The first network module is used to generate multiple backbone feature maps;

[0151] The second network module is used to perform convolution calculation on the backbone feature map obtained by the first network module to obtain an intermediate feature map, to perform convolution calculation on the intermediate feature map to generate the foreground or background classification probability of the candidate region, to perform convolution calculation on the intermediate feature map to generate the bounding box regression parameters of the candidate region, and to fuse the foreground or background classification probability of the candidate region and the bounding box regression parameters of the candidate region to obtain a coarse-grained candidate target region.

[0152] The first cascaded location prediction network module is used to generate preliminary classification results and preliminary bounding boxes based on coarse-grained candidate target regions and all backbone feature maps, and to perform convolution and deconvolution calculations based on the preliminary classification results, preliminary bounding boxes and all backbone feature maps to generate a preliminary segmentation mask;

[0153] The second cascaded location prediction network module is used to generate refined classification results and refined bounding boxes based on the preliminary classification results, preliminary bounding boxes, preliminary segmentation masks, and all backbone feature maps.

[0154] The second concatenated mask prediction network module is used to generate a fine segmentation mask by convolution and deconvolution based on the refined classification results, refined bounding boxes and all backbone feature maps;

[0155] The third-level concatenated location prediction network module is used to generate the final classification result and the final bounding box through convolution and full convolution calculations based on the refined classification result, refined bounding box, fine segmentation mask and all backbone feature maps.

[0156] The third concatenated mask prediction network module is used to generate the final segmentation mask by deconvolution calculation based on the final bounding box.

[0157] Fusion module 40 includes:

[0158] The smoothing unit is used to smooth the edge contours of the final segmentation mask for each channel.

[0159] The edge detection unit is used to perform edge detection on the final segmentation mask after the edge contour has been smoothed.

[0160] The contour extraction unit is used to extract the contour of each channel using the edge detection results;

[0161] The fusion unit is used to merge the results from all channels (contours and the final classification result) into a single image to generate a recognition classification map.

[0162] In specific implementation, the lesion identification and classification system for endoscopic mucosal images can be implemented by referring to the lesion identification and classification method for endoscopic mucosal images in any of the above embodiments. The specific implementation steps will not be repeated here.

[0163] The lesion identification and classification system for endoscopic mucosal images in this embodiment can not only classify lesions, but also has high accuracy in identifying normal mucosal areas, unclear mucosal areas, abnormal mucosal areas, and their classifications. Based on this, it can improve the efficiency and accuracy of doctors' lesion identification and classification.

[0164] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the functions described above can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0165] Example 6

[0166] This embodiment provides an electronic device that can be directly connected to an endoscope.

[0167] The method of the present invention can implement an electronic device, as shown in FIG6, the electronic device including: a memory; one or more processors; one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for performing a lesion identification and classification method for endoscopic mucosal images according to any of the above embodiments.

[0168] The processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0169] Memory can include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. Furthermore, memory can include any combination of computer-readable storage media; memory can be a semiconductor memory chip, a magnetic disk, or an optical disk.

[0170] The memory stores executable code, which, when processed by the processor, can cause the processor to execute some or all of the methods described above.

[0171] Example 7

[0172] This embodiment provides a computer-readable storage medium, which includes instructions that, when executed on a computer, cause the computer to perform the lesion identification and classification method for endoscopic mucosal images described in any of the above embodiments.

[0173] It should be noted that the descriptions of each embodiment in the above embodiments have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0174] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, systems, apparatuses, and media according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.

[0175] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

Claims

1. A method for lesion identification and classification in endoscopic mucosal images, characterized in that, include: S1. Read the video signal output by the endoscope to obtain the mucosal image; S2. Calculate multiple quality feature values ​​of the mucosal image and determine the image quality of the mucosal image accordingly; S3. Predict the classification results of normal mucosal regions, unclear mucosal regions, abnormal mucosal regions, and abnormal mucosal regions in mucosal images that meet the preset image quality standards. S4. The predicted results are merged into a single image to generate a recognition classification map.

2. The lesion identification and classification method for endoscopic mucosal images as described in claim 1, characterized in that, The calculation of multiple quality feature values ​​of the mucosal image specifically involves: the first image quality detection module calculating the first quality feature value and the second quality feature value of the mucosal image; The second image quality detection module calculates the third quality feature value of the mucosal image, and the third image quality detection module calculates the fourth and fifth quality feature values ​​of the mucosal image. The first quality feature is the gradient mean, the second quality feature is the gradient variance, the third quality feature is the high-frequency energy, the fourth quality feature is the mean of the local variance, and the fifth quality feature is the variance of the local variance.

3. The lesion identification and classification method for endoscopic mucosal images as described in claim 2, characterized in that, The step of calculating multiple quality feature values ​​of the mucosal image includes: The first image quality detection module uses the Sobel gradient operator to calculate the gradient magnitudes in the horizontal and vertical directions of the mucosal image, synthesizes a gradient magnitude map, and calculates the mean and variance of the gradient magnitudes to obtain the gradient mean and gradient variance. The second image quality detection module performs Fourier transform on the mucosal image to obtain the frequency domain transformed image, and calculates the energy of the high-frequency components of the frequency domain transformed image to obtain the high-frequency energy. The third image quality detection module divides the mucosal image into multiple sub-image blocks, calculates the pixel value variance of each sub-image block, calculates the pixel value variance of each sub-image block as the local variance, and calculates the mean of the variances of all sub-image blocks and the variance of the sub-image block variances to obtain the mean of the local variances and the variance of the local variances.

4. The lesion identification and classification method for endoscopic mucosal images as described in claim 1, characterized in that, Specifically, S3 involves using a multi-task cascaded network to predict the classification results of normal mucosal regions, unclear mucosal regions, abnormal mucosal regions, and abnormal mucosal regions in mucosal images that meet preset image quality standards.

5. The lesion identification and classification method for endoscopic mucosal images as described in claim 4, characterized in that, The multi-task cascaded network includes a first network module for generating backbone feature maps, a second network module for generating coarse-grained candidate target regions, and a third network module for refining edge features of the candidate target regions to obtain classification results and segmentation masks. The third network module includes a first cascaded location prediction network module, a first cascaded mask prediction network module, a second cascaded location prediction network module, a second cascaded mask prediction network module, a third cascaded location prediction network module, and a third cascaded mask prediction network module arranged sequentially. The segmentation mask serves as the prediction result for normal mucosal regions, unclear mucosal regions, and abnormal mucosal regions in a mucosal image that meets a preset image quality standard. The classification result serves as the prediction result for the subtyping of abnormal mucosal regions.

6. The lesion identification and classification method for endoscopic mucosal images as described in claim 5, characterized in that, S3 specifically includes: The first network module generates multiple backbone feature maps; The second network module performs convolution calculations on the backbone feature map obtained by the first network module to obtain the intermediate feature map. The second network module calculates the foreground or background classification probability of the candidate region by convolution on the intermediate feature map, and calculates the bounding box regression parameters of the candidate region by convolution on the intermediate feature map. The second network module integrates the foreground or background classification probability of the candidate region and the bounding box regression parameters of the candidate region to obtain a coarse-grained candidate target region. The first-level location prediction network module generates preliminary classification results and preliminary bounding boxes based on coarse-grained candidate target regions and all backbone feature maps; The first-level mask prediction network module generates a preliminary segmentation mask by performing convolution and deconvolution calculations based on the preliminary classification results, preliminary bounding boxes, and all backbone feature maps. The second-level location prediction network module generates refined classification results and refined bounding boxes based on the preliminary classification results, preliminary bounding boxes, preliminary segmentation masks, and all backbone feature maps. The second-level cascaded mask prediction network module generates a fine segmentation mask by convolution and deconvolution based on the refined classification results, refined bounding boxes, and all backbone feature maps. The third-level location prediction network module generates the final classification result and the final bounding box through convolution and full convolution calculations based on the refined classification results, refined bounding boxes, fine segmentation masks and all backbone feature maps. The third-level concatenated mask prediction network module generates the final segmentation mask based on the final bounding box through deconvolution.

7. The lesion identification and classification method for endoscopic mucosal images as described in claim 6, characterized in that, S4 specifically includes: The final segmentation mask for each channel is then smoothed at the edges. Edge detection is performed on the final segmentation mask after edge contour smoothing. Using the edge detection results, extract the contour of each channel; The results from all channels are merged into a single image to generate a recognition and classification map.

8. A lesion identification and classification system for endoscopic mucosal images, characterized in that, include: The acquisition module is used to read the video signal output by the endoscope to obtain mucosal images; The calculation and judgment module is used to calculate multiple quality feature values ​​of the mucosal image and determine the image quality of the mucosal image accordingly. The prediction module is used to predict the classification results of normal mucosal regions, unclear mucosal regions, abnormal mucosal regions, and abnormal mucosal regions in mucosal images that meet preset image quality standards. The fusion module is used to combine the prediction results of the prediction module into a single image to generate a recognition classification map.

9. An electronic device, characterized in that, include: Memory; One or more processors; One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for performing any of the methods for lesion identification and classification of endoscopic mucosal images according to claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes instructions that, when executed on a computer, cause the computer to perform the lesion identification and classification method for endoscopic mucosal images as described in any one of claims 1 to 7.

Images