Intestinal polyp identification method, system and product based on identification model assistance

Abstract

The invention relates to the technical field of image processing, in particular to an intestinal polyp recognition method, system and product based on recognition model assistance, and the method comprises the following steps: S1, obtaining an image of the inner wall of an intestinal tract; s2, identifying and marking a target polyp range on the image through a polyp detection model; s3, inputting the image into a depth map prediction model to obtain a depth map of the target polyp; s4, obtaining edge information and real size information of the target polyp based on the target polyp range and the depth map of the target polyp; and S5, inputting the target polyp image information in the plurality of preset directions into the polyp form prediction model, outputting the form type of the polyp form, and calculating and outputting the volume of the polyp according to the form type of the polyp form. According to the application, accurate recognition, form classification and volume estimation of the polyp can be realized based on an artificial intelligence neural network computer-aided detection system through the intestinal endoscope image.

Description

Technical Field

[0001] This application relates to the field of image processing technology, specifically to a method, system, and product for identifying intestinal polyps based on a recognition model. Background Technology

[0002] Intestinal polyps are tissues that grow from the surface of the intestinal mucosa into the intestinal lumen. Whether a polyp is benign or malignant is directly related to its morphology. Generally, smaller polyps are considered benign with a lower risk of becoming cancerous, while larger polyps are considered malignant with a higher risk. The morphology and structure of polyps are closely related to a physician's clinical decisions, especially in assessing the risk of malignancy and determining the treatment plan.

[0003] Currently, both existing guidelines and clinical techniques for observing polyps using endoscopy primarily rely on two-dimensional dimensional data of the polyps, such as their length or diameter, to assess their growth status. Meanwhile, to assist physicians in accurately identifying polyps, computer-aided detection (CAD) systems are increasingly being used in clinical practice. These systems also rely on the approximate length or area of ​​the polyp for identification and measurement.

[0004] However, two-dimensional dimensional data of polyps alone may not fully reflect their complexity. Polyps with the same surface area can vary greatly in their morphology, height, and growth pattern, resulting in different actual volumes. Morphological and volume data are more indicative of the actual growth status and risk level of polyps than two-dimensional dimensional data. Therefore, achieving accurate assessment of polyp morphology and volume is a primary goal for technicians to better determine the nature and potential risks of polyps. Summary of the Invention

[0005] The purpose of this application is to overcome the shortcomings of the prior art and provide a method, system and product for identifying intestinal polyps based on a recognition model, which can accurately identify, classify and estimate the size of polyps by using intestinal endoscopy images and an artificial intelligence neural network computer-aided detection system.

[0006] Firstly, this application provides a method for identifying intestinal polyps based on a recognition model, the technical solution of which includes the following steps: S1, Obtain images of the intestinal wall through an intestinal endoscope; S2, through the polyp detection model, identifies specific areas of the image as target polyps and marks the range of the target polyps on the image; S3, input the image indicating the range of the target polyp into the depth map prediction model to obtain the depth map of the target polyp; S4, Based on the target polyp range and the depth map of the target polyp, obtain the polyp edge information and the true size information of the polyp corresponding to the current target polyp image; S5: Input the target polyp images in several preset directions, along with their corresponding polyp edge information and actual polyp size information, into the polyp morphology prediction model, output the polyp morphology type, determine the polyp volume calculation formula based on the polyp morphology type, determine the parameters of the polyp volume calculation formula based on the actual polyp size information, calculate and output the polyp volume.

[0007] By adopting the above technical solution, after acquiring an image of the intestinal wall, the presence of polyps is first determined and the range of polyps is marked by a polyp detection model; then, the image with the marked polyp range is converted into a depth map of the target polyp by a depth map prediction model, so as to obtain the depth distance of the polyp in three-dimensional space, thereby accurately determining the boundary range and true size of the polyp; then, the above information is input into a polyp morphology prediction model, so as to obtain the classification judgment of polyp morphology and the volume calculation results.

[0008] Preferably, the polyp detection model employs a DHNN neural network model; The specific method for training the polyp detection model is as follows: A polyp image dataset is obtained as both a training and validation set, wherein the images in the polyp image dataset include a pre-defined polyp region; All images in the training and validation sets are preprocessed into a uniform input format; Expand the training and validation sets through data augmentation; Through repeated training rounds, several alternative polyp detection models were obtained; Obtain a dataset of polyp images as a test set, input each candidate polyp detection model, and select the candidate model with the highest intersection-union ratio of the area of ​​the enclosed polyp range output by the test set to the area of ​​the polyp range in the test set. The polyp image dataset used as the training and validation set, and the polyp image dataset used as the test set, employ publicly available datasets.

[0009] By adopting the above technical solution, a computer-aided detection model can be trained to delineate the range of potential polyps based on intestinal wall endoscopy images.

[0010] Preferably, the depth map prediction model adopts an encoder-decoder neural network model. The encoder uses a ResNet-50 convolutional residual model to convert RGB images into feature vectors for feature extraction. The decoder uses DenseDepth to transform specific features to form the predicted output of the depth map. The specific method for training the depth map prediction model is as follows: A pre-trained model based on ImageNet was used as the starting point for training the depth map prediction model; Obtain intestinal RGB images with polyp images and depth annotations as the training and validation sets; All images in the training and validation sets are preprocessed into a uniform input format; Expand the training and validation sets through data augmentation; By repeatedly training, a depth map prediction model is obtained that converts RGB images into depth maps for output.

[0011] The above technical solution enables the training of a computer-aided detection model that outputs a depth map containing pixel depth information based on a defined polyp range.

[0012] As a preferred method, the specific method for obtaining intestinal RGB images with polyp images and polyp depth annotations is as follows: A CT scan-based 3D reconstruction method is used to convert 2D CT scan images of the intestine into a digital virtual 3D intestinal model. By adjusting the surface texture and internal paths of the digital virtual three-dimensional intestinal model and eliminating individual structural features, a general virtual three-dimensional intestinal model with general applicability is obtained. In a typical virtual 3D intestinal model scenario, the parameters of the camera model, camera field of view, and virtual light source of the virtual intestinal endoscope are set. In a typical virtual 3D intestinal model scenario, polyp module models with different polyp morphology types and sizes are constructed, and the polyp module models are randomly set in different positions within the typical virtual 3D intestinal model. A virtual intestinal endoscope with preset parameters was used to observe the polyp module model, and a virtual image of the polyp module model was obtained from the perspective of the virtual intestinal endoscope. The depth of each point on the virtual image to the plane where the camera model is located was obtained, and the depth was marked on the virtual image to obtain an RGB image of the intestine with polyp image and depth annotation.

[0013] The above technical solution constructs a digital virtual three-dimensional colorectal model through CT three-dimensional reconstruction, and builds a virtual intestinal endoscope and polyp module model within the model. It can accumulate virtual image data taken under different intestinal locations, environments, polyp morphologies, and lighting conditions. Furthermore, the virtual image data can be easily labeled with depth based on the three-dimensional model, thereby accumulating a large amount of training data for the depth map prediction model and improving the accuracy and generalization of the depth map prediction model.

[0014] As a preferred method, the specific method for obtaining the polyp edge information and the true size information of the current target polyp image based on the target polyp range and the depth map of the target polyp is as follows: within the target polyp range, when the rate of change of the depth of the pixel of the target polyp compared with the depth of the adjacent pixel exceeds a threshold, the current pixel is determined to be the polyp edge, the polyp image size is obtained based on the polyp edge, and the true size of the polyp is obtained based on the polyp image size and the polyp depth.

[0015] The above technical solution can obtain the true size of polyps based on intestinal RGB images with polyp images and depth annotations, thus accumulating parameters for subsequent polyp classification and volume calculation.

[0016] Preferably, the polyp morphology prediction model includes a feature extraction layer and a classification layer. The feature extraction layer adopts a CNN architecture neural network, which extracts the edge information and true size information of the input target polyp through several layers of convolution-activation-pooling modules connected in sequence. The classification layer is a fully connected layer that outputs the morphology type of the polyp based on all features. Specific methods for training polyp morphology prediction models include: When building the polyp module model, different types of polyp module models are built based on the morphological type of the polyp, and the polyp morphological type is labeled for each polyp module model; Obtain polyp module models labeled with polyp morphology types, obtain the size parameters of polyp module models in several preset directions, and divide the polyp module models into training, validation, and test sets; Expand the training and validation sets through data augmentation; By repeatedly training, a polyp morphology prediction model is obtained that classifies polyp morphology types based on polyp edge information and polyp real size information in several preset directions.

[0017] The above technical solution can be used to train a computer-aided detection model that classifies polyp morphology based on the actual size parameters of polyps in several preset directions.

[0018] Preferably, the polyp morphology includes flat polyps, raised pedunculated polyps, and raised sessile polyps; the geometric model of the flat polyp is a flat elliptical cylinder, the geometric model of the raised pedunculated polyp is an ellipsoid, and the geometric model of the raised sessile polyp is a semi-ellipsoid.

[0019] The above technical solution distinguishes and defines the specific morphological types of polyps, enabling polyps of different shapes to be classified into the above different morphological types through the polyp morphology prediction model. At the same time, it provides the geometric model corresponding to the polyp morphology type, so that the volume of different types of polyps can be approximated based on the geometric model.

[0020] Preferably, the preset direction includes at least two directions: the direction of the intestinal endoscope lens is perpendicular to the intestinal wall where the target polyp is located, and the direction of the intestinal endoscope lens is parallel to the intestinal wall where the target polyp is located.

[0021] By using the above technical solution, several preset lens directions of the intestinal endoscope are defined, thereby obtaining sufficient information on the actual size of the polyps from the polyp images captured by the limited lens directions. This information is used to determine the polyp morphology and to calculate the polyp volume.

[0022] Secondly, this application provides an intestinal polyp identification system based on an identification model, the technical solution of which includes an intestinal endoscopy module, an identification model module and a calculation module; The intestinal endoscope module is used to observe the inside of the intestine and obtain images of the intestinal wall; The identification model module includes a polyp detection model module, a depth map prediction model module, and a polyp morphology prediction model module. The polyp detection model module marks the range of target polyps on the intestinal wall image. The depth map prediction model module outputs the depth map of the target polyp based on the image with the marked range, and obtains the polyp edge information and the actual size information of the polyp corresponding to the current target polyp image based on the target polyp range and the depth map. The polyp morphology prediction model module outputs the morphology type of the polyp to the calculation module based on the polyp edge information and the actual size information. The calculation module determines the polyp volume calculation formula based on the polyp's actual size information and morphological type, and calculates and outputs the polyp's volume.

[0023] Thirdly, this application provides a computer program product, the technical solution of which includes a computer program or instructions, enabling the computer program or instructions to implement the steps in the above-mentioned method for identifying intestinal polyps based on a recognition model.

[0024] In summary, this application includes at least one of the following beneficial technical effects: 1. This application can automatically identify and mark the target polyp area based on the intestinal wall image taken by an intestinal endoscope. At the same time, based on the depth map of the marked target polyp area, it can more accurately perceive the edge of the target polyp and accurately obtain the true size of the target polyp.

[0025] 2. This application can obtain polyp edge information and polyp real size information in different directions based on the target polyp images obtained in several preset directions, and obtain the polyp morphology type after comprehensive analysis by inputting them into a neural network model.

[0026] 3. This application can calculate and output the volume of polyps based on the determination of the polyp morphology type and the actual size information of the polyps. Compared with the morphological analysis of polyps based solely on two-dimensional polyp data, it can provide more comprehensive and detailed polyp morphology evaluation data, which can effectively improve the rationality of polyp treatment decisions. Attached Figure Description

[0027] Figure 1 This is a schematic diagram of the training method for the polyp detection model in the embodiments of this application; Figure 2 This is a schematic diagram showing the relationship between the polyp image size and the actual polyp size in the embodiments of this application; Figure 3 This is a schematic diagram of the training method of the depth map prediction model in the embodiments of this application; Figure 4 This is a flowchart illustrating the method for obtaining an intestinal RGB image with polyp images and depth annotations in an embodiment of this application. Figure 5 This is a schematic diagram illustrating the classification of the three polyp types in the embodiments of this application; Figure 6 These are representative images of the three types of polyps in the embodiments of this application in a virtual three-dimensional intestinal model scene; Figure 7 In this embodiment of the application, the depth map prediction model converts an RGB image into a representative RGB image of the depth map; Figure 8 for Figure 7 A representative image converted into a depth map; Figure 9 This is a schematic diagram of the training method for the polyp morphology prediction model in the embodiments of this application; Figure 10 This is a flowchart illustrating a method for identifying intestinal polyps based on a recognition model according to an embodiment of this application. Figure 11This is a schematic diagram of the architecture of an intestinal polyp identification system based on a recognition model according to an embodiment of this application. Detailed Implementation

[0028] This specific embodiment is merely an explanation of this application and is not intended to limit it. After reading this specification, those skilled in the art can make modifications to this embodiment without contributing any inventive step, but such modifications are protected by patent law as long as they are within the scope of this application.

[0029] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. It should be noted that in the optional embodiments of this application, the object information and other related data involved require the permission or consent of the object when the embodiments of this application are applied to specific products or technologies, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. That is to say, if the embodiments of this application involve data related to the object, it needs to be obtained with the authorization and consent of the object, the authorization and consent of the relevant departments, and in compliance with the relevant laws, regulations, and standards of the country and region. If personal information is involved in the embodiments, the acquisition of all personal information requires the consent of the individual. If sensitive information is involved, the separate consent of the information subject is required, and the embodiments also need to be implemented with the authorization and consent of the object.

[0030] The embodiments of this application will now be described in further detail with reference to the accompanying drawings.

[0031] This application discloses a model-assisted intestinal polyp identification method to determine the morphology of intestinal polyps and estimate their volume. Compared to existing methods that assess polyp status by obtaining two-dimensional dimensions such as length or diameter, the method of this application expands the information dimensions of the polyps, improving the accuracy and reliability of assessment and risk judgment decisions.

[0032] The intestinal polyp identification method of this application is implemented based on computer-aided detection (CAD) and applies three types of artificial intelligence neural network models: a polyp detection model, a depth map prediction model, and a polyp morphology prediction model. The functions and training methods of each artificial intelligence neural network model are explained below.

[0033] The polyp detection model identifies polyps in images of the intestinal wall captured by an endoscope. After locating the polyp, it marks the area of ​​the identified polyp with a red frame on the image. This marking can be done in real-time on the endoscope image to provide physicians with real-time indications of the potential polyp area. Based on this indication, physicians can further observe the marked area by manipulating the endoscope to improve the accuracy of polyp diagnosis, or selectively ignore false positives from the system.

[0034] In the embodiments of this application, the polyp detection model employs a DHNN neural network model. The DHNN neural network model uses both linear and nonlinear algorithms, combining convolutional layers and mathematical morphology layers, enabling parallel generation of convolutional and morphological architectures to extract linear and nonlinear features. These features are then combined to label the extent of the polyp. Please refer to [link to relevant documentation]. Figure 1 The specific training method for the polyp detection model is as follows.

[0035] S101, Obtain a polyp image dataset as a training set and a validation set, wherein the images in the polyp image dataset include a pre-defined polyp range.

[0036] The polyp image dataset uses publicly available data from the internet. Specifically, the polyp image datasets used as the training and validation sets are Kvasir-SEG, CVC-ClinicDB, CVC-ColonDB, CVC-300, ETIS-LaribPolypDB, EndoCV2022, and EDD2020. The polyp image dataset used as the test set is PICCOL. All datasets include images and bounding boxes indicating the extent of the polyps.

[0037] S102, all images in the training and validation sets are preprocessed into a unified input format. Specifically, in order to ensure that the image dimensions of each model are matched, in the embodiments of this application, the input image size of all models is uniformly adjusted to 640×480 pixels.

[0038] S103 expands the training and validation sets with data augmentation, including horizontal / vertical flipping, position translation in the height / width direction, brightness variation, addition of noise, and contrast enhancement.

[0039] S104, through repeated training rounds, several candidate polyp detection models are obtained. In the embodiments of this application, each candidate polyp detection model is obtained through 100 training rounds.

[0040] S105, Obtain the polyp image dataset as the test set, input it into each candidate polyp detection model, and obtain different marked polyp ranges output by each candidate polyp detection model. Select the model with the highest intersection-union ratio (IUU) between the marked polyp range and the polyp range area in the test set as the final polyp detection model.

[0041] In the embodiments of this application, the depth map prediction model acquires an image marked with polyp range and outputs the depth distance information of each pixel in the polyp range in three-dimensional space with respect to the endoscope lens. Specifically, in the image marked with polyp range, if the depth of a certain pixel changes abruptly and the rate of change increases sharply, it indicates that the pixel is a pixel on the edge of the polyp. Thus, the edge range of the polyp and the size information of the polyp in the image can be obtained through the depth distance.

[0042] The depth map prediction model and the polyp detection model work together. The polyp detection model first identifies and locates the polyp, providing its two-dimensional location information. Then, the depth map prediction model obtains the polyp's depth distance in three-dimensional space, thus accurately identifying the polyp's edges. The combination of the two models improves the accuracy of polyp localization, and the synergy between the models also enables efficient and accurate measurement of the polyp's true size. Please refer to [link / reference]. Figure 2 O1 is the actual polyp size, and O2 is the polyp size in the image. The actual polyp size O1 can be calculated using the polyp size O2 in the image, the depth information L1, and the focal length L2.

[0043] In the embodiments of this application, the depth map prediction model employs an encoder-decoder neural network model. The encoder is responsible for feature extraction, using a ResNet-50 convolutional residual model containing 50 convolutional layers and residual blocks to address the gradient vanishing and gradient augmentation problems during deep network training. The ResNet-50 contains multiple residual blocks, each containing multiple convolutional layers and skip connections, which facilitates information flow throughout the network. The decoder uses DenseDepth, which includes a 2x bilinear upsampling step and two standard convolutional layers to transform specific features into the predicted depth map output. The decoder works in conjunction with the encoder to convert RGB images into feature vectors and generate high-resolution depth maps. For more details, please refer to [link to relevant documentation]. Figure 3 The specific training method for the depth map prediction model is as follows.

[0044] S201 uses a pre-trained model based on ImageNet as the starting point for training the depth map prediction model.

[0045] S202. Obtain intestinal RGB images with polyp images and depth annotations as training and validation sets. The specific acquisition method will be explained in detail later.

[0046] S203, preprocess all images of the training and validation sets into the aforementioned 640×480 pixel image format.

[0047] S204 augments the training and validation sets with data augmentations, including horizontal / vertical flipping, vertical / width position translation, brightness variation, noise addition, and contrast enhancement.

[0048] S205, through repeated training rounds, obtains a depth map prediction model that converts RGB images into depth maps for output.

[0049] To train a depth map prediction model, the images in the training dataset must contain depth information for each pixel in every image. Therefore, the pixels in the images of the training, validation, and test sets must contain labels with depth data. Current real colonoscopes typically use only one lens, making it difficult to simultaneously acquire colonoscopy images and their corresponding true depth information. In the embodiments of this application, CT 3D reconstruction technology is used to create a realistic model for training the depth map model, thereby obtaining a large amount of accurate training data to predict the depth distance of polyps.

[0050] In the embodiments of this application, a workspace for creating a 3D model simulation environment is established within the Unity 3D creation platform. Within this workspace, the intestines, polyps, and camera are modeled. Through modeling, corresponding RGB images of the intestines and annotations of depth information are obtained, thereby acquiring a depth map prediction model dataset. For more details, please refer to... Figure 4 The specific method for obtaining the intestinal RGB images with polyp images and depth annotations for the depth map prediction model dataset is as follows.

[0051] S2021 employs CT scan-based 3D reconstruction to convert 2D CT scan images of the intestine into a digital virtual 3D intestinal model.

[0052] S2022, the surface texture and internal paths of the digital virtual 3D intestinal model were adjusted to eliminate individual structural features, resulting in a general virtual 3D intestinal model with general applicability. Then, it was imported into Unity to create a workspace for a 3D model simulation environment.

[0053] S2023 utilizes Unity's 3D editing tools and scene editor to set parameters for the camera model, camera field of view, and virtual light source of a virtual intestinal endoscope within a typical virtual 3D intestinal model scene. As a specific implementation method, the camera model parameters of the virtual intestinal endoscope can be set with reference to the parameters of certain Olympus electronic colonoscope models.

[0054] S2024 involves constructing polyp module models with different polyp morphologies and sizes within a typical virtual 3D intestinal model. These polyp module models are then randomly positioned within the virtual 3D intestinal model. These different positions not only simulate various colorectal environments (such as colorectal bends, folds, and relatively straight sections) but also create rich combinations of conditions with different polyp morphologies and virtual lighting. This allows for the acquisition of datasets under different conditional combinations, thereby expanding the dataset's dimensionality and improving the generalization ability of the trained model.

[0055] S2025, a virtual intestinal endoscope camera model with preset parameters is used to observe the polyp module model, and a virtual image of the polyp module model is obtained from the perspective of the virtual intestinal endoscope. The depth of each point on the virtual image to the plane where the camera model is located is obtained, and the depth is marked on the virtual image, thus obtaining an intestinal RGB image with polyp image and depth annotation.

[0056] It should also be noted that constructing different polyp morphology types in S2024 requires implementation according to the pre-classification of polyp morphology types in this application. Traditional polyp classification methods subdivide polyps into various types, such as superficial flat, superficial raised, sessile, subpedunculated, and pedunculated, etc. While this provides more detailed polyp analysis criteria, it can easily increase the false alarm rate in automated systems, affecting the accuracy of polyp classification. To improve the model's stability and generalization ability, reduce the risk of misclassification, and ensure the accuracy of volume estimation, please refer to [link to relevant documentation]. Figure 5 We simplify the polyp morphology into three types: Type A, raised pedunculated polyp; Type B, raised sessile polyp; and Type C, flat polyp. The geometric model of Type A, raised pedunculated polyp, is an ellipsoid (ignoring the stalk), and its volume formula is: The geometric model of a Type B sessile polyp is a semi-ellipsoid, and its volume formula is: The geometric model of Type C flat polyps is an elliptical cylinder, and its volume formula is: Representative images of the three types of polyps in a virtual 3D intestinal model are shown below. Figure 6 As shown.

[0057] The depth map prediction model converts an RGB image into a representative image of the depth map, such as... Figure 7 and Figure 8 As shown.

[0058] In the embodiments of this application, the polyp morphology prediction model classifies and outputs polyp morphology types based on depth maps in several preset directions obtained from the depth map prediction model, as well as the corresponding polyp edge information and actual polyp size information. Simultaneously, it outputs the fitting results of various size parameters of the polyp under that classification, enabling polyp volume calculation. It should be noted that since polyps often exhibit irregular shapes, the parameters in the polyp volume calculation formula are parameter estimates obtained by the polyp morphology prediction model based on the actual polyp size information.

[0059] The aforementioned preset directions include at least two directions: one where the direction of the intestinal endoscope lens is perpendicular to the intestinal wall where the target polyp is located, and another where the direction of the intestinal endoscope lens is parallel to the intestinal wall where the target polyp is located. This is to comprehensively reflect the morphological characteristics of the polyp and provide a basis for fitting size parameters. Increasing the number of images from different viewing directions can improve the accuracy of classification and size parameter fitting, and can provide more images from different directions as input data for model prediction. Actual growing polyps may exhibit irregular appearances to some extent, and the polyp morphology prediction model aims to classify polyps into morphologies that most closely resemble the actual polyp type, so that the volume calculation is closest to reality.

[0060] The polyp morphology prediction model includes a feature extraction layer and a classification layer. The feature extraction layer uses a CNN neural network architecture, extracting edge information and true size information of the input target polyp through several sequentially connected convolutional-activation-pooling modules. The classification layer is a fully connected layer that outputs the morphology type of the polyp based on all features. Please refer to [link to relevant documentation]. Figure 9 The specific method for training the polyp morphology prediction model is as follows.

[0061] S301: Obtain polyp module models with polyp morphology type annotations; obtain the size parameters of the polyp module models in several preset directions; and divide the polyp module models into training, validation, and test sets. Images need to be processed into a standard image format of 640×480 pixels.

[0062] S302 augments the training and validation sets with data augmentations, including horizontal / vertical flipping, vertical / width position translation, brightness variation, noise addition, and contrast enhancement.

[0063] S303 is a polyp morphology prediction model that, through repeated training rounds, obtains polyp edge information and polyp real size information based on several preset directions, and classifies polyp morphology types.

[0064] The dataset used to train the polyp morphology prediction model comes from the workspace of the 3D model simulation environment created in Unity. Existing model modules set up during the training of the depth map prediction model can be used for training to improve efficiency.

[0065] Please see Figure 10 The specific implementation of a method for identifying intestinal polyps based on an identification model according to an embodiment of the present invention is as follows.

[0066] S1, using an intestinal endoscope to obtain images of the intestinal wall.

[0067] S2, using a polyp detection model, identifies specific areas in the image as target polyps and marks the target polyp area on the image. A physician then uses an intestinal endoscope to observe and verify from multiple angles to determine the accuracy of the target polyp detection. Several images of the marked target polyp area are then taken in preset directions for the verified area.

[0068] S3. Input the captured image indicating the range of the target polyp into the depth map prediction model to obtain the depth map of the target polyp.

[0069] S4. Based on the target polyp range and the depth map of the target polyp, obtain the polyp edge information and the true size information of the polyp corresponding to the current target polyp image. Specifically, within the target polyp range, when the rate of change of the depth of a pixel of the target polyp compared with the depth of its adjacent pixels exceeds a threshold, the current pixel is determined to be the polyp edge. The polyp image size is obtained based on the polyp edge, and the true size of the polyp is obtained based on the polyp image size and the polyp depth.

[0070] S5: Several target polyp images along with their corresponding polyp edge information and actual polyp size information from preset directions are input into the polyp morphology prediction model. The model outputs the polyp morphology type. Based on the polyp morphology type, a formula for calculating the polyp volume is determined. The parameters of the polyp volume calculation formula are determined based on the actual polyp size information. The volume of the polyp is then calculated and output. It should be noted that since polyps often exhibit irregular shapes, the parameters in the polyp volume calculation formula are parameter estimates obtained by fitting the polyp morphology prediction model based on the actual polyp size information.

[0071] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0072] Please see Figure 11Another embodiment of this application provides an intestinal polyp identification system based on an identification model, including an intestinal endoscopy module 1, an identification model module 2, and a calculation module 3.

[0073] The intestinal endoscope module 1 is used to observe the inside of the intestine and obtain images of the intestinal wall.

[0074] The identification model module 2 includes a polyp detection model module 21, a depth map prediction model module 22, and a polyp morphology prediction model module 23. The polyp detection model module 21 marks the range of target polyps on the intestinal wall image. The depth map prediction model module 22 outputs a depth map of the target polyp based on the image with the marked range, and obtains the polyp edge information and actual size information corresponding to the current target polyp image based on the target polyp range and the depth map. The polyp morphology prediction model module 23 outputs the polyp morphology type to the calculation module based on the polyp edge information and actual size information.

[0075] The calculation module 3 determines the polyp volume calculation formula based on the polyp's actual size information and morphological type, and calculates and outputs the polyp's volume.

[0076] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the intestinal polyp identification system based on the identification model described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0077] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0078] Another embodiment of this application provides a computer program product comprising a computer program or instructions that enables the computer program or instructions to perform the steps in the above-described method for identifying intestinal polyps based on a recognition model.

[0079] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for identifying intestinal polyps based on a recognition model, characterized in that, Includes the following steps: S1, Obtain images of the intestinal wall through an intestinal endoscope; S2, through the polyp detection model, identifies specific areas of the image as target polyps and marks the range of the target polyps on the image; S3, input the image indicating the range of the target polyp into the depth map prediction model to obtain the depth map of the target polyp; S4, Based on the target polyp range and the depth map of the target polyp, obtain the polyp edge information and the true size information of the polyp corresponding to the current target polyp image; S5: Input the target polyp images in several preset directions, along with their corresponding polyp edge information and actual polyp size information, into the polyp morphology prediction model, output the polyp morphology type, determine the polyp volume calculation formula based on the polyp morphology type, determine the parameters of the polyp volume calculation formula based on the actual polyp size information, calculate and output the polyp volume.

2. The method for identifying intestinal polyps based on a recognition model according to claim 1, characterized in that, The polyp detection model uses a DHNN neural network model; The specific method for training the polyp detection model is as follows: A polyp image dataset is obtained as both a training and validation set, wherein the images in the polyp image dataset include a pre-defined polyp region; All images in the training and validation sets are preprocessed into a uniform input format; Expand the training and validation sets through data augmentation; Through repeated training rounds, several alternative polyp detection models were obtained; Obtain a dataset of polyp images as a test set, input each candidate polyp detection model, and select the candidate model with the highest intersection-union ratio of the area of ​​the enclosed polyp range output by the test set to the area of ​​the polyp range in the test set. The polyp image dataset used as the training and validation set, and the polyp image dataset used as the test set, employ publicly available datasets.

3. The method for identifying intestinal polyps based on a recognition model according to claim 2, characterized in that, The depth map prediction model adopts an encoder-decoder neural network model. The encoder uses a ResNet-50 convolutional residual model to convert RGB images into feature vectors for feature extraction. The decoder uses DenseDepth to transform specific features to form the predicted output of the depth map. The specific method for training the depth map prediction model is as follows: A pre-trained model based on ImageNet was used as the starting point for training the depth map prediction model; Obtain intestinal RGB images with polyp images and depth annotations as the training and validation sets; All images in the training and validation sets are preprocessed into a uniform input format; Expand the training and validation sets through data augmentation; By repeatedly training, a depth map prediction model is obtained that converts RGB images into depth maps for output.

4. The method for identifying intestinal polyps based on a recognition model according to claim 3, characterized in that, The specific method for obtaining RGB images of the intestine with polyp images and polyp depth annotations is as follows: A CT scan-based 3D reconstruction method is used to convert 2D CT scan images of the intestine into a digital virtual 3D intestinal model. By adjusting the surface texture and internal paths of the digital virtual three-dimensional intestinal model and eliminating individual structural features, a general virtual three-dimensional intestinal model with general applicability was obtained. In a typical virtual 3D intestinal model scenario, the parameters of the camera model, camera field of view, and virtual light source of the virtual intestinal endoscope are set. In a typical virtual 3D intestinal model scenario, polyp module models with different polyp morphology types and sizes are constructed, and the polyp module models are randomly set in different positions within the typical virtual 3D intestinal model. A virtual intestinal endoscope with preset parameters was used to observe the polyp module model, and a virtual image of the polyp module model was obtained from the perspective of the virtual intestinal endoscope. The depth of each point on the virtual image to the plane where the camera model is located was obtained, and the depth was marked on the virtual image to obtain an RGB image of the intestine with polyp image and depth annotation.

5. The method for identifying intestinal polyps based on a recognition model according to claim 1, characterized in that, In S4, the specific method for obtaining the polyp edge information and the true size information of the current target polyp image based on the target polyp range and the depth map of the target polyp is as follows: within the target polyp range, when the rate of change of the depth of the target polyp pixel compared with the depth of the adjacent pixel exceeds a threshold, the current pixel is determined to be the polyp edge. The polyp image size is obtained based on the polyp edge, and the true size of the polyp is obtained based on the polyp image size and the polyp depth.

6. The method for identifying intestinal polyps based on a recognition model according to claim 4, characterized in that, The polyp morphology prediction model includes a feature extraction layer and a classification layer. The feature extraction layer adopts a CNN architecture neural network, which extracts the edge information and true size information of the input target polyp through several layers of convolution-activation-pooling modules connected in sequence. The classification layer is a fully connected layer that outputs the morphology type of the polyp based on all features. Specific methods for training polyp morphology prediction models include: When building the polyp module model, different types of polyp module models are built based on the morphological type of the polyp, and the polyp morphological type is labeled for each polyp module model; Obtain polyp module models labeled with polyp morphology types, obtain the size parameters of polyp module models in several preset directions, and divide the polyp module models into training, validation, and test sets; Expand the training and validation sets through data augmentation; By repeatedly training, a polyp morphology prediction model is obtained, which classifies polyp morphology types based on polyp edge information and polyp true size information in several preset directions.

7. The method for identifying intestinal polyps based on a recognition model according to claim 6, characterized in that, The morphological types of the polyps include flat polyps, raised pedunculated polyps, and raised sessile polyps; the geometric model of the flat polyps is a flat elliptical cylinder, the geometric model of the raised pedunculated polyps is an ellipsoid, and the geometric model of the raised sessile polyps is a semi-ellipsoid.

8. The method for identifying intestinal polyps based on a recognition model according to claim 6, characterized in that, The preset direction includes at least two directions: the direction of the intestinal endoscope lens is perpendicular to the intestinal wall where the target polyp is located, and the direction of the intestinal endoscope lens is parallel to the intestinal wall where the target polyp is located.

9. A recognition model-assisted intestinal polyp identification system, characterized in that, It includes an intestinal endoscopy module, a recognition model module, and a computing module; The intestinal endoscope module is used to observe the inside of the intestine and obtain images of the intestinal wall; The identification model module includes a polyp detection model module, a depth map prediction model module, and a polyp morphology prediction model module. The polyp detection model module marks the range of the target polyp on the intestinal wall image. The depth map prediction model module outputs the depth map of the target polyp based on the image with the marked target polyp range, and obtains the polyp edge information and the true size information of the polyp corresponding to the current target polyp image based on the target polyp range and the depth map of the target polyp. The polyp morphology prediction model module outputs the morphology type of the polyp to the calculation module based on the polyp edge information and the actual size information of the polyp; The calculation module determines the polyp volume calculation formula based on the polyp's actual size information and morphological type, and calculates and outputs the polyp's volume.

10. A computer program product, characterized in that, The computer program product includes a computer program or instructions that enable the computer program or instructions to perform the steps in the intestinal polyp identification method based on a recognition model assisted by any one of claims 1 to 8.

Images