Colonoscopy image recognition and locating method and apparatus, and product

Abstract

A colonoscopy image recognition and locating method and apparatus, and a product. The method comprises: obtaining an intestinal endoscopic video signal, and accordingly generating an original image; obtaining patient identity information; on the basis of the patient identity information and the original image, retrieving from a colonic vascular matrix vector library a colon haustrum number and a vascular network matrix vector code corresponding to the patient identity information, and determining the number of the taeniae coli; if the colon haustrum number and the vascular network matrix vector code are not found, identifying the taeniae coli in the original image, and numbering the taeniae coli; determining whether the original image includes a colon haustrum region, and if so, performing semantic segmentation on a vascular network region in the colon haustrum region to obtain a vascular network mask map; performing feature extraction on the vascular network mask map to obtain the vascular network matrix vector code; and populating the colonic vascular matrix vector library. The colonoscopy image recognition and locating method and apparatus, and the product realize consistent numbering during different colonoscopy examinations, avoid repeated labeling, and provide low‑cost consistent recognition.

Description

Intestinoscopy image recognition positioning method, device and product

[0001] Cross-reference statement

[0002] The present application claims priority to the Chinese patent application No. 202411230061.7, filed on September 4, 2024, and entitled "Intestinoscopy image recognition positioning method, device and product", the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0003] The present application relates to the technical field of intestinoscopy image processing, in particular to an intestinoscopy image recognition positioning method, device and product. BACKGROUND

[0004] A colonoscope is a kind of fiber endoscope commonly used in clinical practice. It can be inserted through the anus and examined to the rectum, sigmoid colon, descending colon, transverse colon, ascending colon, cecum and a small section of intestine connected to the large intestine (terminal ileum) in a reverse direction. It can clearly reveal intestinal lesions and also treat some intestinal lesions.

[0005] In current colonoscopy, convolutional neural networks (CNN) and other algorithms are usually used to enhance the blood vessel network or other structures in the endoscope image, helping doctors to more clearly identify lesions. For the marking of the taenia coli, doctors usually manually mark the taenia coli. Due to different operation habits of each doctor, the results of manual marking often lack consistency. It is difficult for the same doctor to ensure the use of the same marking method between different detection operations, and it is difficult to ensure the consistency of this marking and historical marking. This leads to the inability to accurately match the previous marking position in subsequent examinations, and the repeated marking operation wastes time.

[0006] The inconsistency of the above-mentioned marking may lead to inaccurate lesion positioning between multiple examinations, affecting the effect of diagnosis and treatment. In some advanced endoscopy systems, magnetic field positioning, CT / MRI image registration and other technologies are used to help doctors with colon positioning. However, these systems require the use of various sensors and other devices, resulting in high system prices and limited application range.

[0007] Therefore, there is an urgent need to develop an intestinoscopy image recognition positioning method and system that is low in cost, consistent in front and back marking, and does not repeat the marking operation. SUMMARY

[0008] In view of the above problems, the present application provides an intestinoscopy image recognition positioning method, device and product.

[0009] The technical solution adopted by the present application to solve the technical problems is as follows:

[0010] In a first aspect, the present application provides a method for recognizing and positioning an enteroscopy image, comprising:

[0011] obtaining an enteroscopy video signal and generating an original image therefrom;

[0012] obtaining patient identity information;

[0013] retrieving a colon bag number and a vascular net matrix vector code corresponding to the patient identity information in a colon vascular matrix vector library according to the patient identity information and the original image, and determining the number of colon bands; if not retrieved, recognizing the colon bands in the original image and numbering the colon bands;

[0014] determining whether the original image contains a colon bag region, and if so, performing semantic segmentation on the vascular net region in the colon bag region to obtain a vascular net mask image;

[0015] performing feature extraction on the vascular net mask image to obtain a vascular net matrix vector code;

[0016] filling the colon vascular matrix vector library.

[0017] In a preferred embodiment, the retrieval of a colon bag number and a vascular net matrix vector code corresponding to the patient identity information in the colon vascular matrix vector library according to the patient identity information and the original image specifically comprises:

[0018] retrieving the positioning information of the patient identity information in the colon vascular matrix vector library;

[0019] based on the positioning information, retrieving the colon bag number in the colon vascular matrix vector library that best matches the original image;

[0020] retrieving the vascular net matrix vector code in the best matching colon bag number that best matches the original image.

[0021] In a preferred embodiment, the retrieval of a colon bag number and a vascular net matrix vector code corresponding to the patient identity information in the colon vascular matrix vector library according to the patient identity information and the original image specifically comprises:

[0022] The retrieval of the vascular net matrix vector code in the best matching colon bag number that best matches the original image specifically comprises: through iterative optimization of the search path, gradually finding the vascular net matrix vector code in the best matching colon bag number that best matches the original image.

[0023] In a preferred embodiment, the recognition of the colon bands in the original image and the numbering of the colon bands specifically comprises:

[0024] normalizing the original image;

[0025] extracting edge features and texture features of the original image after normalization to obtain a first colon band feature map;

[0026] extracting morphological features of the colon band in the first colon band feature map to obtain a second colon band feature map;

[0027] extracting global context information and spatial relationship between the colon band and surrounding tissues in the second colon band feature map to obtain a third colon band feature map;

[0028] fusing the first colon band feature map, the second colon band feature map, and the third colon band feature map to generate a first multi-scale feature map;

[0029] applying an attention mechanism to the first multi-scale feature map for weighted processing to generate a first weighted feature map;

[0030] compressing the first weighted feature map to generate a numerical vector;

[0031] identifying the colon band according to the numerical vector;

[0032] numbering the identified colon band.

[0033] In a preferred embodiment, the numbering of the identified colon band specifically includes: numbering the identified colon band according to the moving line of the endoscope withdrawal, numbering the colon pouch according to the number of the colon band, and determining the number of the colon band according to the numbered colon pouch.

[0034] In a preferred embodiment, the determining whether the original image contains a colon pouch region and performing semantic segmentation on the blood vessel network region in the colon pouch region to obtain a blood vessel network mask map specifically includes:

[0035] determining whether the original image contains a colon pouch region, and if it does, determining a colon pouch image according to the original image;

[0036] normalizing the colon pouch image;

[0037] extracting edge and detail information of the normalized colon pouch image to obtain a first feature map, extracting texture and local shape information of the normalized colon pouch image to obtain a second feature map, and extracting overall morphological information of the colon pouch in the normalized colon pouch image to obtain a third feature map;

[0038] The first feature map is hierarchically aggregated to form a first hierarchical feature map; the second feature map is fused with the first hierarchical feature map to generate a second hierarchical feature map; and the third feature map is fused with the second hierarchical feature map to generate a third hierarchical feature map.

[0039] The first hierarchical feature map, the second hierarchical feature map and the third hierarchical feature map are fused by using a multi-scale feature fusion method to obtain a first multi-scale fusion feature map.

[0040] The global information and context relationship in the first multi-scale fusion feature map are captured, and an attention mechanism is applied to the spatial dimension and the channel dimension to generate a first multi-scale weighted feature map.

[0041] The first multi-scale weighted feature map is converted into a blood vessel network mask map.

[0042] In a preferred embodiment, the feature extraction on the blood vessel network mask map to obtain the blood vessel network matrix vector encoding includes: dividing the blood vessel network mask map into a plurality of grid cells, extracting features of each grid cell and encoding the extracted features into a vector, and combining the vectors of all grid cells into a matrix, which is taken as the blood vessel network matrix vector encoding.

[0043] In a preferred embodiment, the feature extraction on the blood vessel network mask map to obtain the blood vessel network matrix vector encoding specifically includes: dividing the blood vessel network mask map into grid cells according to an N×N grid; extracting geometric center features, blood vessel direction features and blood vessel complexity features for each grid cell, and encoding the extracted geometric center features, blood vessel direction features and blood vessel complexity features into a vector; combining the vectors of all grid cells into an N×N matrix according to the arrangement order of the grid cells in the blood vessel network mask map, extracting a global feature vector of the blood vessel network according to the N×N matrix, and taking the N×N matrix and the global feature vector together as the blood vessel network matrix vector encoding.

[0044] In a second aspect, the present application provides a device for recognizing and positioning a colonoscopy image, comprising:

[0045] A video acquisition module is configured to obtain an endoscopic video signal of an intestinal tract and generate an original image based on the endoscopic video signal;

[0046] A colon bag blood vessel network segmentation module is configured to determine whether the original image contains a colon bag region, and perform semantic segmentation on a blood vessel network region in the colon bag region to obtain a blood vessel network mask map when the original image contains the colon bag region.

[0047] A blood vessel network feature encoding module is configured to perform feature extraction on the blood vessel network mask map to obtain a blood vessel network matrix vector encoding.

[0048] The database filling and searching module is configured to obtain patient identity information, search for a colon bag number and a blood vessel net matrix vector code corresponding to the patient identity information in a colon blood vessel matrix vector library according to the patient identity information and the original image, and fill the colon blood vessel matrix vector library.

[0049] The colon band identification and numbering module is configured to identify and number a colon band in the original image when the colon bag number and the blood vessel net matrix vector code corresponding to the patient identity information are not searched for in the colon blood vessel matrix vector library, and determine the number of the colon band according to the colon bag number and the blood vessel net matrix vector code in the colon blood vessel matrix vector library.

[0050] In a third aspect, the present application provides a computer program product, comprising a computer program, wherein the computer program is executed by a processor to implement the method for identifying and positioning a colonoscopy image according to the first aspect.

[0051] The method, system and product for identifying and positioning a colonoscopy image according to the present application enrich the colon blood vessel matrix vector library by performing semantic segmentation on a blood vessel net region in a colon bag region to obtain a blood vessel net mask graph, performing feature extraction on the blood vessel net mask graph to obtain a blood vessel net matrix vector code, and then filling the colon blood vessel matrix vector library. The historical colon information is obtained by searching the colon blood vessel matrix vector library according to the patient identity information and the original image, and the number of the colon band is obtained, which ensures the consistency of the number in different colonoscopy detections and does not require repeated labeling. In addition, the cost is low because an external endoscope system is not required. BRIEF DESCRIPTION OF DRAWINGS

[0052] Fig. 1 is a flowchart of a method for identifying and positioning a colonoscopy image according to an embodiment.

[0053] Fig. 2 is a framework diagram of a system for identifying and positioning a colonoscopy image according to an embodiment.

[0054] Fig. 3 is a framework diagram of a colon band identification and numbering module according to an embodiment.

[0055] Fig. 4 is a framework diagram of a colon bag blood vessel net segmentation module according to an embodiment.

[0056] Fig. 5 is a functional flowchart of a blood vessel net feature encoding module and a database filling and searching module according to an embodiment. DETAILED DESCRIPTION

[0057] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0058] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. However, it will be recognized by one skilled in the art that the present application can be practiced without the specific details and, therefore, the scope of the present application is not limited to the details disclosed herein.

[0059] It should be noted that the terms "first", "second", and the like in the description are used for descriptive purposes only and should not be construed as indicating or implying relative importance or an indicated number of technical features. Therefore, the features defined as "first", "second" can explicitly or implicitly include at least one of the features. In addition, the technical solutions of various embodiments can be combined with each other, but it must be based on the realization of ordinary skilled in the art, when the combination of technical solutions appears contradictory or unachievable, it should be considered that the combination of technical solutions does not exist, nor within the scope of protection claimed in the present application.

[0060] Colonoscopy is the main tool for colorectal cancer screening and diagnosis, but due to the complex anatomy of the colon and individual differences, it is difficult for doctors to accurately locate the position of the lesion between different examinations. The traditional method relies on the experience and subjective judgment of doctors, and there is a large error. This error may lead to inaccurate positioning of the lesion between multiple examinations, affecting the effect of diagnosis and treatment. If combined with today's positioning system, due to the high cost of the system, it will lead to high price of colonoscopy, and the positioning system equipment is complex, the application is limited. Therefore, an automatic and standardized method is needed to help doctors accurately identify and locate lesions in multiple examinations. For this purpose, the present application provides an identification and positioning method, device and product of colonoscopy image. The following embodiments will be described in detail.

[0061] Embodiment one

[0062] Referring to FIG. 1, the present embodiment provides an identification and positioning method of colonoscopy image, comprising:

[0063] Obtaining an endoscopic video signal in the intestinal tract and generating an original image therefrom;

[0064] Obtaining patient identity information;

[0065] Retrieving the colon bag number and blood vessel matrix vector code corresponding to the patient identity information in the colon blood vessel matrix vector library according to the patient identity information and the original image; if the colon bag number and blood vessel matrix vector code corresponding to the patient identity information in the colon blood vessel matrix vector library are retrieved, the number of the colon band is determined, if not retrieved, the colon band in the original image is identified and numbered, and the number of the colon bag is determined according to the number of the colon band;

[0066] determining whether the original image contains a colon bag region, and if so, performing semantic segmentation on a blood vessel network region in the colon bag region to obtain a blood vessel network mask image;

[0067] performing feature extraction on the blood vessel network mask image to obtain blood vessel network matrix vector encoding;

[0068] filling a colon blood vessel matrix vector library.

[0069] The colon blood vessel matrix vector library includes patient identity information, colon bag number, and blood vessel network matrix vector encoding.

[0070] It should be understood that the above method can be different according to the execution process, for example, the order of obtaining the enteroscopy video signal and generating the original image and obtaining the patient identity information is not limited, so the above method does not represent or imply that all steps must be executed in this order, and a person skilled in the art can change or change the execution order of the above steps on the basis of the present application, and some embodiments of the above method are exemplified below.

[0071] Embodiment two

[0072] Please refer to FIG. 2, the present embodiment provides an enteroscopy image recognition and positioning device, the device comprises:

[0073] a video acquisition module 10 for obtaining an enteroscopy video signal and generating an original image therefrom;

[0074] a colon bag blood vessel network segmentation module 50 for determining whether the original image contains a colon bag region, and for performing semantic segmentation on a blood vessel network region in the colon bag region to obtain a blood vessel network mask image (when the original image contains a colon bag region);

[0075] a blood vessel network feature encoding module 40 for performing feature extraction on the blood vessel network mask image to obtain blood vessel network matrix vector encoding;

[0076] a database filling and retrieval module 20 for obtaining patient identity information, for retrieving colon bag number and blood vessel network matrix vector encoding corresponding to the patient identity information in the colon blood vessel matrix vector library according to the patient identity information and the original image, and for filling the colon blood vessel matrix vector library;

[0077] The colon band recognition and numbering module 30 is configured to recognize and number the colon bands in the original image when the colon bag number and the blood vessel net matrix vector code corresponding to the patient identity information are not retrieved from the colon blood vessel matrix vector library. When the colon bag number and the blood vessel net matrix vector code corresponding to the patient identity information are retrieved from the colon blood vessel matrix vector library, the colon band numbering is determined according to the colon bag number and the blood vessel net matrix vector code in the colon blood vessel matrix vector library.

[0078] Further, the colon band recognition and numbering module 30 is further configured to determine the colon bag number according to the colon band numbering.

[0079] The retrieval result of the colon bag number and the blood vessel net matrix vector code corresponding to the patient identity information in the colon blood vessel matrix vector library according to the patient identity information and the original image is usually one of the following two cases:

[0080] (1) The patient identity information is not retrieved, and it is considered that the patient identity information, the colon bag number corresponding to the patient identity information, and the blood vessel net matrix vector are not retrieved.

[0081] (2) The patient identity information, the colon bag number corresponding to the patient identity information, and the blood vessel net matrix vector are retrieved.

[0082] Embodiment Three

[0083] The embodiment combines the embodiments one and two to specifically describe an enteroscopy image recognition and positioning method and system.

[0084] It can be understood that the colon blood vessel matrix vector library is a database having patient identity information, blood vessel net matrix vector code corresponding to the patient identity information, and colon bag number corresponding to the patient identity information.

[0085] The retrieval result of the colon bag number and the blood vessel net matrix vector code corresponding to the patient identity information in the colon blood vessel matrix vector library according to the patient identity information and the original image is usually one of the following two cases:

[0086] Retrieving the positioning information (such as ID) of the patient identity information in the colon blood vessel matrix vector library;

[0087] If the retrieval result has the positioning information, retrieving the colon bag number in the colon blood vessel matrix vector library that is most matched with the original image based on the positioning information; and then retrieving the blood vessel net matrix vector code in the most matched colon bag number that is most matched with the original image; that is, the colon bag information is the blood vessel net matrix vector code, or the blood vessel net matrix vector code and the colon bag number.

[0088] Correspondingly, in the identification positioning system, the database filling and searching module 20 is configured to search for the colon bag information corresponding to the patient identity information in the colon blood vessel matrix vector library according to the patient identity information and the original image, specifically: configured to search for the positioning information of the patient identity information in the colon blood vessel matrix vector library; configured to search for the colon bag number in the colon blood vessel matrix vector library that is most matched with the original image based on the positioning information; and configured to search for the blood vessel network matrix vector code in the most matched colon bag number that is most matched with the original image.

[0089] The colon blood vessel matrix vector library is filled with the patient identity information, the blood vessel network matrix vector code, and the colon bag number.

[0090] The database filling and searching module 20 is further configured to construct the colon blood vessel matrix vector library, that is, to construct the basic colon blood vessel matrix vector library architecture, specifically including designing the structure of the database, that is, including creating a table, setting a primary key, and indexing.

[0091] The identification and positioning method will be described in detail below.

[0092] It should be noted that there is a colon blood vessel matrix vector library, which can be empty or have certain data, and then the following steps are performed:

[0093] S1, the video acquisition module 10 obtains the enteroscopy video signal, which can be directly read from the output video of the endoscope device, and the video acquisition module 10 generates an original image according to the video; the database filling and searching module 20 obtains the patient identity information and determines whether the colon blood vessel matrix vector library has historical data of the patient according to the patient information, if yes, S2 is performed, otherwise S3 is performed;

[0094] S2, based on the search in S1, the database filling and searching module 20 searches for the colon bag number and the blood vessel network matrix vector code in the colon blood vessel matrix vector library according to the original image, and then the database filling and searching module 20 or the colon bag blood vessel network segmentation module 50 determines the number of the colon band, and S4 is performed;

[0095] S3, the colon band identification and numbering module 30 identifies the colon band in the original image and numbers the colon band, and further determines the colon bag number according to the number of the colon band, and S4 is performed;

[0096] S4, the colon bag blood vessel network segmentation module 50 determines whether the original image contains a colon bag region, and is configured to perform semantic segmentation on the blood vessel network region in the colon bag region to obtain a blood vessel network mask image, and S5 is performed;

[0097] S5, the blood vessel network feature encoding module 40 extracts features from the blood vessel network mask image to obtain a blood vessel network matrix vector code, and S6 is performed;

[0098] S6, the database filling and searching module 20 fills the colon blood vessel matrix vector library according to the patient identity information, the colon bag number and the blood vessel matrix vector code.

[0099] In this embodiment, the video acquisition module 10 is:

[0100] The video signal output by the endoscope device is transmitted to the computer by using a video capture card, the video signal is read frame by frame by using OpenCV, and the video signal is converted into an RGB image format, so as to ensure that the image quality is suitable for subsequent processing, and thus the generation of the original image is realized.

[0101] In this embodiment, the colon band identification and numbering module 30 is:

[0102] The core of this module is to build a neural network system capable of automatically identifying and numbering the colon bands, so as to ensure that each colon band can be accurately identified and assigned a unique number in order during colonoscopy.

[0103] The colon band identification and numbering module 30 realizes its function by using a multi-scale convolutional neural network architecture, as shown in FIG. 3.

[0104] The multi-scale convolutional neural network architecture combines different scale feature extraction techniques to capture multi-level information of the colon bands. The architecture mainly includes the following core parts: an input preprocessing part, a feature extraction part, a feature fusion part, a colon band identification part, and a number output part.

[0105] The input preprocessing part performs standardization processing on the input original image by using a data enhancement technique. The standardization processing mainly includes image normalization and uniform adjustment of the image size to a fixed size, so as to ensure the consistency of the input dimension of the network and improve the calculation efficiency. In this embodiment, the size after normalization is 640x640.

[0106] The feature extraction part includes a low-level feature extraction module, a middle-level feature extraction module and a high-level feature extraction module. The low-level features include edge features and texture features in the original image, the middle-level features are morphological features of the colon bands, and the high-level features include global context information features and spatial relationship features of the colon bands and the surrounding tissues.

[0107] The low-level feature extraction module includes three convolutional layers (Conv), each of which is followed by a maximum pooling layer (Max Pooling), to extract edge features and texture features in the original image after standardization processing. The first colon band feature map is output by the operation of the convolutional layer and the maximum pooling layer, that is, the first colon band feature map includes edge features and texture features.

[0108] Intermediate feature extraction module: This module consists of four groups of residual blocks (Residual Block). Each residual block contains three convolutional layers, which extract the morphological features of the colonic bands. That is, the first colonic band feature map is input, and the second colonic band feature map is output.

[0109] Advanced feature extraction module: Extract global context information and spatial relationship between colonic bands and surrounding tissues from the second colonic band feature map, and output the third colonic band feature map.

[0110] Feature fusion section aims to integrate features extracted from different levels, and further enhance the recognition ability of colonic bands through multi-scale fusion technology. The specific steps include:

[0111] Scale fusion: fuse the first colonic band feature map, the second colonic band feature map, and the third colonic band feature map to generate the first multi-scale feature map.

[0112] Attention mechanism weighting: apply attention mechanism to the first multi-scale feature map for weighted processing to generate the first weighted feature map. This operation enhances important features related to colonic bands while suppressing irrelevant or noise features.

[0113] Feature compression: compress the first weighted feature map to generate a set of numerical vectors for numbering, called high-dimensional numerical vectors. Compression is to compress the 3D data of the picture [spatial dimensions (length and width) and depth (height)] into 2D numerical vectors.

[0114] The colonic band recognition part is used to analyze and process the high-dimensional numerical vectors to identify the specific location of the colonic band.

[0115] After completing feature fusion and compression, the generated high-dimensional numerical vectors are analyzed and processed by the neural network to identify the specific location of the colonic band. The identification process classifies the colonic band using the extracted features and determines the unique identification of each colonic band.

[0116] Numbering output section includes the steps of numbering allocation and numbering confirmation.

[0117] Numbering allocation: by combining the movement path of the endoscope and the detected colonic band position, a unique number is allocated according to the order of the colonic band. In this embodiment, the number of each colonic band will increase with the movement path of the endoscope exiting, ensuring the logical continuity of the number.

[0118] Numbering confirmation: in each frame of image processing, the numbering of the current frame is confirmed or corrected by combining the colonic band numbering information of the previous and next frames. That is, according to the numbered colonic bag, the number of the colonic band is determined, and after determination, the numbered colonic band is completed.

[0119] In the number assignment, the colon pouches are also numbered according to the colon band numbers, i.e. for each colon band number obtained, the corresponding colon pouch number is obtained, for example, according to the withdrawal path (withdrawal of the endoscope), the subsequent colon pouch of the colon band numbered 1 is also numbered 1.

[0120] In the withdrawal path of the withdrawal of the endoscope, it is not always absolutely continuous, and there may be a certain degree of small-scale insertion, for example, the colon band numbers 1 to 4 have been preliminarily completed, and then there is an insertion, which is inserted into the colon band originally numbered 3, which will be numbered 5, but through the number confirmation step, i.e. by checking the colon pouch numbers of the front and rear frames, it can be known that the colon band cannot be numbered 5, but should be numbered 3.

[0121] It can be understood that in some embodiments, if the withdrawal continuity of the withdrawal operation is ensured, the number confirmation step can not be needed.

[0122] In this embodiment, the colon pouch vascular network segmentation module 50 is:

[0123] The core of this module is to accurately determine the image of the colon pouch area through an automated technical means, and to perform semantic segmentation on the vascular network structure in the colon pouch image.

[0124] The main steps of the whole process include colon band area image discrimination and segmentation of the vascular network area to obtain the vascular network mask image using the colon pouch semantic segmentation neural network.

[0125] Colon band area image discrimination: through texture and morphological feature analysis on the original image, and using the set rules to discriminate whether the current original image contains the colon pouch area, the colon pouch image is obtained. At this time, one case is that the original image does not contain areas other than the colon pouch area, and the original image is directly used as the colon pouch image or is used as the colon pouch image after image enhancement or denoising and other pretreatments. Another case is that the original image contains the colon pouch area, and the colon pouch image is obtained by segmenting the current original image.

[0126] The colon pouch semantic segmentation neural network architecture of this model is specially designed for the characteristics of endoscopic images, and a colon pouch semantic segmentation neural network CPV-NSNN (Colon Pouch Vascular Network Segmentation Neural Network) is designed. This architecture mainly enhances the task of image feature extraction through multi-scale fusion and the introduction of attention mechanism.

[0127] The colon bag semantic segmentation neural network architecture comprises an input preprocessing part, a multi-scale feature extraction part, a feature hierarchical aggregation part, a feature map multi-scale fusion part, a context awareness and attention mechanism part, and a semantic segmentation output part. For details, refer to FIG. 4.

[0128] The input preprocessing part: normalize the colon bag image. Specifically, the normalization parameters are adjusted adaptively based on the content of the colon bag image to obtain a normalized colon bag image. In this embodiment, the size of the normalized colon bag image is 640x640, which ensures that the key features of the colon bag image can be fully preserved under different light irradiation and contrast conditions.

[0129] The multi-scale feature extraction part: as the core of the CPV-NSNN, the multi-scale feature extraction part combines multi-scale convolution layers and adaptive convolution kernel technology, aiming to capture multi-level information in the normalized colon bag image through feature extraction of different scales. The specific implementation steps of this part are as follows:

[0130] Designing adaptive convolution kernel: automatically adjusting the size and shape of the convolution kernel according to the feature complexity of the current input colon bag image to capture features of different scales in the colon bag image. First, second, and third size convolution kernels are designed, with the first size being smaller than the second size, which is smaller than the third size.

[0131] First feature map generation: using the first size convolution kernel to capture edge and detail information in the normalized colon bag image to generate the first feature map.

[0132] Second feature map generation: using the second size convolution kernel to extract more complex texture and local shape information in the normalized colon bag image to generate the second feature map.

[0133] Third feature map generation: using the third size convolution kernel to capture overall morphological information of the colon bag in the normalized colon bag image to generate the third feature map.

[0134] The feature hierarchical aggregation part is used to stack the feature maps generated by the multi-scale feature extraction part with different size convolution kernels to form a multi-level feature map set. The specific steps of feature hierarchical aggregation include:

[0135] The first feature map is integrated through the hierarchical aggregation module to form the first level feature map.

[0136] The second feature map is fused with the first level feature map to generate the second level feature map.

[0137] The third feature map is fused with the second level feature map to generate the third level feature map.

[0138] The feature map multi-scale fusion part fuses the first-level feature map, the second-level feature map and the third-level feature map by using a multi-scale feature fusion method to obtain a first multi-scale fusion feature map. This includes combining low-resolution feature maps with high-resolution feature maps through upsampling operations.

[0139] The context awareness and attention mechanism part: the first multi-scale fusion feature map is subjected to global context awareness processing, and by capturing the global information and context relationship in the first multi-scale fusion feature map, the understanding of the entire colon bag region is enhanced. After context awareness, a double attention mechanism is applied to the processed feature map. The first layer of attention mechanism focuses on the spatial dimension, enhancing the key regions related to the colon bag vascular network in the image; the second layer of attention mechanism acts on the channel dimension, ensuring that important information in different feature channels is appropriately weighted. A first multi-scale weighted feature map is generated.

[0140] The semantic segmentation output part: finally, the first multi-scale weighted feature map is converted into a vascular network mask graph through a semantic segmentation output module.

[0141] Based on the features of the vascular network mask graph, a matrix vectorization technology is designed in this embodiment. The "matrix vectorization technology" is a method for structuring and coding vascular network feature information. This technology maps the vascular network features into grid cells and extracts key features (such as geometric center, vessel direction, vessel complexity) within each grid cell to generate corresponding three-dimensional feature vectors. Subsequently, these vectors are combined into a matrix form according to the grid arrangement order, called "matrix vector". This matrix form not only preserves the spatial structure information of the original image, but also compactly represents the image features as coded data that can be used for fast retrieval and comparison. The coded data is finally stored in a specially designed colon vascular matrix vector library (Colon Vascular Vector Matrix). The main steps of the vascular network feature coding module 40 are:

[0142] In this embodiment, the vascular network feature coding module 40 is used to divide the vascular network mask graph into a plurality of grid cells, extract features from each grid cell, and encode the extracted features into vectors. The vectors of all grid cells are combined into a matrix as the vascular network matrix vector code.

[0143] Feature extraction for each grid cell includes extracting geometric center features, vessel direction features, and vessel complexity features for each grid cell.

[0144] Specifically, the vascular network mask graph is divided into N x N grid cells according to N x N grid division, and the vectors of all grid cells are combined into an N x N matrix according to their arrangement order in the vascular network mask graph as the vascular network matrix vector code.

[0145] The blood vessel network feature coding process is described in detail below, referring to FIG. 5:

[0146] Grid division: divide the blood vessel network mask map into N x N grid cells, each of which corresponds to a local area in the blood vessel network mask map, and N is greater than 1.

[0147] Feature extraction: in each grid cell, extract three key features: the geometric center (C x ,C y ), the blood vessel direction (i.e. angle value) (θ), and the blood vessel complexity (i.e. shape complexity) (S), and encode the three key features into a three-dimensional vector.

[0148] First feature coding map generation: based on grid division and feature extraction, represent the geometric center of each grid cell with two-dimensional coordinates to form a local geometric feature map, calculate the geometric center of the blood vessel network in the grid cell, and generate the first feature coding map.

[0149] Geometric center calculation formula:

[0150] where C xi x represents the horizontal coordinate of the geometric center of the i-th grid cell, C yi y represents the vertical coordinate of the geometric center of the i-th grid cell, n represents the number of all points participating in the calculation in a certain grid cell, k represents the index of each point participating in the calculation in a certain grid cell, ranging from 1 to n, and (x k ,y k ) represents the coordinates of the points participating in the calculation in the certain grid cell.

[0151] Second feature coding map generation: use methods such as Hough transform to calculate the main extension direction angle θ of each grid cell in the blood vessel network, referred to as the extension direction angle, and generate the second feature coding map.

[0152] Angle value calculation formula:

[0153] where θ i represents the extension direction angle θ of the i-th grid cell, x' k represents the derivative of x k , and y' k represents the derivative of y k .

[0154] Third feature encoding map generation: the curvature variation and irregularity of the edge are quantified by calculating the Manhattan distance between adjacent edge points in the grid cell. By averaging these Manhattan distances, the shape complexity S is obtained, and the third feature encoding map is generated. The shape complexity S can reflect the complexity of the shape in a certain region. The higher the complexity, the more irregular the shape, and the more details or features it may contain.

[0155] Shape complexity S calculation formula:

[0156] where S i represents the shape complexity of the i-th grid cell, P represents the number of all edge points participating in the shape complexity calculation; j and j-1 are both indexes of points, (x j ,y j ) represents the coordinates of point j, (x j-1 ,y j-1 ) represents the coordinates of point j-1, and the edge points are indexed from 1 to P.

[0157] Matrix vectorization process: for each grid cell, its first feature encoding map, second feature encoding map and third feature encoding map are integrated into a three-dimensional vector V i . V i is stored as a feature vector V i , and finally all the three-dimensional vectors of the grid cells are integrated into a feature fusion matrix M, which is a matrix composed of the three-dimensional feature vectors V i of all grid cells according to their arrangement order in the blood vessel mask map. The three-dimensional vector V i of each cell reflects the geometric center, blood vessel direction and shape complexity in the cell. After these vectors are integrated into matrix M in order, the spatial distribution and morphological features of the blood vessel network in the image can be completely preserved. The three-dimensional vector V i and the first feature fusion matrix M have the following structures:

[0158] Preferably, the embodiment also extracts a global feature vector G of the blood vessel network according to the first feature fusion matrix M. The global feature vector G is obtained by compressing the matrix M through convolution. The global feature vector G summarizes the global structure and morphological features of the entire colon bag image, making it suitable for subsequent lesion matching and retrieval. The feature fusion matrix M and the global feature vector G are jointly used as the blood vessel network matrix vector encoding.

[0159] The database filling and retrieval module 20 stores the first feature fusion matrix M and the global feature vector G into the corresponding colon bag number in the colon blood vessel matrix vector library.

[0160] Referring to FIG. 5, in some embodiments, the database filling and retrieval module 20 is configured to extract the global feature vector G of the vascular network from the first feature fusion matrix M.

[0161] The function implementation of the database filling and retrieval module 20 includes the following steps:

[0162] Retrieving the location information of the patient's identity information in the colon vessel matrix vector library;

[0163] Based on the location information, retrieve the colon bag number in the colon vessel matrix vector library that best matches the original image;

[0164] Retrieving the blood vessel network matrix vector code in the best matching colon bag number that best matches the original image;

[0165] If no blood vessel network matrix vector code is retrieved, the colon band identification and numbering module 30 identifies the colon band in the original image and numbers the colon band.

[0166] In this embodiment, the database filling and retrieval module 20 combines the retrieval algorithm of analog quantum computing. In a non-quantum computing environment, the retrieval efficiency similar to quantum computing is improved through heuristic methods, ensuring that the patient's blood vessel network feature encoding module 40 is quickly and accurately matched and located in a large-scale data set. Specifically, it includes:

[0167] Patient record positioning, determine whether there is the patient's identity information: first, use a classical search algorithm (such as a hash table or an inverted index) or a quantum search algorithm to quickly locate the record that matches the current search condition (such as the ID or name representing the patient's identity information) in the colon vessel matrix vector library. The positioning speed can be accelerated through multi-layer screening or block search;

[0168] Colon bag number matching: using a heuristic search method, such as a heuristic search method similar to Grover's algorithm, quickly match in the multiple blood vessel network matrix vector codes stored in a specific colon bag (such as No. 1 or No. 2 colon bag). This process can accelerate the matching through multi-condition screening or priority-based multi-channel search, and find the best matching colon bag number according to the similarity of the search condition, that is, the blood vessel network matrix vector code set that matches the colon bag. The search condition is not limited to any condition, which can be determined according to the actual situation. The search condition is referred to as the first feature condition, and it can be understood that the highest similarity is the best match.

[0169] Code refinement retrieval: in the blood vessel network matrix vector code set of a certain colon bag found, use a classical optimization algorithm (such as a quantum approximation optimization algorithm) to process the second feature condition (usually including multiple features) matching. By iteratively optimizing the search path, the blood vessel network matrix vector code that best matches the second feature condition is gradually found, thereby achieving accurate matching.

[0170] Result output: Finally, output the colon bag number and blood vessel network matrix vector code that best match the current checking condition.

[0171] As an example of a preceding step in the retrieval process, the colon bag blood vessel network segmentation module 50 determines whether the original image contains a colon bag region, processes the colon bag region through the colon bag blood vessel network segmentation module 50 and the blood vessel network feature coding module 40 to obtain the feature map of the blood vessel network of the entire or partial region, analyzes the geometric center feature, the blood vessel direction feature, and the blood vessel complexity feature to obtain a three-dimensional vector, and finally fuses all the three-dimensional vectors in the feature map into a matrix and compresses it into a global feature vector. Some of the three-dimensional vectors can be used as the first feature condition, and the geometric center feature, the blood vessel direction feature, and the blood vessel complexity feature can be used as the second feature condition. It can be understood that the first feature condition and the second feature condition can be completely the same or can overlap, but the retrieval method and the retrieval success requirement of the colon bag number matching and the coding refinement retrieval are different.

[0172] The premise of the database filling retrieval module 20 filling the colon blood vessel matrix vector library is usually that the database filling retrieval module 20 does not have historical information of a certain patient, but in some embodiments, the colon blood vessel matrix vector library already has historical information of a certain patient, and the blood vessel network matrix vector code in the colon blood vessel matrix vector library is also updated, enriched, or optimized.

[0173] Embodiment four

[0174] In this embodiment, a computer program product is provided, which includes a computer program that, when executed by a processor, implements the above-mentioned method for identifying and positioning a colonoscopy image.

[0175] Embodiment five

[0176] In this embodiment, an electronic device is provided, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the above-mentioned method for identifying and positioning a colonoscopy image when executing the program.

[0177] Embodiment six

[0178] In this embodiment, a non-transitory computer-readable storage medium is provided, which stores a computer program, and the computer program, when executed by a processor, implements the above-mentioned method for identifying and positioning a colonoscopy image.

[0179] The processor can be a central processing unit, and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0180] The above-mentioned memory can include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. In addition, the memory can include a combination of any computer-readable storage media, and the memory can be a semiconductor memory chip, a magnetic disk, an optical disk.

[0181] The memory stores executable code, which, when processed by the processor, can cause the processor to perform part or all of the above-mentioned methods.

[0182] Those skilled in the art in the field to which the present application belongs can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of functional units and modules is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the above-mentioned functions. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit, and the integrated unit can be realized in the form of hardware or in the form of software functional unit. In addition, the specific names of each functional unit and module are only for the convenience of mutual differentiation, and do not limit the protection scope of the present application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the foregoing method embodiments, and will not be described here.

[0183] The effect of the colonoscopy image recognition and positioning method, device and product is that: the blood vessel network mask graph is obtained by performing semantic segmentation on the blood vessel network region in the colon bag region, feature extraction is performed on the blood vessel network mask graph to obtain blood vessel network matrix vector coding, and then the colon blood vessel matrix vector library is filled, so that the richness of the colon blood vessel matrix vector library is realized; the historical colon information is obtained by searching the colon blood vessel matrix vector library according to the patient identity information and the original image, and then the number of the colon band is obtained, so that the consistency of the number in different colonoscopy detections is ensured, and repeated marking is not performed, and at the same time, since an endoscope system such as a magnetic field positioning, CT / MRI image registration is not needed, the cost is low.

[0184] In current colonoscopy, algorithms such as convolutional neural network (CNN) are usually used to enhance the blood vessel network or other structures in the endoscopic image to help doctors identify lesions more clearly. For the marking of the colon band, doctors usually manually mark the colon band, which not only increases the workload of doctors, but also depends on the experience and operation of doctors, and is easily affected by subjective factors, resulting in inaccurate marking position, and further affecting the accurate positioning of subsequent lesions. The present application can intelligently identify the colon band in the original image and number the colon band, improve the accuracy of identification, reduce the workload of doctors, and also ensure the accuracy of subsequent blood vessel feature extraction.

[0185] In the present application, the segmentation technology is used for the colon bag blood vessel network, which can accurately segment the blood vessel network structure in the colon bag, and provide high-quality data required for fine feature extraction.

[0186] In the present application, the key features such as geometric center, direction and shape complexity of the blood vessel network are encoded by three-dimensional vectorization (first forming a vector and then combining into a matrix) to obtain encoded data, so that efficient and fast retrieval can be achieved in the colon blood vessel matrix vector library retrieval process, so that a large amount of patient data can be quickly processed. At the same time, the high data quality of feature extraction is ensured, and the accurate blood vessel network matrix vector encoding improves the accuracy of clinical diagnosis and treatment decision.

[0187] In the present application, the retrieval algorithm of analog quantum computing is applied, in a non-quantum computing environment, the retrieval efficiency similar to quantum computing is realized by heuristic method, and the colon bag code of the patient is quickly and accurately matched and positioned in large-scale data set.

[0188] Based on the accurate matching of the intestinal blood vessel matrix vector library of the present application, the lesions can be accurately positioned, the efficiency of disease determination in clinical work is improved, the risk of misdiagnosis is reduced, and strong support is provided for accurate positioning and tracking of lesions.

[0189] It should be noted that in the above embodiments, the description of each embodiment has its own emphasis, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0190] The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0191] The foregoing is considered as illustrative only of the principles of the application. Further, since numerous modifications and changes will readily occur to those skilled in the art, it is not desired to limit the application to the exact construction and practice described. Accordingly, all suitable modifications and equivalents can be resorted to falling within the scope of the application.

Claims

1. A method of identifying and locating an enteroscopic image, characterized by, The method comprises the following steps: obtaining an intestinal endoscope video signal and generating an original image therefrom; obtaining patient identity information; retrieving a colon bag number and a blood vessel network matrix vector code corresponding to the patient identity information in a colon blood vessel matrix vector library according to the patient identity information and the original image, and determining the number of colon bands; if not retrieved, identifying the colon bands in the original image and numbering the colon bands; determining whether the original image contains a colon bag region, and if so, performing semantic segmentation on the blood vessel network region in the colon bag region to obtain a blood vessel network mask image; extracting features from the blood vessel network mask image to obtain a blood vessel network matrix vector code; filling the colon blood vessel matrix vector library.

2. The method of claim 1, wherein, The method of retrieving a colon bag number and a blood vessel network matrix vector code corresponding to the patient identity information in a colon blood vessel matrix vector library according to the patient identity information and the original image specifically comprises the following steps: retrieving the positioning information of the patient identity information in the colon blood vessel matrix vector library; based on the positioning information, retrieving the colon bag number in the colon blood vessel matrix vector library that is most matched with the original image; retrieving the blood vessel network matrix vector code in the most matched colon bag number that is most matched with the original image.

3. The method of claim 2, wherein, The method of retrieving the colon bag number in the colon blood vessel matrix vector library that is most matched with the original image specifically comprises the following steps: using a heuristic search method to retrieve the colon bag number in the colon blood vessel matrix vector library that is most matched with the original image in terms of the first feature condition, as the most matched colon bag number. The method of retrieving the blood vessel network matrix vector code in the most matched colon bag number that is most matched with the original image specifically comprises the following steps: through iterative optimization of the search path, gradually finding the blood vessel network matrix vector code in the most matched colon bag number in the most matched colon bag number that is most matched with the original image in terms of the second feature condition.

4. The method of claim 1, wherein, The method of identifying the colon bands in the original image and numbering the colon bands specifically comprises the following steps: standardizing the original image; extracting the edge features and texture features of the original image after standardization to obtain a first colon band feature map; extracting the morphological features of the colon bands in the first colon band feature map to obtain a second colon band feature map; extracting the global context information in the second colon band feature map and the spatial relationship between the colon bands and the surrounding tissues to obtain a third colon band feature map; fusing the first colon band feature map, the second colon band feature map, and the third colon band feature map to generate a first multi-scale feature map; applying an attention mechanism to the first multi-scale feature map to perform weighted processing thereon to generate a first weighted feature map; compressing the first weighted feature map to generate a numerical vector; identifying the colon bands according to the numerical vector; numbering the identified colon bands.

5. The method of claim 4, wherein, The method of numbering the identified colon bands specifically comprises the following steps: numbering the identified colon bands according to the moving line of the endoscope withdrawal, numbering the colon bag according to the number of the colon bands, and determining the number of the colon bands according to the numbered colon bag.

6. The method of claim 1, wherein, The method of determining whether the original image contains a colon bag region, and if so, performing semantic segmentation on the blood vessel network region in the colon bag region to obtain a blood vessel network mask image specifically comprises the following steps: determining whether the original image contains a colon bag region, and if so, determining a colon bag image according to the original image; Normalizing the colon pouch image; extracting edge and detail information of the normalized colon pouch image to obtain a first feature map, extracting texture and local shape information of the normalized colon pouch image to obtain a second feature map, and extracting overall shape information of the colon pouch of the normalized colon pouch image to obtain a third feature map; hierarchically aggregating the first feature map to form a first hierarchical feature map, fusing the second feature map with the first hierarchical feature map to generate a second hierarchical feature map, and fusing the third feature map with the second hierarchical feature map to generate a third hierarchical feature map; fusing the first hierarchical feature map, the second hierarchical feature map and the third hierarchical feature map by using a multi-scale feature fusion method to obtain a first multi-scale fusion feature map; applying an attention mechanism to the spatial dimension and the channel dimension to generate a first multi-scale weighted feature map; transforming the first multi-scale weighted feature map into a blood vessel network mask graph.

7. The method of claim 1, wherein, The feature extraction on the blood vessel network mask graph to obtain the blood vessel network matrix vector code includes: dividing the blood vessel network mask graph into a plurality of grid cells, extracting features of each grid cell and encoding the extracted features into a vector, and combining the vectors of all grid cells into a matrix, which is taken as the blood vessel network matrix vector code.

8. The method of claim 7, wherein, The feature extraction on the blood vessel network mask graph to obtain the blood vessel network matrix vector code specifically includes: dividing the blood vessel network mask graph into grid cells according to N×N grid; extracting geometric center features, blood vessel direction features and blood vessel complexity features for each grid cell, and encoding the extracted geometric center features, blood vessel direction features and blood vessel complexity features into a vector; combining the vectors of all grid cells into an N×N matrix according to the arrangement order of the grid cells in the blood vessel network mask graph, extracting a global feature vector of the blood vessel network according to the N×N matrix, and taking the N×N matrix and the global feature vector together as the blood vessel network matrix vector code.

9. An apparatus for identifying and locating an enteroscopic image, characterized by, It includes: A video acquisition module is configured to obtain an enteroscopy video signal and generate an original image based on the enteroscopy video signal; A colon pouch blood vessel network segmentation module is configured to determine whether the original image contains a colon pouch region, and to perform semantic segmentation on a blood vessel network region in the colon pouch region to obtain a blood vessel network mask graph when the original image contains the colon pouch region; A blood vessel network feature encoding module is configured to perform feature extraction on the blood vessel network mask graph to obtain a blood vessel network matrix vector code; A database filling and retrieval module is configured to obtain patient identity information, to retrieve a colon pouch number and a blood vessel network matrix vector code corresponding to the patient identity information from a colon blood vessel matrix vector library according to the patient identity information and the original image, and to fill the colon blood vessel matrix vector library; A colon band identification and numbering module is configured to identify a colon band in the original image and number the colon band when the colon blood vessel matrix vector library does not retrieve the colon pouch number and the blood vessel network matrix vector code corresponding to the patient identity information, and to determine the number of the colon band according to the colon pouch number and the blood vessel network matrix vector code in the colon blood vessel matrix vector library.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by a processor to implement the method for identifying and positioning enteroscopy images according to any one of claims 1 to 8.

Images