A method, device and product for recognizing and positioning colonoscopy images

Abstract

This invention discloses a method, apparatus, and product for recognizing and locating colonoscopy images. The method includes: obtaining an intestinal endoscopy video signal and generating an original image based on it; obtaining patient identity information; retrieving the colonic haustra number and vascular network matrix vector code corresponding to the patient identity information from a colonic vascular matrix vector library based on the patient identity information and the original image, and determining the number of the colonic taenia; if not found, identifying and numbering the colonic taenia in the original image; determining whether the original image contains a colonic haustra region; if so, performing semantic segmentation on the vascular network region within the colonic haustra region to obtain a vascular network mask image; extracting features from the vascular network mask image to obtain a vascular network matrix vector code; and filling the colonic vascular matrix vector library. This invention achieves consistency in numbering across different colonoscopy examinations, avoids duplicate marking, and has low cost for consistent recognition.

Description

Technical Field

[0001] The present invention relates to the technical field of colonoscopy image processing, and in particular to a method, device and product for recognizing and positioning colonoscopy images. Background Art

[0002] Colonoscopy is a fiber endoscope commonly used in clinical practice. It can be inserted through the anus and moved retrogradely downward to examine the rectum, sigmoid colon, descending colon, transverse colon, ascending colon, cecum, and a small section of the small intestine connected to the large intestine (the end of the ileocecal end). Intestinal lesions can be clearly found, and some intestinal lesions can also be treated.

[0003] In today's colonoscopy, algorithms such as convolutional neural networks (CNN) are usually used to enhance the vascular network or other structures in the endoscopic image to help doctors identify lesions more clearly. For the marking of the colon band, doctors usually mark the colon band manually. Due to the different operating habits of each doctor, the results of manual marking are often inconsistent. It is difficult for the same doctor to use the same marking method between different inspection operations, and it is difficult to ensure the consistency of the current marking and historical marking. This leads to the inability to accurately match the previous marking position in subsequent inspections, and repeated marking operations waste time.

[0004] The above-mentioned problem of inconsistent markings may lead to different lesion positioning between multiple examinations, resulting in inaccurate lesion determination, affecting the effectiveness of diagnosis and treatment. Some advanced endoscope systems use magnetic field positioning, CT / MRI image registration and other technologies to help doctors locate the colon. However, these systems require the use of various sensors and other equipment, which leads to high system prices and limited application range.

[0005] Therefore, in order to solve the problem of lack of a colonoscopy identification and positioning method with low cost, consistent front and back markings, and no repeated marking operations, it is urgent to develop a colonoscopy image identification and positioning method and system. Summary of the invention

[0006] In view of the above problems, the present invention provides a method, device and product for recognizing and locating colonoscopy images.

[0007] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0008] In a first aspect, the present invention provides a method for recognizing and locating a colonoscopy image, comprising:

[0009] Obtain intestinal endoscopy video signals and generate original images accordingly;

[0010] Obtain patient identification information;

[0011] Retrieving the colon bag number and vascular network matrix vector code corresponding to the patient identity information in the colon vascular matrix vector library according to the patient identity information and the original image, and determining the number of the colon band; if not retrieved, identifying the colon band in the original image and numbering the colon band;

[0012] Determine whether the original image contains a colon bag region, and if so, perform semantic segmentation on the vascular network region in the colon bag region to obtain a vascular network mask map;

[0013] Extracting features from the vascular network mask image to obtain a vascular network matrix vector code;

[0014] Populate the colon vascular matrix vector library.

[0015] In a preferred embodiment, the method of retrieving the colon bag number and the vascular network matrix vector code corresponding to the patient identity information in the colon vascular matrix vector library according to the patient identity information and the original image specifically includes:

[0016] Retrieving the location information of the patient's identity information in the colon vascular matrix vector library;

[0017] Based on the positioning information, the colon bag number that best matches the original image is retrieved from the colon vascular matrix vector library;

[0018] Retrieve the vascular network matrix vector encoding that best matches the original image in the colon bag number that best matches the colon bag number.

[0019] In a preferred embodiment, the step of retrieving the colon bag number in the colon vascular matrix vector library that best matches the original image is as follows: using a heuristic search method to retrieve the colon bag number in the colon vascular matrix vector library that has the highest similarity to the first characteristic condition of the original image as the best matching colon bag number;

[0020] The method of retrieving the vascular network matrix vector encoding that best matches the original image in the most matching colon bag number is specifically: by iteratively optimizing the search path, gradually finding the vascular network matrix vector encoding that best matches the second characteristic condition of the original image in the most matching colon bag number.

[0021] In a preferred embodiment, the step of identifying the colon bands in the original image and numbering the colon bands specifically includes:

[0022] Performing standardization processing on the original image;

[0023] Extract edge features and texture features of the original image after standardization to obtain the first colon band feature map;

[0024] Extracting the morphological features of the colon band of the first colon band feature map to obtain a second colon band feature map;

[0025] Extracting global context information in the second colon band feature map and the spatial relationship between the colon band and surrounding tissues to obtain a third colon band feature map;

[0026] The first colon band feature map, the second colon band feature map, and the third colon band feature map are fused to generate a first multi-scale feature map;

[0027] Applying an attention mechanism to weight the first multi-scale feature map to generate a first weighted feature map;

[0028] Compressing the first weighted feature map to generate a numerical vector;

[0029] identifying a colon zone according to the numerical vector;

[0030] The identified colonic bands were numbered.

[0031] In a preferred embodiment, the numbering of the identified colon bands is specifically as follows: numbering the identified colon bands according to the moving route of the endoscope, numbering the colon bags according to the number of the colon bands, and determining the number of the colon bands according to the numbered colon bags.

[0032] In a preferred embodiment, the determining whether the original image includes a colon bag region, and if so, performing semantic segmentation on the vascular network region in the colon bag region to obtain a vascular network mask map specifically includes:

[0033] Determine whether the original image contains a colon bag area, and if so, determine a colon bag image according to the original image;

[0034] Normalized colon bag image;

[0035] Extract edge and detail information of the normalized colon bag image to obtain a first feature map, extract texture and local shape information of the normalized colon bag image to obtain a second feature map, and extract overall morphological information of the colon bag of the normalized colon bag image to obtain a third feature map;

[0036] The first feature map is hierarchically aggregated to form a first-level feature map; the second feature map is fused with the first-level feature map to generate a second-level feature map; the third feature map is fused with the second-level feature map to generate a third-level feature map;

[0037] A multi-scale feature fusion method is used to fuse the first-level feature map, the second-level feature map and the third-level feature map to obtain a first multi-scale fused feature map;

[0038] Capturing the global information and contextual relationship in the first multi-scale fusion feature map, applying the attention mechanism to the spatial dimension and the channel dimension, and generating the first multi-scale weighted feature map;

[0039] The first multi-scale weighted feature map is converted into a vascular network mask map.

[0040] In a preferred embodiment, the feature extraction of the vascular network mask image to obtain the vascular network matrix vector code includes: dividing the vascular network mask image into a number of grid units, extracting features from each grid unit and encoding the extracted features into a vector, combining the vectors of all grid units into a matrix, and using the matrix as the vascular network matrix vector code.

[0041] In a preferred embodiment, the feature extraction of the vascular network mask image to obtain the vascular network matrix vector encoding specifically includes: dividing the vascular network mask image into grid units according to an N×N grid; extracting the geometric center feature, vascular direction feature, and vascular complexity feature of each grid unit, and encoding the extracted geometric center feature, vascular direction feature, and vascular complexity feature into a vector; combining the vectors of all grid cells into an N×N matrix according to the arrangement order of the grid cells in the vascular network mask image, extracting the global feature vector of the vascular network according to the N×N matrix, and the N×N matrix and the global feature vector are jointly used as the vascular network matrix vector encoding.

[0042] In a second aspect, the present invention provides a device for recognizing and locating a colonoscopy image, comprising:

[0043] A video acquisition module, used for acquiring intestinal endoscope video signals and generating original images accordingly;

[0044] A colon bag vascular network segmentation module is used to determine whether the original image contains a colon bag region, and to perform semantic segmentation on the vascular network region in the colon bag region to obtain a vascular network mask map when the original image contains the colon bag region;

[0045] A vascular network feature encoding module is used to extract features from the vascular network mask image to obtain a vascular network matrix vector encoding;

[0046] A database filling and retrieval module is used to obtain patient identity information, and is used to retrieve the colon bag number and vascular network matrix vector code corresponding to the patient identity information in the colon vascular matrix vector library according to the patient identity information and the original image, so as to fill the colon vascular matrix vector library;

[0047] The colon band recognition and numbering module is used to identify and number the colon band in the original image when the colon bag number and vascular network matrix vector code corresponding to the patient's identity information are not retrieved in the colon vascular matrix vector library, and is used to determine the colon band number according to the colon bag number and vascular network matrix vector code in the colon vascular matrix vector library.

[0048] In a third aspect, the present invention provides a computer program product, comprising a computer program, characterized in that when the computer program is executed by a processor, it implements a method for identifying and locating a colonoscopy image as described in the first aspect.

[0049] The present invention provides a method, system and product for identifying and locating colonoscopy images, which obtain a vascular network mask map by semantic segmentation of a vascular network area in a colon bag area, extract features from the vascular network mask map to obtain a vascular network matrix vector code, and then fill a colon vascular matrix vector library, thereby enriching the colon vascular matrix vector library; retrieve the colon vascular matrix vector library according to the patient's identity information and the original image to obtain historical colon information, and then obtain the number of the colon band, thereby ensuring the consistency of the numbering during different colonoscopy inspections without repeated marking, and at the same time, because no external endoscope system is required, the cost is low. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 A flow chart of a method for recognizing and locating colonoscopy images provided in Example 1.

[0051] Figure 2 A framework diagram of a colonoscopy image recognition and positioning system provided in Example 2.

[0052] Figure 3 This is a framework diagram of the colon band identification and numbering module provided in Example 3.

[0053] Figure 4 This is a framework diagram of the colon bag vascular network segmentation module provided in Example 3.

[0054] Figure 5 This is a functional flow chart of the vascular network feature encoding module and the database filling and retrieval module provided in the third embodiment. DETAILED DESCRIPTION

[0055] In order to more clearly understand the above-mentioned objects, features and advantages of the present invention, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0056] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0057] It should be noted that the descriptions involving "first", "second", etc. in this application are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in this field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0058] Colonoscopy is the main tool for screening and diagnosing colorectal cancer, but due to the complex anatomical structure of the colon and individual differences, it is difficult for doctors to accurately locate the position of lesions between different examinations. Traditional methods rely on the doctor's experience and subjective judgment, and there are large errors. This error may lead to inaccurate lesion positioning between multiple examinations, affecting the effectiveness of diagnosis and treatment. If combined with today's positioning system, due to the high cost of the system, the price of colonoscopy will be high, and the positioning system equipment is complex and its application is limited. Therefore, an automated and standardized method is needed to help doctors accurately identify and locate lesions in multiple examinations. To this end, the present invention provides a method, device and product for identifying and positioning colonoscopic images. The present invention and its effects are described in detail below with several embodiments.

[0059] Embodiment 1

[0060] See also Figure 1 This embodiment provides a method for identifying and locating a colonoscopy image, including:

[0061] Obtain intestinal endoscopy video signals and generate original images accordingly;

[0062] Obtain patient identification information;

[0063] Retrieve the colon bag number and vascular network matrix vector code corresponding to the patient identity information in the colon vascular matrix vector library according to the patient identity information and the original image; if the colon bag number and vascular network matrix vector code corresponding to the patient identity information in the colon vascular matrix vector library are retrieved, determine the number of the colon band; if not retrieved, identify the colon band in the original image and number the colon band, and determine the colon bag number according to the number of the colon band;

[0064] Determine whether the original image contains a colon bag region, and if so, perform semantic segmentation on the vascular network region in the colon bag region to obtain a vascular network mask map;

[0065] Extracting features from the vascular network mask image to obtain a vascular network matrix vector code;

[0066] Populate the colon vascular matrix vector library.

[0067] The colon vascular matrix vector library includes patient identification information, colon bag numbers and vascular network matrix vector codes.

[0068] It should be understood that the above method may vary depending on the execution process. For example, there is no limitation on the order of obtaining the intestinal endoscopy video signal and generating the original image and obtaining the patient's identity information based thereon. Therefore, the above method does not represent or imply that all steps must be executed in this order. A person of ordinary skill in the art can transform or change the execution order of the above steps based on the present invention. Some embodiments of the above method are illustrated below.

[0069] Embodiment 2

[0070] See also Figure 2 This embodiment provides a device for identifying and locating a colonoscopy image, the device comprising:

[0071] The video acquisition module 10 is used to obtain the intestinal endoscope video signal and generate the original image accordingly;

[0072] The colon bag vascular network segmentation module 50 is used to determine whether the original image contains the colon bag area, and to (when the original image contains the colon bag area) perform semantic segmentation on the vascular network area in the colon bag area to obtain a vascular network mask map;

[0073] The vascular network feature encoding module 40 is used to extract features from the vascular network mask image to obtain a vascular network matrix vector encoding;

[0074] A database filling and retrieval module 20 is used to obtain patient identity information, and to retrieve the colon bag number and vascular network matrix vector code corresponding to the patient identity information in the colon vascular matrix vector library according to the patient identity information and the original image, so as to fill the colon vascular matrix vector library;

[0075] The colon band identification and numbering module 30 is used to identify the colon band in the original image and number the colon band when the colon vascular matrix vector library fails to retrieve the colon bag number and vascular network matrix vector code corresponding to the patient's identity information, and is used to determine the colon band number according to the colon bag number and vascular network matrix vector code in the colon vascular matrix vector library (when the database filling and retrieval module 20 retrieves the colon bag number and vascular network matrix vector code corresponding to the patient's identity information in the colon vascular matrix vector library).

[0076] Furthermore, the colon band identification and numbering module 30 is also used to determine the colon bag number according to the colon band number.

[0077] The retrieval result of retrieving the colon bag number and the vascular network matrix vector code corresponding to the patient identity information in the colon vascular matrix vector library according to the patient identity information and the original image is usually one of the following two:

[0078] (1) The patient identity information is not retrieved. In this case, it is considered that the patient identity information, the colon bag number corresponding to the patient identity information, and the vascular network matrix vector are not retrieved;

[0079] (2) Retrieve the patient's identity information, the colon bag number corresponding to the patient's identity information, and the vascular network matrix vector.

[0080] Embodiment 3

[0081] This embodiment specifically describes a method and system for recognizing and locating colonoscopy images in combination with Embodiment 1 and Embodiment 2.

[0082] It can be understood that the colon vascular matrix vector library is a database having patient identification information, vascular network matrix vector codes corresponding to the patient identification information, and colon bag numbers corresponding to the patient identification information.

[0083] According to the patient identity information and the original image, the colon bag number and the vascular network matrix vector code corresponding to the patient identity information in the colon vascular matrix vector library are retrieved. If not retrieved, the colon band in the original image is identified and the colon band is numbered, which specifically includes:

[0084] Retrieve the location information (eg, ID) of the patient's identity information in the colon vascular matrix vector library;

[0085] If the search result has the positioning information, based on the positioning information, the colon bag number that best matches the original image in the colon vascular matrix vector library is retrieved; then the vascular network matrix vector code that best matches the original image in the best matching colon bag number is retrieved; that is, the colon bag information is the vascular network matrix vector code, or the vascular network matrix vector code and the colon bag number.

[0086] Correspondingly, in the identification and positioning system, the database filling retrieval module 20 is used to retrieve the colon bag information corresponding to the patient identity information in the colon vascular matrix vector library according to the patient identity information and the original image, specifically: used to retrieve the positioning information of the patient identity information in the colon vascular matrix vector library; used to retrieve the colon bag number in the colon vascular matrix vector library that best matches the original image based on the positioning information; used to retrieve the vascular network matrix vector code that best matches the original image in the best matching colon bag number.

[0087] The method for filling the colon vascular matrix vector library is specifically used to fill the colon vascular matrix vector library with patient identity information, vascular network matrix vector code and colon bag number.

[0088] The database filling and retrieval module 20 is also used to construct a colon vascular matrix vector library, that is, to construct a basic colon vascular matrix vector library architecture, which specifically includes designing a database structure, that is, including creating tables, setting primary keys, indexes, etc.

[0089] The identification and positioning method is described in detail below.

[0090] It should be noted that there is a colon vascular matrix vector library, which may be empty or have certain data, and then the following steps are performed:

[0091] S1, the video acquisition module 10 obtains the intestinal endoscope video signal, which can be directly reading the video output by the endoscope device, and the video acquisition module 10 generates the original image according to the video; the database filling and retrieval module 20 obtains the patient's identity information and determines whether the colon vascular matrix vector library has the patient's historical data according to the patient information, if so, proceed to S2, otherwise proceed to S3;

[0092] S2, the database filling retrieval module 20 retrieves the colon bag number and the vascular network matrix vector code in the colon vascular matrix vector library according to the original image based on the retrieval in S1, and then the database filling retrieval module 20 or the colon bag vascular network segmentation module 50 determines the number of the colon band, and then proceeds to S4;

[0093] S3, the colon band recognition and numbering module 30 recognizes the colon band in the original image and numbers the colon band, and further determines the colon bag number according to the colon band number, and then proceeds to S4;

[0094] S4, the colon bag vascular network segmentation module 50 determines whether the original image contains the colon bag area, and is used to perform semantic segmentation on the vascular network area in the colon bag area to obtain a vascular network mask map, and then proceed to S5;

[0095] S5, the vascular network feature encoding module 40 extracts features from the vascular network mask image to obtain a vascular network matrix vector encoding, and then proceeds to S6;

[0096] S6. The database filling and retrieval module 20 fills the colon vascular matrix vector library according to the patient identification information, the colon bag number and the vascular network matrix vector code.

[0097] In this embodiment, the video acquisition module 10 is:

[0098] Using a video capture card, the video signal output by the endoscope device is transmitted to the computer. OpenCV is used to read the video signal frame by frame and transcode it into RGB image format to ensure that the image quality is suitable for subsequent processing. This realizes the generation of the original image.

[0099] In this embodiment, the colon band identification and numbering module 30 is:

[0100] The core of this module is to build a neural network system that can automatically identify and number colon bands, ensuring that during colonoscopy, each colon band can be accurately identified and assigned a unique number in sequence.

[0101] The colon band identification and numbering module 30 implements its functions through a multi-scale convolutional neural network architecture, see Figure 3 .

[0102] The multi-scale convolutional neural network architecture combines feature extraction techniques at different scales to capture the multi-level information of the colon band. The architecture mainly includes the following core parts: input preprocessing part, feature extraction part, feature fusion part, colon band recognition part, and numbering output part.

[0103] The input preprocessing part uses data enhancement technology to standardize the input original image. Standardization mainly includes image normalization, which adjusts the image size to a fixed size to ensure the consistency of the network input dimension and improve the calculation efficiency. In this embodiment, the normalized size is 640×640.

[0104] The feature extraction part includes low-level feature extraction module, intermediate feature extraction module and high-level feature extraction module. Low-level features include edge features and texture features in the original image, intermediate features are morphological features of the colon band, and high-level features include global context information features and spatial relationship features between the colon band and surrounding tissues.

[0105] Low-level feature extraction module: This module includes three convolutional layers (Conv), each of which is followed by a maximum pooling layer (Max Pooling) to extract edge features and texture features from the normalized original image. The first colon feature map is output through the operation of the convolutional layer and the maximum pooling layer, that is, the first colon feature map includes edge features and texture features.

[0106] Intermediate feature extraction module: This module consists of four groups of residual blocks. Each residual block contains three convolutional layers to extract the morphological features of the colon belt. That is, the first colon belt feature map is used as input and the second colon belt feature map is output.

[0107] Advanced feature extraction module: extracts the global context information in the second colon band feature map and the spatial relationship between the colon band and the surrounding tissues, and outputs the third colon band feature map.

[0108] The feature fusion part aims to integrate the features extracted from different levels and further enhance the recognition ability of the colon band through multi-scale fusion technology. The specific steps include:

[0109] Scale fusion: The first colon band feature map, the second colon band feature map, and the third colon band feature map are fused to generate a first multi-scale feature map.

[0110] Attention mechanism weighting: The first multi-scale feature map is weighted by applying the attention mechanism to generate the first weighted feature map. This operation enhances the important features related to the colon band while suppressing irrelevant or noise features.

[0111] Feature compression: compress the first weighted feature map to generate a set of numerical vectors for final numbering, called high-dimensional numerical vectors. Compression is to compress the image's 3D data [spatial dimensions (length and width) and depth (height)] into a 2D numerical vector.

[0112] The colon band identification part is used to analyze and process the high-dimensional numerical vector to identify the specific position of the colon band.

[0113] After feature fusion and compression, the generated high-dimensional numerical vector is analyzed and processed by a neural network to identify the specific location of the colon band. This recognition process uses the above extracted features to classify the colon bands and determine the unique identity of each colon band.

[0114] The number output part includes the steps of number allocation and number confirmation.

[0115] Number assignment: By combining the movement path of the endoscope and the detected colonic band positions, unique numbers are assigned according to the order of the colonic bands. In this embodiment, the number of each colonic band will increase as the endoscope exits the movement path to ensure the logical continuity of the numbering.

[0116] Number confirmation: In the processing of each frame of the image, the number of the current frame will be confirmed or corrected by combining the colonic band numbering information of the previous and next frames. That is, the number of the colonic band is determined according to the numbered colonic bags. After the confirmation, the numbering of the identified colonic band is completed.

[0117] In the number allocation, the colon bags are also numbered according to the colon band number, that is, each time a colon band number is obtained, the corresponding colon bag number is obtained. For example, according to the withdrawal path of the endoscope, the colon bag subsequent to the colon band numbered 1 is also numbered 1.

[0118] In the moving path of the retreat, it is usually not an absolutely continuous retreat, and a certain degree of small advance may occur. For example, the numbering of the colon belts numbered 1 to 4 has been preliminarily completed, and then the advance occurs, and the colon belt originally numbered 3 is advanced. The colon belt will be numbered 5, but through the number confirmation step, that is, by checking the colon bag numbers of the previous and next frames, it can be known that the colon belt cannot be numbered 5, but should be 3.

[0119] It is understandable that in some embodiments, if the exit continuity of the mirror withdrawal operation is guaranteed, the number confirmation step may not be required.

[0120] In this embodiment, the colon bag vascular network segmentation module 50 is:

[0121] The core of this module is to accurately determine the images with colon bag areas through automated technical means, and to perform semantic segmentation on the vascular network structure in the colon bag images.

[0122] The main steps of the whole process include image recognition of the colon belt area and segmentation of the vascular network area using the colon bag semantic segmentation neural network to obtain a vascular network mask map.

[0123] Colon bag area image discrimination: By analyzing the texture and morphological features of the original image, and using the set rules to discriminate whether the current original image contains the colon bag area, a colon bag image is obtained. At this time, one case is that the original image does not contain areas other than the colon bag area, and the original image is directly used as a colon bag image or is used as a colon bag image after image enhancement or denoising. Another case is that the original image contains the colon bag area, and the colon bag image is obtained by segmenting the current original image.

[0124] The colon pouch semantic segmentation neural network architecture of this model is specially designed for the characteristics of endoscopic images. A colon pouch semantic segmentation neural network CPV-NSNN (Colon Pouch Vascular Network Segmentation Neural Network) is used to strengthen the image feature extraction task mainly through the introduction of multi-scale fusion and attention mechanism.

[0125] The neural network architecture of the colon bag semantic segmentation includes input preprocessing, multi-scale feature extraction, feature hierarchical aggregation, feature map multi-scale fusion, context perception and attention mechanism, and semantic segmentation output. Figure 4 .

[0126] Input preprocessing part: normalized colon bag image. Specifically, the normalization parameters are adaptively adjusted based on the content of the colon bag image to obtain a normalized colon bag image. This embodiment uses 640×640 to ensure that the key features of the colon bag image can be fully preserved under different lighting and contrast conditions.

[0127] The multi-scale feature extraction: The multi-scale feature extraction part, as the core of CPV-NSNN, combines multi-scale convolutional layers and adaptive convolution kernel technology, aiming to capture multi-level information in the normalized colon bag image through feature extraction at different scales. The following are the specific implementation steps of this part:

[0128] Design adaptive convolution kernel: According to the feature complexity of the current input colon bag image, the size and shape of the convolution kernel are automatically adjusted to capture features of different scales in the colon bag image. The first size convolution kernel, the second size convolution kernel and the third size convolution kernel are designed. The first size is smaller than the second size and smaller than the third size.

[0129] First feature map generation: The first size convolution kernel is used to capture the edge and detail information in the normalized colon bag image to generate the first feature map.

[0130] Second feature map generation: A convolution kernel of a second size is used to extract more complex texture and local shape information in the normalized colon bag image to generate a second feature map.

[0131] Third feature map generation: The third size of the convolution kernel is used to capture the overall morphological information of the colon bag in the normalized colon bag image to generate the third feature map.

[0132] The feature hierarchical aggregation part is used to stack the feature maps generated by the convolution kernels of different sizes in the multi-scale feature extraction part level by level to form a multi-level feature map set. The specific steps of feature hierarchical aggregation include:

[0133] The first feature map is preliminarily integrated through a hierarchical aggregation module to form a first-level feature map;

[0134] The second feature map is fused with the first-level feature map to generate a second-level feature map;

[0135] The third feature map is fused with the second level feature map to generate a third level feature map.

[0136] The multi-scale fusion part of the feature map adopts a multi-scale feature fusion method to fuse the first-level feature map, the second-level feature map and the third-level feature map to obtain a first multi-scale fused feature map, which includes combining the low-resolution feature map with the high-resolution feature map through an upsampling operation.

[0137] Context-aware and attention mechanism part: Global context-aware processing is performed on the first multi-scale fusion feature map to enhance the understanding of the entire colon bag area by capturing the global information and contextual relationship in the first multi-scale fusion feature map. After context-aware, a dual attention mechanism is applied to the processed feature map. The first layer of attention mechanism focuses on the spatial dimension to enhance the key areas related to the colon bag vascular network in the image; the second layer of attention mechanism acts on the channel dimension to ensure that the important information in different feature channels is appropriately weighted. Generate the first multi-scale weighted feature map.

[0138] Semantic segmentation output part: Finally, the first multi-scale weighted feature map is converted into a vascular network mask map through the semantic segmentation output module.

[0139] This embodiment designs a matrix vectorization technology based on the characteristics of the vascular network mask map. The "matrix vectorization technology" is a method for structured encoding of vascular network feature information. This technology generates corresponding three-dimensional feature vectors by mapping the vascular network features into grid cells and extracting key features (such as geometric center, vascular direction, and vascular complexity) in each grid cell. Subsequently, these vectors are combined into a matrix form according to the grid arrangement order, which is called a "matrix vector". This matrix form can not only retain the spatial structural information of the original image, but also compactly represent the image features as coded data that can be used for rapid retrieval and comparison. Finally, the coded data is stored in a specially designed colon vascular matrix vector library (Colon Vascular Vector Matrix). The main steps of the vascular network feature encoding module 40 are:

[0140] In this embodiment, the vascular network feature encoding module 40 is used to divide the vascular network mask image into several grid units, extract features from each grid unit and encode the extracted features into vectors, and combine the vectors of all grid units into a matrix as a vascular network matrix vector encoding.

[0141] Feature extraction for each grid cell includes extracting geometric center features, blood vessel direction features, and blood vessel complexity features for each grid cell.

[0142] Specifically, the vascular network mask map is N × N The grid is divided into N × N grid cells, and the vectors of all grid cells are combined according to the order of their arrangement in the vascular network mask map into N × N The matrix is ​​used as the vascular network matrix vector encoding.

[0143] The following is a detailed description of the vascular network feature encoding process, see Figure 5 :

[0144] Mesh division: Divide the vascular network mask map into N × N Each grid cell corresponds to a local area in the vascular network mask map. N Greater than 1.

[0145] Feature extraction: In each grid cell, extract the geometric center ( C x , C y ), blood vessel direction (i.e. angle value) ( θ ), vascular complexity (i.e. shape complexity) ( S ) three key features and encode the three key features into a three-dimensional vector.

[0146] Generation of the first feature coding map: Based on grid division and feature extraction, the geometric center of each grid unit is represented by two-dimensional coordinates to form a local geometric feature map, and the geometric center of the vascular network in the grid unit is calculated to generate the first feature coding map.

[0147] The geometric center calculation formula is:

[0148]

[0149] in, C xi Indicates i The horizontal coordinate of the geometric center of the grid cell, C yi Indicates i The vertical coordinate of the geometric center of the grid cell, n represents the number of all points involved in the calculation in a specific grid cell, k Represents the index of each point involved in the calculation in a specific grid cell, ranging from 1 to n , ( x k , y k ) represents the coordinates of the points participating in the calculation in that particular grid cell.

[0150] Second feature coding map generation: Use Hough transform and other methods to calculate the main extension direction angles in each grid cell of the vascular network θ , referred to as the extension direction angle, generates the second feature coding map.

[0151] Angle value calculation formula:

[0152]

[0153] in, Indicatesi The extension direction angle of the grid cells θ , express x k The derivative of express y k The derivative of .

[0154] Third feature encoding map generation: quantify the curvature change and irregularity of the edge by calculating the Manhattan distance between adjacent edge points in the grid cell. By averaging these Manhattan distances, we get the shape complexity S , generate the third feature encoding map. Shape complexity S It reflects the complexity of the shapes in a particular area. The higher the complexity, the more irregular the shape is and the more details or features it may contain.

[0155] Shape complexity S Calculation formula:

[0156]

[0157] in, S i Indicates i The shape complexity of the grid cells, P Represents the number of all edge points involved in the shape complexity calculation; j and j -1 is used as the index of the point. x j , y j ) indicates a point j The coordinates of x j-1 , y j-1 ) indicates a point j -1 coordinates, edge points from 1 to P Index sorting.

[0158] Matrix vectorization process: For each grid cell, its first feature coding map, second feature coding map, and third feature coding map are integrated into a three-dimensional vector . As a feature vector, store the vector , and finally integrate the three-dimensional vectors of all grid cells into a feature fusion matrix M , feature fusion matrix M To transform the three-dimensional feature vectors of all grid cells The matrix is ​​composed according to the order of arrangement of its grid cells in the vascular network mask map. The three-dimensional vector of each cell All of them reflect the geometric center, blood vessel direction and shape complexity within the unit. These vectors are sequentially integrated into the matrix M After that, the spatial distribution and morphological characteristics of the vascular network in the image can be completely preserved. And the first feature fusion matrix M The structure is as follows:

[0159]

[0160]

[0161] Preferably, this embodiment also combines the first feature matrix M Extracting global feature vectors of vascular networks G , the global eigenvector G It is through convolution of the matrix M The global feature vector is obtained by compression. G The global structure and morphological features of the entire colon bag image are summarized, making it suitable for subsequent lesion matching and retrieval. Feature fusion matrix M and the global eigenvector G Together they are encoded as a vascular network matrix vector.

[0162] The database filling retrieval module 20 fuses the first feature matrix M and the global eigenvector G The entry is stored in the corresponding colon bag number in the colon vascular matrix vector library.

[0163] See also Figure 5 In some embodiments, the database filling retrieval module 20 is used to fuse the matrix according to the first feature M Extracting global feature vectors of vascular networks G .

[0164] The function implementation of the database filling retrieval module 20 includes the following steps:

[0165] Retrieving the location information of the patient's identity information in the colon vascular matrix vector library;

[0166] Based on the positioning information, the colon bag number that best matches the original image is retrieved from the colon vascular matrix vector library;

[0167] Retrieve the vascular network matrix vector encoding that best matches the original image in the colon bag number that best matches;

[0168] If the vascular network matrix vector code is not retrieved, the colon band identification and numbering module 30 identifies the colon band in the original image and numbers the colon band.

[0169] In this embodiment, the database filling retrieval module 20 combines the retrieval algorithm of simulated quantum computing, and achieves the retrieval efficiency improvement similar to quantum computing through heuristic methods in a non-quantum computing environment, ensuring the rapid and accurate matching and positioning of the patient's vascular network feature encoding module 40 in a large-scale data set. Specifically including:

[0170] Patient record location to determine whether there is the patient's identity information: First, use a classical search algorithm (such as a hash table or inverted index) or a quantum search algorithm in the colon vascular matrix vector library to quickly locate records that match the current search criteria (such as an ID or name representing the patient's identity information). Multi-layer screening or block search can be used to speed up the location process;

[0171] Colon bag number matching: Use a heuristic search method, such as a heuristic search method similar to the Grover algorithm, to quickly match multiple vascular network matrix vector codes stored in a specific colon bag (such as colon bag No. 1 or No. 2). This process can accelerate the matching through multiple condition screening or priority-based multi-channel search, and find the most matching colon bag number based on the similarity of the retrieval conditions, that is, the vascular network matrix vector code set that matches the colon bag. The retrieval condition is not limited to any condition, and can be determined according to the actual situation. The retrieval condition is called the first characteristic condition. It can be understood that the highest similarity is the most matched.

[0172] Code refinement retrieval: In the set of vascular network matrix vector codes of a certain colon bag found, a classical optimization algorithm (such as quantum approximate optimization algorithm) is used to process the second characteristic condition (usually including multiple characteristics) matching. By iteratively optimizing the search path, the vascular network matrix vector code that best meets the second characteristic condition is gradually found, thereby achieving accurate matching.

[0173] Result output: Finally, the colon bag number and vascular network matrix vector encoding that best matches the current examination conditions are output.

[0174] As an example of a pre-order step in the retrieval process, the colon bag vascular network segmentation module 50 determines whether the original image contains a colon bag area, and the colon bag area is processed by the colon bag vascular network segmentation module 50 and the vascular network feature encoding module 40 to obtain a feature map of the vascular network of all or part of the area, and the geometric center feature, vascular direction feature and vascular complexity feature are analyzed to obtain a three-dimensional vector. Finally, all three-dimensional vectors in the feature map are fused into a matrix and compressed into a global feature vector. Part of the three-dimensional vector can be used as the first feature condition, and the geometric center feature, vascular direction feature and vascular complexity feature can be used as the second feature condition. It can be understood that the first feature condition and the second feature condition can be exactly the same or overlapped, but the retrieval method and the requirements for successful retrieval are different for the colon bag number matching and coding refinement retrieval.

[0175] The premise that the database filling and retrieval module 20 fills the colon vascular matrix vector library is usually that the database filling and retrieval module 20 does not have the historical information of a certain patient. However, in some embodiments, the colon vascular matrix vector library already has the historical information of a certain patient, and the colon vascular matrix vector library is also updated, enriched or optimized. The vascular network matrix vector encoding therein.

[0176] Embodiment 4

[0177] In this embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the method for identifying and locating a colonoscopy image is implemented.

[0178] Embodiment 5

[0179] In this embodiment, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method for identifying and locating a colonoscopy image when executing the program.

[0180] Embodiment 6

[0181] In this embodiment, the present invention provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned method for identifying and locating a colonoscopy image is implemented.

[0182] The processor may be a central processing unit, or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0183] The memory may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage. In addition, the memory may include any combination of computer-readable storage media, such as semiconductor memory chips, magnetic disks, and optical disks.

[0184] The memory stores executable codes, and when the executable codes are processed by the processor, the processor can execute part or all of the methods described above.

[0185] The technicians in the field to which the present invention belongs can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0186] The effects of the method, device and product for identifying and locating colonoscopic images of the present invention are as follows: the present invention obtains a vascular network mask map by semantic segmentation of the vascular network area in the colon bag area, obtains a vascular network matrix vector code by feature extraction of the vascular network mask map, and then fills the colon vascular matrix vector library, thereby enriching the colon vascular matrix vector library; obtains historical colon information by searching the colon vascular matrix vector library according to the patient's identity information and the original image, and then obtains the number of the colon band, thereby ensuring the consistency of the numbering during different colonoscopic examinations, and no repeated marking is performed. At the same time, since there is no need to connect an external endoscope system such as magnetic field positioning, CT / MRI image registration, etc., the cost is low.

[0187] In today's colonoscopy, algorithms such as convolutional neural networks (CNN) are usually used to enhance the vascular network or other structures in the endoscopic image to help doctors identify lesions more clearly. For the marking of the colon bands, doctors usually mark the colon bands manually, which not only increases the workload of doctors, but also depends on the experience and operation of doctors. It is easily affected by subjective factors and leads to inaccurate marking positions, which in turn affects the precise positioning of subsequent lesions. The present invention can intelligently identify the colon bands in the original image and number the colon bands, which improves the accuracy of recognition, reduces the workload of doctors, and can also ensure the accuracy of subsequent vascular feature extraction.

[0188] The present invention adopts segmentation technology for the colon bag vascular network, which can accurately segment the vascular network structure in the colon bag and provide high-quality data required for refined feature extraction.

[0189] The method of three-dimensional vector coding (first forming vectors and then combining them into matrices) of key features such as the geometric center, direction and shape complexity of the vascular network in the present invention obtains coded data, which enables efficient and rapid retrieval in the colon vascular matrix vector library retrieval process, so that a large amount of patient data can be quickly processed. At the same time, high data quality of feature extraction is ensured, and accurate vascular network matrix vector coding improves the accuracy of clinical diagnosis and treatment decisions.

[0190] The present invention applies a retrieval algorithm that simulates quantum computing. In a non-quantum computing environment, a heuristic method is used to achieve quantum computing-like retrieval efficiency improvement, ensuring rapid and accurate matching and positioning of the patient's colon bag code in large-scale data sets.

[0191] The precise matching of the intestinal vascular matrix vector library based on the present invention is conducive to the accurate positioning of lesions, improves the efficiency of disease determination in clinical work, reduces the risk of misdiagnosis, and provides strong support for the precise positioning and tracking of lesions.

[0192] It should be noted that in the above embodiments, the description of each embodiment has its own emphasis, and for parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0193] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system and method according to multiple embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a part of a module, a program segment or a code, and the part of the module, the program segment or the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or the flow chart, and the combination of the square boxes in the block diagram and / or the flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0194] The above is only a preferred embodiment of the present invention. It should be noted that although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they know the basic creative concept. Therefore, the attached claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention. Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for recognizing and locating a colonoscopy image, characterized in that: include: Obtain intestinal endoscopy video signals and generate original images accordingly; Obtain patient identification information; According to the patient identification information and the original image, the colon bag number and the vascular network matrix vector code corresponding to the patient identification information in the colon vascular matrix vector library are retrieved to determine the number of the colon band; if not retrieved, the colon band in the original image is identified and the colon band is numbered; Determine whether the original image contains a colon bag region, and if so, perform semantic segmentation on the vascular network region in the colon bag region to obtain a vascular network mask map; Extracting features from the vascular network mask image to obtain a vascular network matrix vector code; If the colon bag number and vascular network matrix vector code corresponding to the patient identity information are retrieved in the colon vascular matrix vector library, the colon vascular matrix vector library is updated; if not retrieved, the colon vascular matrix vector library is filled.

2. A method for recognizing and locating a colonoscopy image according to claim 1, characterized in that: The method of retrieving the colon bag number and the vascular network matrix vector code corresponding to the patient identity information in the colon vascular matrix vector library according to the patient identity information and the original image specifically includes: Retrieving the location information of the patient's identity information in the colon vascular matrix vector library; Based on the positioning information, the colon bag number that best matches the original image is retrieved from the colon vascular matrix vector library; Retrieve the matrix vector encoding of the vascular network that best matches the original image in the colon bag number that best matches it.

3. A method for recognizing and locating a colonoscopy image as claimed in claim 2, characterized in that: The method of retrieving the colon bag number that best matches the original image in the colon vascular matrix vector library is specifically as follows: using a heuristic search method to retrieve the colon bag number that has the highest similarity to the first characteristic condition of the original image in the colon vascular matrix vector library as the best matching colon bag number; The method of retrieving the vascular network matrix vector encoding that best matches the original image in the most matching colon bag number is specifically: by iteratively optimizing the search path, gradually finding the vascular network matrix vector encoding that best matches the second characteristic condition of the original image in the most matching colon bag number.

4. The method for recognizing and locating a colonoscopy image according to claim 1, characterized in that: The identifying of the colon bands in the original image and numbering the colon bands specifically includes: Performing standardization processing on the original image; Extract edge features and texture features of the original image after standardization to obtain the first colon band feature map; Extracting the morphological features of the colon band of the first colon band feature map to obtain a second colon band feature map; Extracting global context information in the second colon band feature map and the spatial relationship between the colon band and surrounding tissues to obtain a third colon band feature map; The first colon band feature map, the second colon band feature map, and the third colon band feature map are fused to generate a first multi-scale feature map; Applying an attention mechanism to weight the first multi-scale feature map to generate a first weighted feature map; Compressing the first weighted feature map to generate a numerical vector; identifying a colon zone according to the numerical vector; The identified colonic bands were numbered.

5. A method for recognizing and locating a colonoscopy image as claimed in claim 4, characterized in that: The specific method of numbering the identified colon bands is as follows: numbering the identified colon bands according to the moving route of the endoscope, numbering the colon bags according to the number of the colon bands, and determining the number of the colon bands according to the numbered colon bags.

6. The method for recognizing and locating a colonoscopy image according to claim 1, characterized in that: The step of determining whether the original image includes a colon bag region, and if so, performing semantic segmentation on the vascular network region in the colon bag region to obtain a vascular network mask map specifically includes: Determine whether the original image contains a colon bag area, and if so, determine a colon bag image according to the original image; Normalized colon bag image; Extract edge and detail information of the normalized colon bag image to obtain a first feature map, extract texture and local shape information of the normalized colon bag image to obtain a second feature map, and extract overall morphological information of the colon bag of the normalized colon bag image to obtain a third feature map; The first feature map is hierarchically aggregated to form a first-level feature map; the second feature map is fused with the first-level feature map to generate a second-level feature map; the third feature map is fused with the second-level feature map to generate a third-level feature map; A multi-scale feature fusion method is used to fuse the first-level feature map, the second-level feature map and the third-level feature map to obtain a first multi-scale fused feature map; Capturing the global information and contextual relationship in the first multi-scale fusion feature map, applying the attention mechanism to the spatial dimension and the channel dimension, and generating the first multi-scale weighted feature map; The first multi-scale weighted feature map is converted into a vascular network mask map.

7. The method for recognizing and locating a colonoscopy image according to claim 1, characterized in that: The step of extracting features from the vascular network mask image to obtain a vascular network matrix vector code includes: dividing the vascular network mask image into a number of grid units, extracting features from each grid unit and encoding the extracted features into a vector, combining the vectors of all grid units into a matrix, and using the matrix as the vascular network matrix vector code.

8. The method for recognizing and locating a colonoscopy image according to claim 7, characterized in that: The feature extraction of the vascular network mask image to obtain the vascular network matrix vector encoding specifically includes: extracting the vascular network mask image according to N × N The grid is divided into grid units; the geometric center feature, vascular direction feature, and vascular complexity feature are extracted from each grid unit, and the extracted geometric center feature, vascular direction feature, and vascular complexity feature are encoded into a vector; the vectors of all grid units are combined according to the arrangement order of the grid units in the vascular network mask map. N × N The matrix, according to N × N The matrix extracts the global feature vector of the vascular network. N × N The matrix and the global eigenvector are used together as the vascular network matrix vector encoding.

9. A device for identifying and locating colonoscopy images, characterized in that: include: A video acquisition module, used for acquiring intestinal endoscope video signals and generating original images accordingly; A colon bag vascular network segmentation module is used to determine whether the original image contains a colon bag region, and to perform semantic segmentation on the vascular network region in the colon bag region to obtain a vascular network mask map when the original image contains the colon bag region; A vascular network feature encoding module is used to extract features from the vascular network mask image to obtain a vascular network matrix vector encoding; a database filling and retrieval module, used to obtain patient identity information, to retrieve the colon bag number and vascular network matrix vector code corresponding to the patient identity information in the colon vascular matrix vector library according to the patient identity information and the original image, to update the colon vascular matrix vector library when the colon bag number and vascular network matrix vector code corresponding to the patient identity information are retrieved, and to fill the colon vascular matrix vector library when the colon bag number and vascular network matrix vector code corresponding to the patient identity information are not retrieved; The colon band identification and numbering module is used to identify the colon band in the original image and number the colon band when the colon vascular matrix vector library fails to retrieve the colon bag number and vascular network matrix vector code corresponding to the patient's identity information, and is used to determine the colon band number according to the colon bag number and vascular network matrix vector code in the colon vascular matrix vector library when the colon vascular matrix vector library retrieves the colon bag number and vascular network matrix vector code corresponding to the patient's identity information.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, it implements a method for identifying and locating a colonoscopy image as described in any one of claims 1 to 8.

Images