Gastrointestinal endoscope video analysis method, system and equipment and storage medium

Abstract

The invention relates to the technical field of medical image processing, in particular to a gastrointestinal endoscope video analysis method which comprises the following steps: acquiring a gastrointestinal endoscope video; performing frame extraction processing on the gastrointestinal endoscope video to obtain a plurality of gastrointestinal endoscope images; inputting the gastrointestinal endoscope image into an image abnormal region detection model, analyzing the gastrointestinal endoscope image by the image abnormal region detection model, identifying whether an abnormal region exists in the gastrointestinal endoscope image, and outputting an abnormal region identification result; and recording the gastrointestinal endoscope image of which the abnormal region identification result is that the abnormal region exists as an abnormal image, inputting the abnormal image into the focus classification model, analyzing the abnormal image by the focus classification model, and outputting a focus prediction result. The invention further provides a gastrointestinal endoscope video analysis system and device and a storage medium. The method can effectively assist gastrointestinal endoscope doctors in judging various focuses, and the stability and accuracy of diagnosis results are improved.

Description

Technical Field

[0001] The present invention relates to the field of medical image processing technology, and in particular to a gastroenteroscopic video analysis method, system, device and storage medium. Background Art

[0002] Gastroenteroscopy plays an important role in the diagnosis and treatment of digestive system diseases, including diagnosis of gastrointestinal diseases, sampling biopsies, therapeutic operations, monitoring treatment effects and screening for early cancer. During the gastroenteroscopy, the doctor inserts the gastroenteroscopy into the patient's body and starts the camera system to record and properly store the real-time image of the stomach or intestines. The doctor can watch the video and carefully observe the shape, color, texture and other characteristics of the stomach or intestines to find possible lesions or abnormalities.

[0003] However, since gastroenteroscopy videos contain a large amount of data and information, traditional manual analysis methods often consume a lot of time and energy. In addition, manual analysis methods are easily affected by the doctor's subjective factors, resulting in instability and inaccuracy in the diagnosis results. On the one hand, gastroenteroscopists need to undergo a long period of professional training and many years of clinical diagnosis experience to have a relatively stable and accurate judgment; on the other hand, a gastroenteroscopist often needs to perform more than 20 gastroscopy or colonoscopy cases a day, and his attention and energy will drop significantly in the later stage, affecting the diagnosis.

[0004] In recent years, with the rapid development of computer vision and artificial intelligence technology, medical imaging diagnosis methods based on deep learning have gradually become a research hotspot. Some deep learning-based gastroenteroscopy-assisted diagnosis methods have also emerged. For example, the Chinese patent with publication number CN111899229A discloses a method for assisting the diagnosis of early gastric cancer based on deep learning multi-model fusion technology. The patent first uses a classification model to classify white light images and NBI images, then uses a target detection model to frame the lesion area, and then crops the image in the lesion frame and sends it to the instance segmentation network for lesion classification and lesion area contour extraction. Finally, the result is output through a decision model to determine early cancer. This method uses a total of 4 deep learning models. Not only is the model structure too complex and redundant, but network training requires a large number of positive and negative sample images of early cancer as training sets. However, the acquisition of early cancer case images is not only difficult but also scarce. Furthermore, deep learning methods based on supervised learning require pre-provided labeled data as the true value of model training. The labeling of true value data not only requires expert physicians to use labeling software for labeling, but also requires reference to biopsy pathology as a reference. The labeling workload is very huge. This is also an important reason why the current medical image diagnosis method based on supervised learning is difficult to achieve significant development. At the same time, in the diagnosis of gastrointestinal lesions in gastrointestinal video, the lesions in the gastrointestinal area are very complex and there are many types of lesions. There may even be some rare diseases that are difficult for even experienced experts to judge. This may cause a large number of missed detections, leaving safety risks to people's health. Summary of the invention

[0005] In order to effectively assist gastroenteroscopic physicians in making judgments on various lesions and improve the stability and accuracy of diagnostic results, a gastroenteroscopic video analysis method, system, device and storage medium are provided.

[0006] In order to achieve the above-mentioned purpose of the present invention, the present invention provides a gastroenteroscopic video analysis method.

[0007] The present invention provides a gastroenteroscopic video analysis method, comprising:

[0008] Obtain gastrointestinal endoscopy video;

[0009] Perform frame extraction processing on the gastroenteroscopic video to obtain a number of gastroenteroscopic images;

[0010] The gastrointestinal endoscopy image is input into the image abnormal region detection model, and the image abnormal region detection model analyzes the gastrointestinal endoscopy image, identifies whether there is an abnormal region in the gastrointestinal endoscopy image, and outputs the abnormal region recognition result;

[0011] The gastrointestinal endoscopy image with abnormal region recognition result as the abnormal region is recorded as an abnormal image, and the abnormal image is input into the lesion classification model. The lesion classification model analyzes the abnormal image and outputs the lesion prediction result.

[0012] Optionally, the abnormal image region detection model is a variational autoencoder, and the abnormal image region detection model processes the gastrointestinal endoscopy image in the following steps:

[0013] Map the gastrointestinal endoscopy image x to a low-dimensional latent space to obtain latent variables, and generate a reconstructed gastrointestinal endoscopy image x based on the latent variables. ’ , and obtain the reconstructed gastrointestinal endoscopy image x ’ The difference score of each pixel between the gastrointestinal endoscopy image x;

[0014] Compare the difference score to the preset difference score threshold:

[0015] If the difference score exceeds the preset difference score threshold, the pixel point position corresponding to the difference score is an abnormal area;

[0016] If the difference score does not exceed the preset difference score threshold, the pixel point position corresponding to the difference score is a normal area;

[0017] The abnormal area recognition result is determined based on the comparison result of the difference score of each pixel point with the preset difference score threshold.

[0018] Optionally, the method further comprises:

[0019] The abnormal area of ​​the gastrointestinal endoscope image is integrated, and the image range corresponding to the abnormal area is intercepted from the gastrointestinal endoscope image using a rectangular frame to obtain an abnormal image.

[0020] Optionally, the steps of processing the abnormal image by the lesion classification model include:

[0021] The features of abnormal images are extracted, and the images are classified into predefined categories to obtain the recognition results of abnormal images; the recognition results include abnormal images as known lesions and abnormal images as unknown lesions:

[0022] If the recognition result is that the abnormal image is a known lesion, the lesion prediction result includes the lesion category, the probability corresponding to the lesion category, and the mask image of the lesion area in the abnormal image;

[0023] If the recognition result is that the abnormal image is an unknown lesion, the lesion prediction result includes a mask image of the lesion area in the abnormal image.

[0024] Optionally, the method further comprises the step of training an image abnormal region detection model:

[0025] A training data set is obtained, wherein the training data set includes a number of gastrointestinal endoscopy sample images without lesions;

[0026] Construct the network structure of the abnormal region detection model of the image to be trained;

[0027] The network of the abnormal region detection model of the image to be trained is trained using the training data set. In each training, the loss function is determined according to the output result of the abnormal region detection model of the image to be trained;

[0028] The network parameters of the image abnormal region detection model to be trained are optimized according to the loss function to obtain the final image abnormal region detection model.

[0029] Optionally, the loss function L VAE Including the reconstruction loss L recon and KL divergence loss L KL , the expression of the loss function is L VAE =L recon +β·L KL , β is a hyperparameter that controls the KL divergence weight.

[0030] Optionally, mark the high-incidence area of ​​lesions in the gastrointestinal endoscopy sample image, and record the area other than the high-incidence area of ​​lesions in the gastrointestinal endoscopy sample image as the background area, and reconstruct the loss L recon Including reconstruction loss in lesion-prone areas and background region reconstruction loss

[0031] α is the coefficient for controlling the reconstruction loss weight of the lesion-prone area, (1-α) is the coefficient for controlling the reconstruction loss weight of the background area, and α is greater than 0.5 and α is less than 1.

[0032] In order to solve the above problems, the present invention also provides a gastroenteroscopic video analysis system, the system comprising:

[0033] An acquisition module, used to acquire gastrointestinal endoscopy video;

[0034] An image generation module is used to perform frame extraction processing on the gastroenteroscopic video to obtain a number of gastroenteroscopic images;

[0035] An abnormal region recognition module is used to input the gastrointestinal endoscope image into the image abnormal region detection model, and the image abnormal region detection model analyzes the gastrointestinal endoscope image, identifies whether there is an abnormal region in the gastrointestinal endoscope image, and outputs the abnormal region recognition result;

[0036] The lesion prediction module is used to record the gastrointestinal endoscopic image with abnormal area recognition results as an abnormal image, input the abnormal image into the lesion classification model, and the lesion classification model analyzes the abnormal image and outputs the lesion prediction result.

[0037] In order to solve the above problem, the present invention further provides an electronic device, the electronic device comprising:

[0038] at least one processor; and,

[0039] a memory communicatively connected to the at least one processor; wherein,

[0040] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the above-mentioned gastroenteroscopic video analysis method.

[0041] In order to solve the above problems, the present invention also provides a computer-readable storage medium, in which at least one computer program is stored. The at least one computer program is executed by a processor in an electronic device to implement the above-mentioned gastroenteroscopic video analysis method.

[0042] In summary, this application includes the following beneficial technical effects:

[0043] The present invention uses an image abnormal region detection model to identify gastrointestinal endoscopic images with lesion regions, and then uses a lesion classification model to identify lesion regions in abnormal images and output lesion prediction results for reference by gastrointestinal endoscopists, thereby making up for missed detections caused by fatigue and negligence of physicians and improving the stability and accuracy of gastrointestinal endoscopic examination results.

[0044] The use of a neural network model constructed by a variational autoencoder based on unsupervised learning to detect lesion areas in gastrointestinal endoscopy images can solve the problem in the prior art that the supervised learning-based method requires professional physicians to annotate data lesions, which causes a large workload and the difficulty in obtaining negative sample data, greatly reducing the time and cost of collecting and annotating training data. The present invention uses an unsupervised learning variational autoencoder network as an image abnormal area detection model, combined with a lesion classification model using a supervised learning classification network, which can effectively identify and detect known lesions and improve the accuracy of gastrointestinal endoscopy results. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A schematic diagram of a flow chart of a gastroenteroscopic video analysis method provided by an embodiment of the present invention;

[0046] Figure 2 A schematic diagram of the training steps of an abnormal image region detection model provided by an embodiment of the present invention;

[0047] Figure 3 A schematic diagram of the structure of an electronic device for implementing the gastroenteroscopic video analysis method provided in one embodiment of the present invention.

[0048] Reference numerals: 10, processor; 11, memory; 12, communication bus; 13, communication interface.

[0049] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0050] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0051] In the description of the present invention, it is necessary to understand that the terms "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0052] In the description of the present invention, unless otherwise specified and limited, it should be noted that the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a mechanical connection or an electrical connection, or it can be the internal connection between two components. It can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to the specific circumstances.

[0053] Reference Figure 1 FIG. 2 is a flow chart of a gastroenteroscopic video analysis method provided by an embodiment of the present invention. In this embodiment, the gastroenteroscopic video analysis method includes:

[0054] S1. Obtain gastroenteroscopy video.

[0055] Specifically, the gastrointestinal endoscopic video is the original video output from the gastrointestinal endoscopic device. The gastrointestinal endoscopic video can be imported through an external device (such as a USB flash drive, a removable hard disk, etc.), or read through a memory or a database, or accessed through a video collector to the video stream image output by the gastrointestinal endoscopic device. This application does not impose any restrictions on the method of obtaining the gastrointestinal endoscopic video.

[0056] S2. Frame extraction is performed on the gastroenteroscopic video to obtain a number of gastroenteroscopic images.

[0057] In a preferred embodiment of the present application, firstly, a gastrointestinal endoscopic image is subjected to frame extraction processing to obtain N single-frame initial images (N is a positive integer greater than 0), and the above N single-frame original images are respectively subjected to image effective area cropping processing to obtain N gastrointestinal endoscopic images containing only the effective area.

[0058] The video images output by the original gastroenteroscope device contain some information related to the gastroenteroscope device and the acquisition environment, such as the configuration parameters of the endoscope, the parameters of the operating environment, the parts of the image, etc. This information is not helpful for lesion prediction and will increase the computing power cost of the computer when processing the image. Therefore, it is necessary to first cut out the effective gastroenteroscope image area in the center to increase the speed of the computer's subsequent processing of the gastroenteroscope image. The method of interception can be to intercept according to the preset length, width and center point parameters (for gastroenteroscope videos output by different models of gastroenteroscope devices, the length, width and center point position of the intercepted area need to be adjusted, and the user can adjust the size according to the actual situation); or the effective area of ​​the intercepted image can be determined by first binarizing the image and then extracting the largest connected area.

[0059] In a preferred implementation of this embodiment, after the effective area is intercepted from the original image, the effective area image intercepted from the original image is denoised using a median filtering algorithm to obtain a final gastroenteroscopic image. Since the signal transmission line is affected by strong current signals from other nearby devices, the video stream image output by the gastroenteroscopic device generally has some salt and pepper noise; the median filter uses a fixed-size window to slide on the image, and then replaces the value of the center pixel with the median of the pixels in the window. The median filter can effectively remove salt and pepper noise while retaining the edges and details of the image, thereby improving image quality and the accuracy of image analysis.

[0060] S3. Input the gastrointestinal endoscopy image into the image abnormal region detection model. The image abnormal region detection model analyzes the gastrointestinal endoscopy image, identifies whether there is an abnormal region in the gastrointestinal endoscopy image, and outputs the abnormal region recognition result.

[0061] Specifically, the steps of processing the gastrointestinal endoscopy image by the image abnormal region detection model include:

[0062] S310, mapping the gastrointestinal endoscopy image x to a low-dimensional latent space to obtain latent variables, and generating a reconstructed gastrointestinal endoscopy image x according to the latent variables ’ , and obtain the reconstructed gastrointestinal endoscopy image x ’ The difference score of each pixel between the gastrointestinal endoscopy image x;

[0063] S320: Compare the difference score with a preset difference score threshold:

[0064] If the difference score exceeds the preset difference score threshold, the pixel position corresponding to the difference score is an abnormal area;

[0065] If the difference score does not exceed the preset difference score threshold, the pixel point position corresponding to the difference score is a normal area.

[0066] S330: Determine an abnormal area recognition result based on a comparison result of the difference score of each pixel point with a preset difference score threshold.

[0067] In this embodiment, the difference score threshold is set to T. The setting of T is adaptive. For each different image block (for example, the entire gastrointestinal endoscopy image is divided into 100*100 local areas), the mean μ and standard deviation σ of the gray value of each image block are calculated, and the difference score threshold T of the corresponding image block is adaptively set according to the mean μ and standard deviation σ of the image block:

[0068] T=μ+k·σ

[0069] Among them, k is an adjustment coefficient, which can be determined based on experimental experience or data characteristics. A larger k value can enhance the sensitivity of detection.

[0070] The abnormal area of ​​the gastrointestinal endoscope image is integrated, and the image range corresponding to the abnormal area is intercepted from the gastrointestinal endoscope image using a rectangular frame to obtain an abnormal image.

[0071] A mask operation is performed on the pixel positions whose scores are greater than the difference score threshold T to obtain an abnormal area mask image; the minimum bounding rectangle of the abnormal area mask is taken, and the minimum bounding rectangle frame is intercepted on the original gastrointestinal endoscopy image x to obtain the abnormal image y, and the abnormal image y is the lesion area image; if the entire gastrointestinal endoscopy image has no abnormal area mask, it means that the gastrointestinal endoscopy image has no abnormal area, and the result of no lesion is output.

[0072] In this embodiment, before using the image abnormal region detection model to process the gastrointestinal endoscopy image, the image abnormal region detection model needs to be trained using training data.

[0073] Reference Figure 2 , the training steps for the image abnormal area detection model are:

[0074] S301, obtaining a training data set, where the training data set includes a number of gastrointestinal endoscopy sample images without lesions;

[0075] S302, constructing a network structure of an abnormal region detection model for an image to be trained;

[0076] In a preferred implementation of this embodiment, the image abnormal region detection model is a variational autoencoder, which can be used for anomaly detection tasks. By learning the distribution of data and generating new samples, abnormal samples that are different from training data can be identified.

[0077] The full name of variational autoencoder is Variational Autoencoder, or VAE for short. It combines the ideas of autoencoder and probabilistic graphical model. When modeling the generative model, VAE explicitly defines the conditional probability distribution and learns the parameters of the generative model through maximum likelihood estimation, so that it can generate samples similar to the training data and obtain the abnormality score of the samples. In the video-assisted diagnosis of gastroenterology, a large number of normal gastroenterology images without lesions can be used to train a VAE model, which solves the problem that the supervised learning-based method in the existing technology requires professional physicians to annotate the data lesions, which causes a large workload and the difficulty in obtaining negative sample data. It greatly reduces the time and cost of training data collection and annotation, so that the cross-domain combination of AI+gastroenterology can be effectively and quickly implemented. In actual use, the trained image abnormal area detection model can be used to detect the area with lesions (abnormal area) in actual clinical diagnosis by using the abnormal score between the generated image and the input image, and then the abnormal area in the image can be classified through a classification network to classify the lesions. The network structure of the VAE-based image abnormal area detection model includes an encoder and a decoder. The encoder is used to map the input data (gastrointestinal endoscope image x) to a low-dimensional latent space to obtain the distribution parameters of the latent variable z. The model generates samples of the latent variable z by sampling from the standard normal distribution and using the parameters output by the encoder. The distribution parameters of the latent variable z are the mean μ and standard deviation σ of the latent variable z; the decoder is used to map the samples of the latent variable z back to the original image space and regenerate the reconstructed gastrointestinal endoscope image x. ’ .

[0078] S303, using the training data set to train the network of the abnormal region detection model of the image to be trained, and in each training, determining the loss function according to the output result of the abnormal region detection model of the image to be trained.

[0079] S304. Optimize the network parameters of the image abnormal region detection model to be trained according to the loss function to obtain a final image abnormal region detection model.

[0080] The encoder encodes the input gastrointestinal endoscopy image x into the distribution parameters (mean and variance) of the latent space, while the decoder remaps the latent variables back to the image space. The training is completed after the network parameters of the training image abnormal region detection model converge.

[0081] The loss function L of the abnormal region detection model of the image to be trained VAE Including the reconstruction loss L recon and KL divergence loss L KL , reconstruction loss L reconThe KL divergence loss L is used to measure the difference between the reconstructed image output after the variational encoder and the input gastrointestinal endoscopy sample image. KL It is used to measure the difference between the potential distribution of the gastrointestinal endoscopy sample images output by the variational encoder and the standard normal distribution. By minimizing the KL divergence, VAE can ensure that the distribution of the latent space is close to the standard normal distribution.

[0082] The expression of the loss function is L VAE =L recon +β·L KL , where β is a hyperparameter that controls the KL divergence weight;

[0083] The expression of reconstruction loss is L recon =E q(z|x) [log p(x|z)],

[0084] Where q(z|x) represents the approximate posterior probability distribution of the latent variable z generated by the input gastrointestinal endoscopy sample image x, E q(z|x) represents the expected value calculation of the posterior probability distribution q(z|x) of the latent variable z, and p(x|z) represents the probability distribution of the reconstructed input gastrointestinal endoscope sample image x given the latent variable z;

[0085] The expression of KL divergence loss is

[0086]

[0087] Where μ is the mean of the latent variable z output by the encoder, σ is the variance of the latent variable z output by the encoder, and D KL (q(z|x)||p(z)) represents the information loss of distribution q(z|x) relative to distribution p(z), p(z) represents the prior distribution learned on the latent variable z; i represents the i-th group of encoding, in other words, i is the index of the number of encoding groups, n is the total number of encoding groups, and n is a positive integer greater than 0.

[0088] In order to make the image abnormal region detection model based on variational autoencoder more suitable for abnormality detection of gastrointestinal endoscopic lesion tasks, the patent of this invention makes the following improvements to the loss function of the image abnormal region detection model for training: introducing a weighting mechanism to increase the emphasis on high-incidence areas of lesions.

[0089] In a preferred implementation of this embodiment, the high-incidence area of ​​lesions in the gastrointestinal endoscopy sample image is marked, and a high-incidence area mask is introduced; the area other than the high-incidence area of ​​lesions in the gastrointestinal endoscopy sample image is recorded as the background area.

[0090] Reconstruction loss L recon Including reconstruction loss in lesion-prone areas and background region reconstruction loss

[0091]

[0092] α is the coefficient for controlling the reconstruction loss weight of the lesion-prone areas, (1-α) is the coefficient for controlling the reconstruction loss weight of the background area, α is greater than 0.5 and α is less than 1; higher weights are applied to lesion-prone areas so that the model pays more attention to these areas during the reconstruction process. Lesion-prone areas can be specified by experienced gastroscopy experts as prior information.

[0093] In addition, high-level perceptual loss is used when calculating reconstruction loss: perceptual loss can better capture key features such as lesions by comparing images in a high-level feature space rather than just a pixel-level comparison, and can compare the differences between the input image and the reconstructed image in the feature space. The high-level perceptual loss calculation formula is as follows:

[0094]

[0095] in, represents the feature map corresponding to the i-th layer in the encoder or decoder, i is the layer index of the encoder or decoder, x is the input image, To reconstruct the image, ||.|| 2 Indicates second normal form.

[0096] In summary, the final loss function is as follows:

[0097]

[0098] Reconstruct the loss for high-incidence areas of lesions under high-level perception, Reconstruction loss for high-level perceived background regions.

[0099] The ability of the image abnormality area detection model to detect image abnormalities can timely detect rare lesion areas that are easily overlooked during gastrointestinal endoscopy. While solving patients' health risks, it also increases the hospital's gastrointestinal rare disease data, which helps promote research on rare gastrointestinal diseases.

[0100] S4. Record the gastrointestinal endoscopy image with abnormal region identification result as an abnormal region as an abnormal image, input the abnormal image into the lesion classification model, and the lesion classification model analyzes the abnormal image and outputs the lesion prediction result.

[0101] The steps of the lesion classification model for abnormal images include:

[0102] S41, extracting features of the abnormal image, and obtaining recognition results of the abnormal image by classifying the image into predefined categories; the recognition results include abnormal images as known lesions and abnormal images as unknown lesions:

[0103] S42, analyzing the recognition result: if the recognition result is that the abnormal image is a known lesion, the lesion prediction result includes the lesion category, the probability corresponding to the lesion category, and the mask image of the lesion area in the abnormal image;

[0104] If the recognition result is that the abnormal image is an unknown lesion, the lesion prediction result includes a mask image of the lesion area in the abnormal image.

[0105] In this embodiment, the lesion categories include early gastric cancer, gastric cancer, benign gastric ulcer, malignant gastric ulcer, gastritis, esophagitis, early esophageal cancer, esophageal cancer, enteritis, early intestinal cancer, intestinal cancer, etc.

[0106] The abnormal image is input into the trained lesion classification model, which analyzes the abnormal image and identifies which category the lesion in the abnormal image belongs to, including early gastric cancer, gastric cancer, benign gastric ulcer, malignant gastric ulcer, gastritis, esophagitis, early esophageal cancer, esophageal cancer, enteritis, early intestinal cancer, intestinal cancer and unknown lesions, and obtains the probability value of the lesion.

[0107] When the lesion classification model detects that the abnormal area of ​​the gastrointestinal endoscopic image is a known lesion, the lesion category and the corresponding probability of the lesion category are displayed in the lesion prediction result, which can provide a reference for the physician and help the physician to judge the lesions in the gastrointestinal endoscopic image; in addition, the lesion prediction result also includes a lesion area marking image (that is, the lesion area is marked with a mask in the original gastrointestinal endoscopic image), which can prompt the physician the location of the lesion area. When the physician makes further judgment on the lesion condition, he or she can take a biopsy at the location indicated in the lesion area marking image for pathological analysis, which can effectively assist gastrointestinal endoscopists in making judgments on various lesions.

[0108] For unknown lesions, the lesion prediction results include a mask image of the lesion area in the abnormal image. The gastroenterologist collects biopsy samples corresponding to the unknown lesion area based on the prompts of the mask image and conducts further pathological analysis to improve the stability and accuracy of the diagnostic results.

[0109] In a preferred implementation of this embodiment, the gastrointestinal endoscopy image analysis method further includes training a lesion classification model, and the specific training steps include:

[0110] S431, collecting M lesion sample images containing known lesions, and labeling the lesion sample images according to lesion types.

[0111] Wherein, M is a positive integer greater than 1. The images of known lesions are generally retained in the hospital database and are classified and stored according to the lesion category. Therefore, the data of known lesions are easy to obtain and label classified.

[0112] S432, sending the collected M lesion sample images to the classification network for training. In each training, the loss function is determined according to the output result of the lesion classification model.

[0113] The total number of categories of the lesion classification model is X+1, of which the number of categories of known lesions is X, and the other type is other lesions, that is, unknown lesions; therefore, the total number of categories is X+1.

[0114] The classification network model can use currently popular classification networks such as VggNet, ResNet, MobileNet, EfficientNet, etc.

[0115] VGGNet stands for Visual Geometry Group Network. VGGNet is a deep learning model that features smaller convolution kernels and stacked convolution layers, which can extract more image features while controlling the number of parameters.

[0116] ResNet is the abbreviation of Residual Network. ResNet solves the gradient vanishing problem in deep networks by introducing residual learning, making it possible to train deeper networks. The core idea is to add the input directly to the output of the layer through skip connections to form a residual block.

[0117] MobileNet is a lightweight deep neural network designed for mobile and embedded devices. It uses depth-wise separable convolutions to reduce the complexity and computation of the model, making it particularly suitable for resource-constrained devices.

[0118] EfficientNet is an efficient convolutional neural network model that optimizes network width, depth, and resolution through a compound scaling method to achieve optimal performance under given computing resources.

[0119] S434. Optimize the network parameters of the lesion classification model according to the loss function to obtain the final lesion classification model.

[0120] The lesion classification model can effectively identify and detect known lesions, make up for missed lesions caused by physicians' fatigue and negligence, and improve the stability and accuracy of gastrointestinal endoscopy results.

[0121] Based on the same inventive concept, an embodiment of the present invention provides a gastroenteroscopic video analysis system.

[0122] The gastroenteroscopic video analysis system of the present invention can be loaded in an electronic device. According to the functions to be implemented, the gastroenteroscopic video analysis system includes:

[0123] An acquisition module, used to acquire gastrointestinal endoscopy video;

[0124] An image generation module is used to perform frame extraction processing on the gastroenteroscopic video to obtain a number of gastroenteroscopic images;

[0125] An abnormal region recognition module is used to input the gastrointestinal endoscope image into the image abnormal region detection model, and the image abnormal region detection model analyzes the gastrointestinal endoscope image, identifies whether there is an abnormal region in the gastrointestinal endoscope image, and outputs the abnormal region recognition result;

[0126] The lesion prediction module is used to record the gastrointestinal endoscopic image with abnormal area recognition results as an abnormal image, input the abnormal image into the lesion classification model, and the lesion classification model analyzes the abnormal image and outputs the lesion prediction result.

[0127] The module described in the present invention may also be referred to as a unit, which refers to a series of computer program segments that can be executed by a processor of an electronic device and can complete fixed functions, and is stored in a memory of the electronic device.

[0128] The various variations and specific examples of the gastrointestinal endoscopic video analysis method provided in the above embodiment are also applicable to the gastrointestinal endoscopic video analysis system of the present embodiment. Through the above detailed description of the gastrointestinal endoscopic video analysis method, those skilled in the art can clearly know the implementation method of the gastrointestinal endoscopic video analysis system in the present embodiment. For the sake of brevity of the specification, it will not be described in detail here.

[0129] The present application also discloses an electronic device, such as Figure 3 , which is a schematic diagram of the structure of an electronic device for a gastroenteroscopic video analysis method provided by an embodiment of the present invention. The electronic device may include at least one processor 10, a memory 11 connected to the at least one processor for communication, a communication bus 12, and a communication interface 13, and may also include a computer program stored in the memory 11 and executable on the processor 10, such as a program for the gastroenteroscopic video analysis method.

[0130] In some embodiments, the processor 10 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips. The processor 10 is the control core (Control Unit) of the electronic device, and uses various interfaces and lines to connect various components of the entire electronic device, and executes or executes programs or modules stored in the memory 11 (for example, the executed gastroenteroscopic video analysis method, etc.), and calls the data stored in the memory 11 to execute various functions of the electronic device and process data.

[0131] The memory 11 includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a mobile hard disk, a multimedia card, a card-type memory (for example, an SD or DX memory, etc.), a magnetic memory, a disk, an optical disk, etc. In some embodiments, the memory 11 may be an internal storage unit of an electronic device, such as a mobile hard disk of the electronic device. In other embodiments, the memory 11 may also be an external storage device of an electronic device, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Further, the memory 11 may also include both an internal storage unit of the electronic device and an external storage device. The memory 11 can not only be used to store application software and various types of data installed in the electronic device, such as the code of the gastroenteroscopic video analysis method program, but also can be used to temporarily store data that has been output or is to be output.

[0132] The communication bus 12 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is configured to realize connection and communication between the memory 11 and at least one processor 10, etc.

[0133] The communication interface 13 is used for communication between the above-mentioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device and other electronic devices. The user interface may be a display (Display), an input unit (such as a keyboard (Keyboard)), and optionally, the user interface may also be a standard wired interface, a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode, organic light-emitting diode) touch device, etc. Among them, the display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device and to display a visual user interface.

[0134] Figure 3 Only an electronic device with components is shown, and those skilled in the art will understand that Figure 3 The structure shown does not constitute a limitation on the electronic device, and may include fewer or more components than shown, or combine certain components, or arrange the components differently. For example, although not shown, the electronic device may also include a power supply (such as a battery) for supplying power to each component. Preferably, the power supply may be logically connected to at least one processor 10 through a power management device, so that functions such as charging management, discharging management, and power consumption management are implemented through the power management device. The power supply may also include any components such as one or more DC or AC power supplies, recharging devices, power failure detection circuits, power converters or inverters, power status indicators, etc. The electronic device may also include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be repeated here.

[0135] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited by this structure.

[0136] Furthermore, if the module / unit integrated in the electronic device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile.

[0137] The present application embodiment provides a computer-readable storage medium, for example, including: any entity or device, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM) that can carry the computer program code. The computer-readable storage medium stores a computer program that can be loaded by a processor and execute the gastroenteroscopic video analysis method of the above embodiment.

[0138] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", "an implementation", "a preferred implementation" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0139] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

Claims

1. A gastroenteroscopic video analysis method, characterized in that: Obtain gastrointestinal endoscopy video; Perform frame extraction processing on the gastroenteroscopic video to obtain a number of gastroenteroscopic images; The gastrointestinal endoscopy image is input into the image abnormal region detection model, and the image abnormal region detection model analyzes the gastrointestinal endoscopy image, identifies whether there is an abnormal region in the gastrointestinal endoscopy image, and outputs the abnormal region recognition result; The gastrointestinal endoscopy image with abnormal region recognition result as the abnormal region is recorded as an abnormal image, and the abnormal image is input into the lesion classification model. The lesion classification model analyzes the abnormal image and outputs the lesion prediction result.

2. The gastroenteroscopic video analysis method according to claim 1, characterized in that: The image abnormal region detection model is a variational autoencoder. The image abnormal region detection model processes gastrointestinal endoscopy images in the following steps: Map the gastrointestinal endoscopy image x to a low-dimensional latent space to obtain latent variables, and generate a reconstructed gastrointestinal endoscopy image x based on the latent variables. ’ , and obtain the reconstructed gastrointestinal endoscopy image x ’ The difference score of each pixel between the gastrointestinal endoscopy image x; Compare the difference score to the preset difference score threshold: If the difference score exceeds the preset difference score threshold, the pixel point position corresponding to the difference score is an abnormal area; If the difference score does not exceed the preset difference score threshold, the pixel point position corresponding to the difference score is a normal area; The abnormal area recognition result is determined based on the comparison result of the difference score of each pixel point with the preset difference score threshold.

3. The gastroenteroscopic video analysis method according to claim 2, characterized in that: The method further comprises: The abnormal area of ​​the gastrointestinal endoscope image is integrated, and the image range corresponding to the abnormal area is intercepted from the gastrointestinal endoscope image using a rectangular frame to obtain an abnormal image.

4. The gastroenteroscopic video analysis method according to any one of claims 1 to 3, characterized in that: The steps of the lesion classification model for abnormal images include: The features of abnormal images are extracted, and the images are classified into predefined categories to obtain the recognition results of abnormal images; the recognition results include abnormal images as known lesions and abnormal images as unknown lesions: If the recognition result is that the abnormal image is a known lesion, the lesion prediction result includes the lesion category, the probability corresponding to the lesion category, and the mask image of the lesion area in the abnormal image; If the recognition result is that the abnormal image is an unknown lesion, the lesion prediction result includes a mask image of the lesion area in the abnormal image.

5. The gastroenteroscopic video analysis method according to claim 2, characterized in that: The method also includes a step of training an image abnormal region detection model: A training data set is obtained, wherein the training data set includes a number of gastrointestinal endoscopy sample images without lesions; Construct the network structure of the abnormal region detection model of the image to be trained; The network of the abnormal region detection model of the image to be trained is trained using the training data set. In each training, the loss function is determined according to the output result of the abnormal region detection model of the image to be trained; The network parameters of the image abnormal region detection model to be trained are optimized according to the loss function to obtain the final image abnormal region detection model.

6. The gastroenteroscopic video analysis method according to claim 5, characterized in that: Loss function L VAE Including the reconstruction loss L recon and KL divergence loss L KL , the expression of the loss function is L VAE =L recon +β·L KL , β is a hyperparameter that controls the KL divergence weight.

7. The gastroenteroscopic video analysis method according to claim 6, characterized in that: The method further includes: marking the high-incidence area of ​​lesions in the gastrointestinal endoscope sample image, recording the area other than the high-incidence area of ​​lesions in the gastrointestinal endoscope sample image as the background area, and reconstructing the loss L recon Including reconstruction loss in lesion-prone areas and background region reconstruction loss α is the coefficient for controlling the reconstruction loss weight of the lesion-prone area, (1-α) is the coefficient for controlling the reconstruction loss weight of the background area, and α is greater than 0.5 and α is less than 1.

8. A gastroenteroscopic video analysis system, used to implement the gastroenteroscopic video analysis method according to any one of claims 1 to 7, characterized in that: include: An acquisition module, used to acquire gastrointestinal endoscopy video; An image generation module is used to perform frame extraction processing on the gastroenteroscopic video to obtain a number of gastroenteroscopic images; An abnormal region recognition module is used to input the gastrointestinal endoscope image into the image abnormal region detection model, and the image abnormal region detection model analyzes the gastrointestinal endoscope image, identifies whether there is an abnormal region in the gastrointestinal endoscope image, and outputs the abnormal region recognition result; The lesion prediction module is used to record the gastrointestinal endoscopic image with abnormal area recognition results as an abnormal image, input the abnormal image into the lesion classification model, and the lesion classification model analyzes the abnormal image and outputs the lesion prediction result.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor (10); and, a memory (11) communicatively connected to the at least one processor (10); Wherein, the memory (11) stores a computer program executable by the at least one processor (10), and the computer program is executed by the at least one processor (10) so that the at least one processor (10) can execute the gastrointestinal endoscopic video analysis method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer storage medium stores a computer program; when the computer program is executed by the processor, the gastroenteroscopic video analysis method according to any one of claims 1 to 7 is implemented.

Images