A deep learning-based in-vivo gastroscope residence time detection method

Abstract

The application discloses a gastroscope in-vivo residence time detection method based on deep learning, relates to the technical field of gastroscope detection, and comprises the following steps: acquiring an in-vivo and in-vitro environment two-classification data set; constructing a two-classification network based on a Vision Transformer; training the two-classification network based on the Vision Transformer by using the in-vivo and in-vitro environment two-classification data set; for a complete gastroscope video frame sequence to be detected for gastroscope in-vivo residence time, extracting a frame as a representative frame every certain number of intervals, feeding all the representative frames into the trained two-classification network based on the Vision Transformer for classification, searching for the last group of in-vivo frame sequence numbers Id_a from in-vitro to in-vivo and the last group of in-vivo frame sequence numbers Id_b from in-vivo to in-vitro, and calculating the gastroscope in-vivo residence time. The application has high automation processing level, does not need additional manual processing in the whole process, can greatly reduce workload and improve time efficiency, and can avoid missing and misreading caused by careless artificial retrospective observation in the traditional method.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of gastroscopy detection, in particular to a method for detecting in-vivo residence time of gastroscopy based on deep learning. BACKGROUND

[0002] Gastroscopy detection is a common medical examination method, which inserts a flexible gastroscope into the oral cavity, esophagus, stomach, duodenum and other digestive tract sites of the patient for examination to observe the lesions and abnormalities of the digestive tract. It can be used for monitoring or diagnosing gastric inflammation, gastric ulcer, esophageal cancer, gastric cancer and other digestive tract diseases, as well as digestive tract bleeding, foreign bodies, inflammation, etc. The length of the in-vivo residence time of gastroscopy will affect the effect of gastroscopy detection, so the in-vivo residence time needs to be recorded to determine the standardization and effectiveness of this detection.

[0003] The traditional gastroscopy detection time measurement method generally measures by reviewing the collected gastroscopy video, and the in-vivo timing starts from the last time entering the patient's oral cavity and ends when leaving the oral cavity. The gastroscopy video is usually recorded from the external environment, and there is a possibility of multiple entering and leaving the oral cavity. The traditional timing method needs manpower to observe second by second, which is time-consuming and laborious, and is prone to miss the entering and leaving the oral cavity segments, resulting in timing errors. SUMMARY

[0004] The present application provides a method for detecting the in-vivo residence time of gastroscopy based on a medical scene classification network of Vision Transformer, which saves a lot of manpower and time resources, avoids the defects of the existing manual review recording detection time method, such as long time consumption and timing errors, and efficiently and accurately records the in-vivo residence detection time of gastroscopy.

[0005] The technical scheme adopted by the present application is as follows:

[0006] A method for detecting the in-vivo residence time of gastroscopy based on deep learning, comprising the following steps:

[0007] S1, pre-processing a plurality of collected gastroscopy videos to obtain a data set of in-vivo and in-vitro environment binary classification;

[0008] S2, constructing a binary classification network based on Vision Transformer, which has a structure from the input side to the output side, including an input layer, a Patch Embedding layer, a Positional Encoding layer, a Transformer Encoder layer, a global average pooling layer, a Dropout layer and an output layer, and the loss function used is

[0009] loss=-y×log(p)-(1-y)×log(1-p)

[0010] wherein y represents a true label, 0 or 1, 0 represents an in-vitro frame, 1 represents an in-vivo frame, p is a probability that the model predicts the input image belongs to class 1, and in the process of training the network, the parameters in the network are updated by back propagation using an Adam optimizer to minimize the value of the loss function;

[0011] S3, training a binary classification network based on Vision Transformer using the in-vivo and in-vitro environment binary classification dataset;

[0012] S4, for a complete gastroscope video frame sequence to be detected for gastroscope in-vivo residence time, a frame is extracted every certain number of intervals as a representative frame, and all representative frames are sent to the trained binary classification network based on Vision Transformer for classification to search for the last group of in-vivo frame sequence numbers Id_a from in-vitro to in-vivo, and the last group of in-vivo frame sequence numbers Id_b from in-vivo to in-vitro;

[0013] S5, calculating the gastroscope in-vivo residence time according to the formula (Id_a-Id_b) / (FPS), wherein FPS represents the frame rate of the original video.

[0014] In a preferred embodiment of the present application, step S1 specifically comprises: for the gastroscope video data, selecting a gastroscope frame every 6 frames in each frame sequence, identifying the device model using an OCR model for the first frame in each sequence, cropping the sequence according to the imaging position corresponding to the device model to obtain a gastroscope image and uniformly reducing resize to 256x256, then classifying the gastroscope image after cropping and reducing, giving the labels of in-vivo frames and in-vitro frames, and then putting them into the corresponding folders respectively to obtain the in-vivo and in-vitro environment binary classification dataset.

[0015] In a preferred embodiment of the present application, in the in-vivo and in-vitro environment binary classification dataset, the proportion of in-vivo frames to in-vitro frames is 7:3.

[0016] In a preferred embodiment of the present application, in step S2, the Patch Embedding layer is used to divide the gastroscope imaging area image in the in-vivo and in-vitro environment binary classification dataset into a plurality of small blocks with a size of patch_size x patch_size using convolution, and convert each small block into a vector token, and the dimension of each token is embedding_dim.

[0017] In a preferred embodiment of the present application, the Transformer Encoder layer comprises a plurality of Transformer Encoder Blocks, each Block comprising a multi-head self-attention layer and a feed-forward neural network layer, and a residual connection and normalization are performed on the input and output of each Block.

[0018] In a preferred embodiment of the present application, the global average pooling layer is used to average pool the features of all vectors to obtain a global feature vector.

[0019] In a preferred embodiment of the present application, the Dropout layer is used to randomly inactivate the output of a portion of neurons.

[0020] In a preferred embodiment of the present application, in step S3, in the data set for in-vivo and in-vitro environment classification, 80% of the in-vivo frame and in-vitro frame data are used as the training set, 10% of the in-vivo frame and in-vitro frame data are used as the validation set, and 10% of the in-vivo frame and in-vitro frame data are used as the test set, and the training effect of the VisionTransformer-based binary classification network is comprehensively evaluated using the precision, recall and F1 value three indicators.

[0021] Compared with the prior art, the present application has the following advantages:

[0022] 1) The level of automation is high, and the entire process does not require additional manual processing, which can greatly reduce the workload and improve time efficiency. The end-to-end framework we designed does not require any additional manual operation, and the entire process from input video to output in-vivo residence time is automatically completed by the framework, and has a fast processing speed.

[0023] 2) It can avoid the problem of overlooking and misreading caused by careless human retrospective observation in traditional methods. Our framework analyzes the video frame by frame, and the recognition accuracy of in-vivo and in-vitro classification is as high as 99.7%, which can effectively avoid the problem of overlooking and misreading.

[0024] 3) The OCR model used in the present application can accurately identify the device model and match the model to obtain the corresponding imaging position. Therefore, our method is suitable for all types of gastroscopy equipment and does not require manual adjustment of the cropping area.

[0025] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the following embodiments of the present application are described in detail below, and the accompanying drawings are described as follows. BRIEF DESCRIPTION OF DRAWINGS

[0026] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiments will be briefly introduced as follows. It should be understood that the following drawings only show some of the embodiments of the present application, and therefore should not be regarded as a limitation on the scope. For those of ordinary skill in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0027] Figure 1 Structure diagram of the Vision Transformer binary classification network used for the binary classification of the present application;

[0028] Figure 2 Structure diagram of the Transformer encoding module in the present application;

[0029] Figure 3 Structure diagram of the MLP module in the present application. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, but not all the embodiments.

[0031] The present application provides a gastroscope in-vivo residence time detection method based on a medical scene classification network of Vision Transformer. The method is based on pre-processing of multiple collected gastroscope videos, and uses the data set for training the model of in-vivo and in-vitro environment binary classification. The method uses the advantage of Vision Transformer that can better obtain global information of images and the high-precision performance in image classification tasks, and uses the network model with trained weights and parameters to fully extract global features of gastroscope images for classification. Then, the time of in-vivo detection is calculated through the classified frame sequence, so as to realize the task of efficient and accurate detection of in-vivo residence time.

[0032] The style of the gastroscope video is affected by the model of the collection device. The imaging of different devices is in different positions of the video frame, and different lenses, light sources and image processing technologies will affect the imaging effect. Therefore, the device model is recognized by using an OCR model for text recognition, the imaging area is segmented according to the device model and pre-processed, the in-vivo frame and the in-vitro frame are manually classified to make a data set for training the binary classification network of Vision Transformer. The trained model is used to classify the test gastroscope video frame sequence, and the time interval of the last entry and exit cavity is calculated.

[0033] The present application specifically includes the following steps:

[0034] Step 1: We contacted hospitals and collected multiple videos of gastroscopy tests, and then processed the videos frame by frame. In each frame sequence, we selected one gastroscopy frame by dividing 6 frames. We used an OCR recognition model to identify the device model in the first frame of each sequence, cropped the gastroscopy image, and reduced it to an appropriate size. Then we manually classified the cropped image and gave it a label of in-vivo or in-vitro frame.

[0035] Since the imaging positions of different devices may not be the same, and the non-imaging area may interfere with the detection of the classification model. Therefore, we used a pre-trained OCR recognition model to identify the device model in the non-imaging area, and cropped the gastroscopy image according to the imaging position corresponding to the device model. Since the imaging area of a single device model is fixed and usually square, we only selected the first frame to identify the device model for the entire sequence. The shape of the cropping box is square, and the cropped sequence will be resized to 256x256.

[0036] We manually classified the processed images into in-vivo and in-vitro frames, and put them into corresponding folders, respectively, to obtain the in-vivo and in-vitro environment binary classification dataset, which was used to train the binary classification network based on Vision Transformer. The ratio of in-vivo and in-vitro frames is 7:3.

[0037] Step 2: Design a binary classification network based on Vision Transformer, as shown in Figure 1 The structure and working process are as follows:

[0038] Input layer: The network receives a single gastroscopy frame as input, crops the gastroscopy imaging area, and scales the size of the area to a 256x256 RGB image, matching the processed training data. The processed image is used as input for the subsequent steps.

[0039] Patch Embedding operation: In this operation, the input layer output image is divided into multiple small blocks with the size of patch_size x patch_size using convolution, and each small block is converted into a vector token. The dimension of each token is embedding_dim. Here we input the processed endoscopic imaging area picture size of 256x256, and the patch_size is set to 16. Each image will be blocked into 256x256 / 16x16 = 256 patches, and after the image block linear projection operation, the pixels in each patch are flattened into a one-dimensional vector with a size of embedding_dim = 16x16x3 = 768. That is, there are a total of 256 tokens, and the dimension of each token is 768. Therefore, the dimension after this step is 256x768. Subsequently, the 256 tokens will be subjected to a position encoding operation.

[0040] Position encoding operation: In order to preserve the spatial position information of the image, the network adds a position encoding vector to each of the 256 tokens. This process is implemented using a sine-cosine function encoding. Here, the 256 tokens with a dimension of 768 obtained in the previous step have a dimension of 257x768 after adding the position encoding, i.e., the output is 257 token vectors.

[0041] Transformer encoder layer: The 257 token vectors processed by the position encoding operation are input into the Transformer encoder layer. This layer contains 4 Transformer encoding modules, each of which contains two layer normalization operations, a multi-head self-attention layer, and an MLP module. The network structure of the Transformer encoding module is shown in Figure 2 , and the network structure of the MLP module is shown in Figure 3 . After the first layer normalization operation of the 257 token vectors, the weight of each token is obtained through the multi-head self-attention layer, and then added to the corresponding token. The 257 tokens after addition are subjected to layer normalization operation again, and finally input into the MLP module. In the MLP module, first pass through the fully connected layer, the overall size of the 257 tokens changes from 257x768 to 257x3072, then pass through the GELU activation function, and then pass through the Dropout layer to reduce overfitting. Next, pass through the second fully connected layer, the overall size changes from 257x3072 back to 257x768, pass through the second Dropout layer, and output 257 tokens with a size of 768 to the MLP classification layer.

[0042] MLP classification layer: The 257x768 tokens of the final output of the Transformer encoder layer are input into the MLP classification layer. After a layer normalization operation, the output of the class token in the 257 tokens is extracted to obtain a 1x768 vector. This step is achieved by using a slicing operation to separate the class information. Then the 1x768 vector is passed through a fully connected layer and a tanh activation function, and finally through a second fully connected layer to obtain a 1x768 vector.

[0043] Output layer: Since our classification task is binary classification, the output layer of our network only contains one neuron and a sigmoid activation function to convert the 1x768 dimensional global feature vector output by the MLP classification layer into a probability, which represents the probability that the sample belongs to the positive sample (in vivo). We take the probability that the input image belongs to the positive sample (in vivo) as the classification criterion, greater than 50% is determined as a positive sample (in vivo), and less than 50% is determined as a negative sample (ex vivo), and finally output the class of in vivo or ex vivo.

[0044] Loss function: During training, the network we designed adjusts and calibrates the model weights through backpropagation of the loss function, checks the output of the output layer with the true value, and thus achieves the effect of optimizing the model. We choose the binary cross-entropy loss function as the loss function of the network. Its formula is:

[0045] loss = -y x log(p) - (1-y) x log(1-p)

[0046] Where y represents the true label (0 or 1), 0 represents the ex vivo frame, 1 represents the in vivo frame, and p is the probability that the model predicts the input image to belong to class 1 (in vivo frame). During training, the parameters in the network are updated through backpropagation to minimize the value of the loss function, where we choose the Adam optimizer to update the parameters in the network.

[0047] Step 3: Train the binary classification network based on Vision Transformer using the in vivo and ex vivo environment binary classification dataset.

[0048] We divide the in vivo and ex vivo environment binary classification dataset prepared in step 1 into training, validation, and test sets, with the training set containing 80% positive and negative samples (in vivo frames, ex vivo frames), the validation set containing 10% positive and negative samples, and the test set containing 10% positive and negative samples. We use precision, recall, and F1 score to comprehensively evaluate the performance of the binary classification model, ensuring that the classification accuracy of the model on the test set is above 98.5%. The trained classification model is used for in vivo frame time calculation.

[0049] Step 4: design an algorithm to calculate the in-vivo frame time, realize the complete end-to-end in-vivo detection time framework.

[0050] For a complete gastroscopy video frame sequence, we extract a frame as a representative frame every certain number of intervals, set the sequence number of all representative frames in the sequence from 0 to 1 unit increment in turn, and send them into the binary classification model for classification to obtain the same number of in-vivo (out-of-vivo) determination results as the sequence image, and the picture sequence number corresponding to each result. Search for the last set of in-vivo frame sequence numbers Id_a from out-of-vivo to in-vivo, and the last set of in-vivo frame sequence numbers Id_b from in-vivo to out-of-vivo,

[0051] Step 5: calculate the in-vivo residence time of gastroscopy, the calculation formula is

[0052] (Id_a-Id_b)*alpha inter / (FPS)

[0053] Wherein, alpha inter represents the number of original frames between adjacent representative frames plus 1, and FPS represents the frame rate of the original video, and the calculated time unit is second.

[0054] Through the above five steps, the in-vivo detection time in gastroscopy can be obtained.

[0055] The above only describes the preferred embodiments of the present application and is not used to limit the present application. For those skilled in the art, the present application can have various changes and variations. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A deep learning-based in-gastroscope residence time detection method, characterized in that, Includes the following steps: S1. Preprocess multiple acquired gastroscopy videos to obtain a dataset for binary classification of in vivo and in vitro environments; S2. Construct a binary classification network based on Vision Transformer. Its structure, from input to output, includes an input layer, a Patch Embedding layer, a Positional Encoding layer, a Transformer Encoder layer, a global average pooling layer, a Dropout layer, and an output layer. The loss function used is... loss=-y×log(p)-(1-y)×log(1-p) Where y represents the true label, which is 0 or 1, 0 represents the in-frame and 1 represents the in-frame, and p is the probability that the model predicts the input image belongs to class 1. During the training of the network, the parameters in the network are updated by backpropagation using the Adam optimizer to minimize the value of the loss function. S3. A Vision Transformer-based binary classification network is trained using a dataset for binary classification of in vivo and in vitro environments. S4. For the complete gastroscopy video frame sequence of the gastroscopy residence time to be detected, extract a frame at certain intervals as a representative frame, send all representative frames into a trained binary classification network based on Vision Transformer for classification, and search for the last set of in vivo frame sequence numbers Id_a from outside to inside, and the last set of in vivo frame sequence numbers Id_b from inside to outside. S5. Calculate the intraoperative time of the endoscope according to the formula (Id_a-Id_b) / (FPS), where FPS represents the frame rate of the original video. 2.The deep learning-based in-vivo gastroscope retention time detection method of claim 1, wherein, Step S1 specifically includes: For gastroscopy video data, select one gastroscopy frame from 6 frames in each frame sequence, select the first frame in each sequence and use an OCR model to identify the device model, crop the sequence according to the imaging position corresponding to the device model to obtain the gastroscopy image and uniformly resize it to 256x256, then classify the cropped and reduced gastroscopy image, give labels for in vivo frames and out vivo frames, and then put them into the corresponding folders to obtain a dataset for binary classification of in vivo and out vivo environments. 3.The deep learning-based in-vivo gastroscope retention time detection method of claim 2, wherein, In the dataset of binary classification of in vivo and in vitro environments, the ratio of in vivo frames to in vitro frames is 7:

3.

4. The method for detecting the in vivo residence time of a gastroscope based on deep learning according to claim 2, characterized in that, In step S2, the Patch Embedding layer is used to divide the image of the gastroscopy imaging region in the dataset of binary classification of in vivo and in vitro environments into multiple small blocks of size patch_size x patch_size using convolution, and convert each small block into a vector token, with each token having the dimension of embedding_dim.

5. The method for detecting the in vivo residence time of a gastroscope based on deep learning according to claim 4, characterized in that, The Transformer Encoder layer contains multiple Transformer Encoder Blocks. Each Block contains a multi-head self-attention layer and a feedforward neural network layer. Residual connections and normalization are performed in the input and output of each Block.

6. The method for detecting the in vivo residence time of a gastroscope based on deep learning according to claim 5, characterized in that, The global average pooling layer is used to average the features of all vectors to obtain a global feature vector.

7. The method for detecting the in vivo residence time of a gastroscope based on deep learning according to claim 6, characterized in that, Dropout layers are used to randomly deactivate a subset of the neuron's output.

8. The method for detecting the in vivo residence time of a gastroscope based on deep learning according to claim 1, characterized in that, In step S3, in the dataset for binary classification of in vivo and in vitro environments, 80% of the in vivo and in vitro frames are used as the training set, 10% of the in vivo and in vitro frames are used as the validation set, and 10% of the in vivo and in vitro frames are used as the test set. The training effect of the binary classification network based on Vision Transformer is comprehensively evaluated using three metrics: precision, recall, and F1 score.

Images