Gastroscope in-vivo detection time measuring method

Abstract

The invention discloses a gastroscope in-vivo detection time measuring method, and relates to the technical field of gastroscope detection, and the gastroscope in-vivo detection time measuring method comprises the following steps: obtaining a data set of in-vivo and in-vitro environment dichotomy; the method comprises the following steps of: constructing a binary classification network based on a Vision Transform; training a dichotomy network based on Vision Transformer by adopting a data set of dichotomy of in-vivo and in-vitro environments; for a complete gastroscope video frame sequence of the in-vivo retention time of the gastroscope to be detected, extracting a frame at a certain interval as a representative frame, sending all the representative frames into the trained Vision Transform-based binary classification network for classification, searching the last group of in-vivo frame serial numbers Ida from the outside to the inside, and determining the in-vivo retention time of the gastroscope to be detected. And the last group of in-vivo frame serial numbers Idb from in vivo to in vitro; and calculating the retention time in the gastroscope body. The automatic processing level is high, extra manual processing is not needed in the whole process, the workload can be greatly reduced, the time efficiency can be improved, and the problems of missed watching and wrong watching caused by careful manual retrospective observation in a traditional method can be solved.

Description

Technical Field

[0001] The present invention relates to the technical field of gastroscope detection, and more specifically, to a method for measuring the in-vivo detection time of a gastroscope. Background Art

[0002] Gastroscope detection is a commonly used medical examination method. By inserting a flexible gastroscope into the digestive tract parts of a patient, such as the oral cavity, esophagus, stomach, and duodenum, it is used to observe the lesions and abnormalities in the digestive tract. It can be used to monitor or diagnose digestive tract diseases such as gastritis, gastric ulcer, esophageal cancer, and gastric cancer, as well as situations such as digestive tract bleeding, foreign bodies, and inflammation. The length of the in-vivo residence time of the gastroscope will affect the effect of gastroscope detection. Therefore, it is necessary to record the in-vivo residence time to determine the standardization and effectiveness of this detection.

[0003] The traditional method for measuring the gastroscope detection time is generally to review the collected gastroscope videos for measurement. The in-vivo timing starts from the last time it enters the patient's oral cavity and ends when it leaves the oral cavity for the last time. And the gastroscope video is usually recorded starting from the external environment, and there is a possibility of entering and exiting the oral cavity multiple times. The traditional timing method requires manual observation second by second, which is time-consuming and laborious, and it is easy to miss the segments of entering and exiting the oral cavity, resulting in timing errors. Summary of the Invention

[0004] The present invention aims to provide a method for detecting the in-vivo residence time of a gastroscope based on a medical scene classification network of Vision Transformer, so as to save a large amount of human and time resources, avoid the defects of the existing method of manually reviewing and recording the detection time, such as being time-consuming and having timing errors, and efficiently and accurately record the time of the gastroscope staying and detecting in the body.

[0005] The technical solution adopted by the present invention is as follows:

[0006] A method for measuring the in-vivo detection time of a gastroscope includes the following steps:

[0007] S1. Preprocess multiple collected gastroscope videos to obtain a dataset for binary classification of in-vivo and in-vitro environments;

[0008] S2. Construct a binary classification network based on Vision Transformer. Its structure from the input side to the output side includes an input layer, a Patch Embedding layer, a Positional Encoding layer, a Transformer Encoder layer, a global average pooling layer, a Dropout layer, and an output layer. The loss function used is

[0009] loss=-y×log(p)-(1-y)×log(1-p)

[0010] Among them, y represents the true label, which is 0 or 1. 0 represents an external frame, and 1 represents an internal frame. p is the probability that the model predicts that the input image belongs to class 1. During the training of the network, the parameters in the network are updated by backpropagation using the Adam optimizer to minimize the value of the loss function.

[0011] S3. Train a binary classification network based on Vision Transformer using a dataset for binary classification of the internal and external environments.

[0012] S4. For the complete sequence of gastroscope video frames whose in-vivo residence time is to be detected, extract one frame as a representative frame at regular intervals. Send all the representative frames into the trained binary classification network based on Vision Transformer for classification, and search for the serial number Id_a of the last group of internal frames from outside to inside and the serial number Id_b of the last group of internal frames from inside to outside.

[0013] S5. Calculate the in-vivo residence time of the gastroscope according to the formula (Id_a - Id_b) / (FPS), where FPS represents the frame rate of the original video.

[0014] In a preferred embodiment of the present invention, step S1 specifically includes: for gastroscope video data, select one gastroscope frame every 6 frames in each frame sequence. In the first frame of each sequence, use the OCR model to identify the device model. Crop the sequence according to the imaging position corresponding to the device model to obtain a gastroscope image and uniformly resize it to 256x256. Then classify the cropped and resized gastroscope image, give the labels of the internal and external frames, and then put them into the corresponding folders respectively to obtain a dataset for binary classification of the internal and external environments.

[0015] In a preferred embodiment of the present invention, in the dataset for binary classification of the internal and external environments, the ratio of internal frames to external frames is 7:3.

[0016] In a preferred embodiment of the present invention, in step S2, the Patch Embedding layer is used to divide the image of the gastroscope imaging area in the dataset for binary classification of the internal and external environments into multiple small blocks of size patch_size x patch_size using convolution, and convert each small block into a vector token, and the dimension of each token is embedding_dim.

[0017] In a preferred embodiment of the present invention, the Transformer Encoder layer includes multiple Transformer Encoder Blocks. Each Block contains a multi-head self-attention layer and a feed-forward neural network layer. Residual connections and normalization are performed on the input and output of each Block.

[0018] In a preferred embodiment of the present invention, the global average pooling layer is used to perform average pooling on the features of all vectors to obtain a global feature vector.

[0019] In a preferred embodiment of the present invention, the Dropout layer is used to randomly deactivate the outputs of a part of the neurons.

[0020] In a preferred embodiment of the present invention, in step S3, in the dataset for binary classification of in-vivo and in-vitro environments, 80% of the in-vivo frame and in-vitro frame data are used as the training set, 10% of the in-vivo frame and in-vitro frame data are used as the validation set, and 10% of the in-vivo frame and in-vitro frame data are used as the test set. Three metrics, namely precision, recall, and F1 value, are used to comprehensively evaluate the training effect of the binary classification network based on Vision Transformer.

[0021] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0022] 1) The level of automation processing is high. The entire process does not require additional manual processing, which can greatly reduce the workload and improve time efficiency. The end-to-end framework we designed does not require any redundant manual operations. The entire process from the input video to the output of the in-vivo residence time is automatically completed by the framework, and it has a fast processing speed.

[0023] 2) It can avoid problems such as missed or incorrect viewing caused by careless retrospective observation in traditional methods. Our framework analyzes the video frame by frame, and the recognition accuracy for binary classification of in-vivo and in-vitro is as high as 99.7%, which can effectively avoid the problems of missed or incorrect viewing.

[0024] 3) The OCR model used in the present invention can accurately identify the device model and obtain the corresponding imaging position by matching the model. Therefore, our method is applicable to all models of gastroscope examination devices without the need for manual adjustment of the cropping area.

[0025] To make the above objects, features, and advantages of the present invention more obvious and understandable, specific embodiments of the present invention are hereinafter given, and in conjunction with the accompanying drawings, the detailed description is as follows. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the accompanying drawings required for use in the embodiments. It should be understood that the following drawings only show certain embodiments of the present invention and should not be regarded as limiting the scope. For those of ordinary skill in the art, without creative efforts, other related drawings can also be obtained based on these drawings.

[0027] Figure 1 The structure diagram of the Vision Transformer binary classification network for binary classification of the present invention;

[0028] Figure 2 The structure diagram of the Transformer encoding module in the present invention;

[0029] Figure 3 The structure diagram of the MLP module in the present invention. Specific implementation manners

[0030] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention.

[0031] The present invention provides a method for detecting the in-vivo residence time of a gastroscope in a medical scene classification network based on Vision Transformer. It preprocesses multiple collected gastroscope videos, makes a dataset for binary classification of in-vivo and in-vitro environments for training the model, and utilizes the advantages of Vision Transformer in better obtaining global image information and its high-precision performance in image classification tasks. Through the network model with trained weights and parameters, it fully extracts the global features of gastroscope images for classification, and then calculates the residence time during in-vivo detection through the classified frame sequence, realizing the task of efficient and accurate detection of in-vivo residence time.

[0032] Regarding the influence of the gastroscope video style by the model of the acquisition device, the imaging of different devices is at different positions in the video frame, and different lenses, light sources, and image processing technologies will all affect the imaging effect. Therefore, we use the OCR model to recognize the device model by text, segment the imaging area according to the device model and perform preprocessing, and manually classify in-vivo frames and in-vitro frames to make a dataset for training the binary classification network of Vision Transformer. Use the trained model to classify the test gastroscope video frame sequence and calculate the time interval of the last entry and exit from the oral cavity.

[0033] The present invention specifically includes the following steps:

[0034] Step 1: We contacted the hospital and collected video data of multiple gastroscopy examinations, and then performed frame segmentation on the videos. In each frame sequence, one gastroscope frame was selected from every six frames. For the first frame in each sequence, the OCR recognition model was used to identify the device model. The sequence was cropped to obtain the gastroscope image and resized to an appropriate size. Subsequently, the cropped images were manually classified to give the labels of in-vivo frames and out-of-vivo frames.

[0035] Since the imaging positions of gastroscopes of different devices may not be the same, and the non-imaging areas will interfere with the detection of the classification model. Therefore, we used a pre-trained OCR recognition model to identify the device model marked in the non-imaging area, and cropped the gastroscope image according to the imaging position corresponding to the device model. Since the imaging area of a single device model is fixed and usually square, we only selected the first frame to identify the device model for cropping the entire sequence. The shape of the cropping frame is square, and the cropped sequence will be uniformly resized to 256x256.

[0036] The processed images were manually classified into in-vivo frames and out-of-vivo frames, and placed in the corresponding folders respectively, thus obtaining a dataset for binary classification of in-vivo and out-of-vivo environments, which was used to train a binary classification network based on Vision Transformer. Among them, the ratio of in-vivo frames to out-of-vivo frames is 7:3.

[0037] Step 2: Design a binary classification network based on Vision Transformer, as Figure 1 shown, and its structure and working process are as follows:

[0038] Input layer: The network receives a single gastroscope frame as input, crops out the gastroscope imaging area, and scales the size of this area to a 256x256 RGB image to match the processed training data. The processed image is used as the input for the subsequent steps.

[0039] Patch Embedding Operation: In this operation, convolution is used to divide the image output by the input layer into multiple small patches of size patch_size x patch_size, and each small patch is converted into a vector token. The dimension of each token is embedding_dim. Here, the size of the processed gastroscope imaging area image we input is 256x256, and patch_size is set to 16. Each image will be divided into 256x256 / 16x16 = 256 patches. After the linear projection operation of the image patches, the pixels in each patch are flattened into a one-dimensional vector, with the dimension embedding_dim = 16x16x3 = 768, that is, there are 256 tokens in total, and the dimension of each token is 768. Therefore, after this step, the dimension is 256x768. Subsequently, these 256 tokens will undergo position encoding operation.

[0040] Position Encoding Operation: To retain the spatial position information of the image, the network adds each of the 256 tokens to a position encoding vector, and this process is implemented using sine and cosine function encoding. Here, after adding the position encoding to the 256 tokens with a dimension of 768 obtained in the previous step, the dimension becomes 257x768, that is, the output is 257 token vectors.

[0041] Transformer Encoder Layer: The 257 token vectors processed by the position encoding operation are input into the Transformer encoder layer. This layer contains 4 Transformer encoding modules, and each encoding module contains two layer normalization operations, a multi-head self-attention layer, and an MLP module. The network structure of the Transformer encoding module is as Figure 2 shown, and the network structure of the MLP module is as Figure 3 shown. After the first layer normalization operation on the 257 token vectors, the weights of each token are obtained through the multi-head self-attention layer, and then added to the corresponding tokens. After the addition, the 257 tokens are subjected to layer normalization operation again, and finally input into the MLP module. In the MLP module, first through the fully connected layer, the overall size of the 257 tokens changes from 257x768 to 257x3072, then through the GELU activation function, and then through the Dropout layer to reduce overfitting. Next, through the second fully connected layer, the overall size changes back from 257x3072 to 257x768, and after the second Dropout layer, 257 tokens with a size of 768 are output to the MLP classification layer.

[0042] MLP Classification Layer: The 257x768 tokens of the final output of the Transformer encoder layer are input into the MLP classification layer. After a layer normalization operation, the output of the class token among the 257 tokens is extracted to obtain a 1x768 vector. This step is achieved by using slicing operations to separately extract the category information. Subsequently, the 1x768 vector passes through a fully connected layer and a tanh activation function, and finally through a second fully connected layer to obtain a 1x768 vector.

[0043] Output Layer: Since our classification task is binary classification, the output layer of our network only contains one neuron and one sigmoid activation function, which is used to convert the 1x768-sized global feature vector output by the MLP classification layer into a probability. This probability represents the probability that the sample belongs to the positive sample (in the body). We use the probability that the input image belongs to the positive sample (in the body) as the classification decision condition. If it is greater than 50%, it is determined as a positive sample (in the body), and if it is less than 50%, it is determined as a negative sample (outside the body). Finally, the output determines the category of in the body or outside the body.

[0044] Loss Function: During the training process, the network we designed adjusts and calibrates the model weights through backpropagation of the loss function, and verifies the output result of the output layer with the true value, thereby achieving the effect of optimizing the model. We choose the binary cross-entropy loss function as the loss function of the network. Its formula is:

[0045] loss = -y × log(p) - (1 - y) × log(1 - p)

[0046] Among them, y represents the true label (0 or 1), 0 represents the frame outside the body, 1 represents the frame in the body, p is the probability that the model predicts that the input image belongs to class 1 (frame in the body). During the training process, the parameters in the network are updated through backpropagation to minimize the value of the loss function. Among them, we choose the Adam optimizer to update the parameters in the network.

[0047] Step 3: Train a binary classification network based on Vision Transformer using a dataset for binary classification of in-body and out-of-body environments.

[0048] We divide the dataset for binary classification of in-body and out-of-body environments made in Step 1. Among them, the training set contains 80% positive and negative samples (frames in the body, frames outside the body), the validation set contains 10% positive and negative samples, and the test set contains 10% positive and negative samples. We use three metrics, precision, recall, and F1 value, to comprehensively evaluate the effect of the binary classification model, ensuring that the classification accuracy of the in-body and out-of-body classification model on the test set reaches more than 98.5%. The trained classification model is used to calculate the in-body frame time.

[0049] Step 4: Design an algorithm to calculate the in-body frame time and implement a complete end-to-end in-body detection time framework.

[0050] For a complete sequence of gastroscope video frames, we extract one frame at regular intervals as representative frames. We set serial numbers for all the representative frames in the sequence starting from 0 and incrementing by 1 in order of acquisition. These frames are sent into a binary classification model for classification to obtain in-body (out-of-body) determination results that are the same number as the sequence images, as well as the picture serial numbers corresponding to each result. Search for the serial number Id_a of the last group of in-body frames from out-of-body to in-body, and the serial number Id_b of the last group of in-body frames from in-body to out-of-body.

[0051] Step 5: Calculate the in-body residence time of the gastroscope. The calculation formula is

[0052] (Id_a - Id_b) * α inter / (FPS)

[0053] where α inter represents the number of original frames between adjacent representative frames plus 1, and FPS represents the frame rate of the original video. The calculated time unit is seconds.

[0054] Through the above five steps, the in-body detection time during gastroscope examination can be obtained.

[0055] The above are only the preferred embodiments of the present invention and are not used to limit the present invention. For those skilled in the art, the present invention can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for measuring the in-vivo detection time of a gastroscope, characterized in that, It includes the following steps: S1. Preprocess multiple collected gastroscope videos to obtain a dataset for binary classification of in-vivo and ex-vivo environments; S2. Construct a binary classification network based on Vision Transformer. Its structure from the input side to the output side includes an input layer, a Patch Embedding layer, a Positional Encoding layer, a Transformer Encoder layer, a global average pooling layer, a Dropout layer, and an output layer. The loss function it uses is loss=-y×log(p)-(1 - y)×log(1 - p) where y represents the true label, which is 0 or 1. 0 represents an ex-vivo frame, 1 represents an in-vivo frame, p is the probability that the model predicts that the input image belongs to class 1. During the process of training the network, through backpropagation, the Adam optimizer is used to update the parameters in the network to minimize the value of the loss function; S3. Use the dataset for binary classification of in-vivo and ex-vivo environments to train the binary classification network based on Vision Transformer; S4. For the complete gastroscope video frame sequence of the gastroscope whose in-vivo residence time is to be detected, extract one frame as a representative frame at regular intervals. Send all the representative frames into the trained binary classification network based on Vision Transformer for classification, and search for the serial number Id_a of the last group of in-vivo frames from ex-vivo to in-vivo, and the serial number Id_b of the last group of in-vivo frames from in-vivo to ex-vivo; S5. Calculate the in-vivo residence time of the gastroscope according to the formula (Id_a - Id_b) / (FPS), where FPS represents the frame rate of the original video.

2. The method for detecting the in-vivo retention time of a gastroscope based on deep learning according to claim 1, wherein Step S1 specifically includes: For gastroscope video data, select one gastroscope frame every 6 frames in each frame sequence. In each sequence, select the first frame and use the OCR model to identify the device model. Crop the sequence according to the imaging position corresponding to the device model to obtain a gastroscope image and uniformly resize it to 256x256. Subsequently, classify the cropped and resized gastroscope image, give the labels of in-vivo frames and ex-vivo frames, and then put them into the corresponding folders respectively to obtain a dataset for binary classification of in-vivo and ex-vivo environments.

3. The method for detecting the in-vivo residence time of a gastroscope based on deep learning according to claim 2, wherein In the dataset for binary classification of in-vivo and ex-vivo environments, the ratio of in-vivo frames to ex-vivo frames is 7:

3.

4. The method for detecting the in-vivo residence time of a gastroscope based on deep learning according to claim 2, wherein In step S2, the Patch Embedding layer is used to divide the image of the gastroscope imaging area in the dataset for binary classification of in-vivo and ex-vivo environments into multiple small blocks of size patch_size x patch_size using convolution, and convert each small block into a vector token, and the dimension of each token is embedding_dim.

5. The method for detecting the in-vivo residence time of a gastroscope based on deep learning according to claim 4, wherein, The Transformer Encoder layer contains multiple Transformer Encoder Blocks. Each Block contains a multi-head self-attention layer and a feed-forward neural network layer. Residual connections and normalization are performed in the input and output of each Block.

6. The method for detecting the in-vivo retention time of a gastroscope based on deep learning according to claim 5, wherein The global average pooling layer is used to perform average pooling on the features of all vectors to obtain a global feature vector.

7. The method for detecting the in-vivo residence time of a gastroscope based on deep learning according to claim 6, wherein, The Dropout layer is used to randomly deactivate the outputs of some neurons.

8. The method for detecting the in-vivo residence time of a gastroscope based on deep learning according to claim 1, characterized in that, In step S3, in the dataset for binary classification of in-vivo and in-vitro environments, 80% of the in-vivo frame and in-vitro frame data are used as the training set, 10% of the in-vivo frame and in-vitro frame data are used as the validation set, and 10% of the in-vivo frame and in-vitro frame data are used as the test set. Three metrics, precision, recall, and F1 value, are used to comprehensively evaluate the training effect of the binary classification network based on Vision Transformer.

Images