Bronchus part identification method and device

The time and spatial information of bronchoscopy videos were extracted through convolutional neural networks, and combined with dense optical flow calculations, the problem of inaccurate identification in bronchoscopy in the prior art is solved, and automatic and accurate bronchial site recognition is achieved.

CN118918083APending Publication Date: 2024-11-08ZHEJIANG HERUI MEDICAL TECH CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202410968742.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-18
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

In the existing bronchoscopy technology, the risks brought by equipment changes and software algorithms require high image clarity and insufficient time dimension, resulting in insufficient accuracy in the identification of bronchial parts.

Method used

By obtaining the historical video collected by bronchoscopy, it is marked and preprocessed, and divided into training sets and test sets. The time and space information of video frames and the motion change information are extracted using a convolutional neural network, and the bronchial output is output by combining the fully connected layer and the classification layer. The probability value of the part is automatically recognized.

Benefits of technology

It realizes fully automatic bronchial site recognition without human intervention, and the identification results are more accurate and the operation process is simplified.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918083A_ABST
    Figure CN118918083A_ABST
Patent Text Reader

Abstract

The invention provides a bronchus part identification method and device, and the method comprises the steps: obtaining a historical video collected through a bronchoscope, marking the historical video as a data set, and dividing the data set into a training set and a test set; inputting the training set into a convolutional neural network for training until the convolutional neural network converges, and obtaining a part recognition model; judging that the test video in the test set is a video acquired by in-vivo detection or in-vitro detection and collected by a bronchoscope, and inputting the test video determined as the in-vivo detection into the part recognition model to obtain a recognition result; by training the part recognition model, the bronchus part can be automatically recognized, and the recognition result is obtained.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of deep learning, and in particular to a bronchial site identification method and device. Background Art

[0002] The bronchus is a tree-like structure with complex internal branches. The current identification of bronchial sites requires doctors to pay attention to the lung images collected by the bronchoscope during the examination, and to remember the examined sites and the sites to be entered for a long time. Inexperienced doctors may get lost during the examination. Existing bronchoscopic auxiliary examination methods can be divided into two types: bronchoscopic hardware equipment and software algorithms.

[0003] In related technologies, bronchoscopy is to set up various additional sensors on the bronchoscope to assist in positioning in the lungs. The computer background can infer the actual position of the bronchoscope at this time through the sensor feedback data. However, bronchoscopy is an invasive examination, and any changes to the equipment need to consider whether it will cause harm to the human body and bring additional risks or harm. The software algorithm collects data from various parts of the bronchus, marks the bifurcations of each part, uses the target detection algorithm or image segmentation algorithm for training, and uses the trained algorithm to infer the current part. However, the software algorithm has high requirements for image clarity, and the time dimension is not precise enough, resulting in inaccurate identification of the bronchial part. Summary of the invention

[0004] The present disclosure provides a bronchial site identification method and device to at least solve the above technical problems existing in the prior art.

[0005] According to a first aspect of the present disclosure, a method for identifying a bronchial part is provided, the method comprising:

[0006] Obtaining historical videos collected through a bronchoscope, annotating the historical videos as a data set, and dividing the data set into a training set and a test set;

[0007] Inputting the training set into a convolutional neural network for training until the convolutional neural network converges to obtain a part recognition model;

[0008] It is determined that the test video in the test set is a video obtained by in vivo detection or in vitro detection collected by a bronchoscope, and the test video determined to be an in vivo detection is input into the part recognition model to obtain a recognition result.

[0009] In one possible implementation manner, after acquiring the historical video collected through the bronchoscope, the method further includes:

[0010] The historical video is preprocessed to obtain a video to be processed.

[0011] In one possible implementation, preprocessing the historical video includes:

[0012] Scaling the picture size of the historical video to a preset size, and using the historical video with the preset picture size as the first video to be processed;

[0013] The duration of the segments of the first video to be processed is obtained, and the first video to be processed whose segment duration is shorter than a preset duration is eliminated to obtain a second video to be processed; and the second video to be processed is determined as a data set.

[0014] In one possible implementation, the training set includes training videos; and the convolutional neural network includes:

[0015] A first branch, a second branch, a fully connected layer, and a classification layer, wherein the first branch and the second branch are both connected to the fully connected layer, and the fully connected layer is connected to the classification layer;

[0016] The first branch is used to extract time information and space information of video frame images of training videos;

[0017] The second branch is used to extract motion change information between adjacent video frame images of the training video;

[0018] The fully connected layer is used to combine the time information and spatial information extracted by the first branch and the motion change information extracted by the second branch to obtain bronchial site feature information;

[0019] The classification layer outputs the probability value of each bronchial part according to the bronchial part characteristic information.

[0020] In one embodiment, the first branch includes a first spatiotemporal convolution layer, a second spatiotemporal convolution layer, a third spatiotemporal convolution layer, a fourth spatiotemporal convolution layer, a fifth spatiotemporal convolution layer and a first pooling layer, which are sequentially connected from top to bottom;

[0021] The second branch includes a downsampling layer, a dense optical flow calculation layer, a sixth spatiotemporal convolution layer, a seventh spatiotemporal convolution layer, an eighth spatiotemporal convolution layer and a second pooling layer connected sequentially from top to bottom;

[0022] The first pooling layer and the second pooling layer are both connected to a fully connected layer, and the fully connected layer is connected to a classification layer.

[0023] In one possible implementation, the determining that the test video in the test set is a video obtained by in vivo detection or in vitro detection collected by bronchoscope includes:

[0024] Get the test video;

[0025] Determining whether the continuous video frame images in the test video are in-vivo detection or out-of-vivo detection based on a preset window length;

[0026] When the continuous video frame images in the preset window length are in vivo detections and the number of continuous video frame images exceeds the preset threshold, it is determined that the current frame image of the test video is an in vivo detection image collected by a bronchoscope; when the continuous video frame images in the preset window length are in vitro detections and the number of continuous video frame images exceeds the preset threshold, it is determined that the current frame image of the test video is an in vitro detection image collected by a bronchoscope.

[0027] In one possible implementation, it further includes:

[0028] Determine whether the video in the training set is a clear image or a blurred image, and determine whether the video detected in the body is a clear image or a blurred image;

[0029] The videos in the training set are determined to be clear images and input into the convolutional neural network for training, and the videos detected in the body are labeled as clear images or blurred images and input into the part recognition model.

[0030] In one possible implementation manner, obtaining the recognition result includes:

[0031] For videos with clear images, identify and output part recognition results;

[0032] For videos with blurred images, the part recognition results are not displayed.

[0033] In one embodiment, the bronchial site includes:

[0034] One or more parts of the pharynx, glottis, trachea, carina, left main bronchus, left upper lobe, left upper lobe proper segment, left upper lobe lingular segment, left lower lobe, left lower lobe basal segment, left lower lobe dorsal segment, right main bronchus, right upper lobe, right intermediate bronchus, right middle lobe, right lower lobe, right lower lobe basal segment, and right lower lobe dorsal segment.

[0035] According to a second aspect of the present disclosure, a bronchial site identification device is provided, the device comprising:

[0036] Obtaining historical videos collected through a bronchoscope, annotating the historical videos as a data set, and dividing the data set into a training set and a test set;

[0037] Inputting the training set into a convolutional neural network for training until the convolutional neural network converges to obtain a part recognition model;

[0038] It is determined that the test video in the test set is a video obtained by in vivo detection or in vitro detection collected by a bronchoscope, and the test video determined to be an in vivo detection is input into the part recognition model to obtain a recognition result.

[0039] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0040] at least one processor; and

[0041] a memory communicatively connected to the at least one processor; wherein,

[0042] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the present disclosure.

[0043] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method described in the present disclosure.

[0044] The bronchial site recognition method and device disclosed in the present invention divides historical videos collected by bronchoscope into training sets and test sets, trains a convolutional neural network with the training set to obtain a site recognition model, and then tests the site recognition model with the test set to obtain a recognition result. The present invention fully automatically recognizes the bronchial site without human intervention, is simple to operate and has more accurate recognition results.

[0045] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, in which:

[0047] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts.

[0048] Figure 1 The schematic diagram shows the implementation process of the bronchial part identification method of the embodiment of the present disclosure. Figure 1 ;

[0049] Figure 2 The schematic diagram shows the implementation process of the bronchial part identification method of the embodiment of the present disclosure. Figure 2 ;

[0050] Figure 3 A schematic diagram of the structure of a bronchial part identification device according to an embodiment of the present disclosure is shown;

[0051] Figure 4 A schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0052] In order to make the purpose, features, and advantages of the present disclosure more obvious and easy to understand, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0053] At present, there are certain problems both in terms of bronchoscope hardware equipment and software algorithms, as follows:

[0054] The shortcomings of the improvement of bronchoscope hardware equipment are that bronchoscopy is an invasive examination, and any changes to the equipment need to consider whether it will cause harm to the human body, and any changes must be reviewed and certified by relevant agencies to ensure that invasive examinations of patients do not bring additional risks or harm. In addition, when calculating sensor feedback data in the background, it is usually necessary to set some empirical values, such as the length of each bronchial tube, etc. The shape and size of each person's bronchi vary greatly, and some patients have variations in their bronchi, such as the possibility of bronchial loss or hyperplasia, which leads to a large cumulative error in the output results.

[0055] The shortcomings of improving software algorithms are that when constructing a data set, high image clarity is required, and most of the images used are of standard morphology, which may lead to overly idealized modeling and inability to cope with the diversity of real inspections. Due to the rotation and movement of the lens during the inspection, it is extremely difficult to obtain images of parts that are similar to the standard morphology. If modeling is based on two-dimensional image information, the two-dimensional model of the timing information contained in the video clip cannot be extracted, and the information in the time dimension is ignored. In addition, during the bronchoscopy, as the operator rotates the lens and moves, the inspection part will move in the picture, and the general model cannot capture the movement information of the part.

[0056] The following describes the bronchial site identification method and device provided by the present application in conjunction with the accompanying drawings.

[0057] like Figure 1 As shown, the present application provides a method for identifying bronchial sites, the method comprising:

[0058] S101, obtaining historical videos collected through a bronchoscope, annotating the historical videos as a data set, and dividing the data set into a training set and a test set;

[0059] The historical videos collected in this application were obtained by collecting bronchoscopy videos in the hospital. The purpose of this application is to identify the corresponding parts of each bronchus in the video. It is understandable that when the bronchus needs to pass through the nasopharynx from the outside to the body, a large amount of irrelevant information will be collected during this process; after entering the body, due to camera shaking, water spraying and other operations, the collected images will also be blurred, which will interfere with the algorithm's identification of the bronchial location.

[0060] The historical videos collected in this application include 250 original bronchoscopic examination videos collected from multiple cooperative hospitals. The examination videos should include one or more parts of the pharynx, glottis, trachea, carina, left main bronchus, left upper lobe, left upper lobe proper segment, left upper lobe lingula, left lower lobe, left lower lobe basal segment, left lower lobe dorsal segment, right main bronchus, right upper lobe, right intermediate bronchus, right middle lobe, right lower lobe, right lower lobe basal segment, and right lower lobe dorsal segment. 200 videos are used as training sets for model training, and 50 videos are used as test sets for model testing.

[0061] In this application, it is necessary to manually annotate the collected bronchoscopy videos in advance and record the time nodes of entering and exiting each part. Then, according to the annotation file, all historical videos are intercepted to construct a part recognition data set, including background, pharynx, glottis, trachea, carina, left main bronchus, left upper lobe, left upper lobe proper segment, left upper lobe lingula, left lower lobe, left lower lobe basal segment, left lower lobe dorsal segment, right main bronchus, right upper lobe, right intermediate bronchus, right middle lobe, right lower lobe, right lower lobe basal segment, right lower lobe dorsal segment. Each clip only contains the main bronchial image area, and the rest of the patient information and equipment information are not included. The screen size of each video is adjusted to 128×128. A separate folder is created for each part, and the intercepted video clips of each part are stored in each part folder.

[0062] S102, inputting the training set into a convolutional neural network for training until the convolutional neural network converges to obtain a part recognition model;

[0063] It is understandable that the convolutional neural network in this application can use an existing neural network or a neural network that may appear in the future, and this application does not limit it. In this application, the model parameters in the convolutional neural network are trained using a training set, and a cross entropy loss function is used to perform 150 forward calculations + back propagation processes. This training method can help the network better learn and capture the complex time, space and motion information in the bronchoscopy video, and improve the generalization ability and robustness of the model.

[0064] S103, determining whether the test video in the test set is a video obtained by in vivo detection or in vitro detection collected by a bronchoscope, and inputting the test video determined to be an in vivo detection into the part recognition model to obtain a recognition result.

[0065] After obtaining the trained part recognition model, the part recognition model is tested using the test set to obtain the probability value of each bronchial part, thereby obtaining the recognition result. In the process of bronchial part recognition, the present application first determines the inside and outside of the body, thereby blocking the video segment outside the body, and the blocked video segment is not subjected to part recognition. This can not only reduce the recognition burden of the part recognition model, but also make the recognition result more accurate.

[0066] In some optional embodiments, after acquiring the historical video collected through the bronchoscope, the method further includes:

[0067] The historical video is preprocessed to obtain a video to be processed.

[0068] In some optional embodiments, preprocessing the historical video includes:

[0069] Scaling the picture size of the historical video to a preset size, and using the historical video with the preset picture size as the first video to be processed;

[0070] The duration of the segments of the first video to be processed is obtained, and the first video to be processed whose segment duration is shorter than a preset duration is eliminated to obtain a second video to be processed; and the second video to be processed is determined as a data set.

[0071] Specifically, in the present application, the size of each video screen is adjusted to a preset size of 128×128. A separate folder is created for each part, and the video clips of each part are stored in each part folder. The duration of each video clip is also obtained in the present application, and video clips with a duration of less than 1 second are eliminated. This is to avoid inaccurate recognition results caused by video clips that are too short. In addition, the present application also needs to perform clear and blurry discrimination on the videos that need to be input into the convolutional neural network in the training set, and only send clear in-vivo video clips to the model for training to avoid blurred images affecting the accuracy of recognition.

[0072] In some optional embodiments, the training set includes training videos; and the convolutional neural network includes:

[0073] A first branch, a second branch, a fully connected layer, and a classification layer, wherein the first branch and the second branch are both connected to the fully connected layer, and the fully connected layer is connected to the classification layer;

[0074] The first branch is used to extract time information and space information of video frame images of training videos;

[0075] The second branch is used to extract motion change information between adjacent video frame images of the training video;

[0076] The fully connected layer is used to combine the time information and spatial information extracted by the first branch and the motion change information extracted by the second branch to obtain bronchial site feature information;

[0077] The classification layer outputs the probability value of each bronchial part according to the bronchial part characteristic information.

[0078] Specifically, Figure 2 As shown, the present application adopts a dual-branch structure of a first branch and a second branch to cope with information from different sources. First, the first branch is used to extract the temporal feature information and spatial feature information of the video, and this process is implemented using spatiotemporal convolution. Spatiotemporal convolution can be split into a combination of 2D convolution and 1D convolution, where 2D convolution is used to capture the features of continuous video frames, while 1D convolution helps capture inter-frame timing information. This makes the algorithm more robust when processing blurred and complex images, and improves the accuracy of bronchial site identification. Secondly, the second branch focuses on processing motion information. During bronchoscopy, the operator's movements will cause the site to move in the picture. In order to solve this problem, the present application adopts a dense optical flow algorithm to capture the motion change information between adjacent frames by tracking pixel displacement. Among them, optical flow is a vector field that describes the motion of objects in an image, that is, tracking the displacement of pixels between consecutive frames. After calculation, the offsets between adjacent frames in the x-direction and y-direction are obtained, and these offsets are again integrated with spatiotemporal convolution using motion information and timing information. This fine motion information is crucial for accurately identifying the bronchial site.

[0079] Finally, through the combination of the fully connected layer and the classification layer oftmax layer, the feature information extracted by the first branch and the second branch are merged to output the probability value of each bronchial part. The technical solution provided by the present application realizes fully automatic operation without human intervention.

[0080] In some optional embodiments, the first branch includes a first spatiotemporal convolution layer, a second spatiotemporal convolution layer, a third spatiotemporal convolution layer, a fourth spatiotemporal convolution layer, a fifth spatiotemporal convolution layer and a first pooling layer, which are sequentially connected from top to bottom;

[0081] The second branch includes a downsampling layer, a dense optical flow calculation layer, a sixth spatiotemporal convolution layer, a seventh spatiotemporal convolution layer, an eighth spatiotemporal convolution layer and a second pooling layer connected sequentially from top to bottom;

[0082] The first pooling layer and the second pooling layer are both connected to a fully connected layer, and the fully connected layer is connected to a classification layer.

[0083] Specifically, in this application, about 40,000 two-dimensional images of various parts are extracted from the intercepted video clips, and a 2D convolutional neural network is pre-trained. The input size of the model is 20×16×112×112×3, where 20 is the batch dimension, 16 is the number of consecutive frames read from the data set fragment each time, and each time only a 112×112 image area is randomly cropped from the video screen to achieve the purpose of data enhancement. When the data is input into the first branch, 5 groups of spatiotemporal convolution modules are sampled to extract the time and space information of the input video frame. The second branch first downsamples the input video frame, and the size of the single frame after sampling is 112×112. The dense optical flow calculation is performed on the adjacent frames of the input to obtain the offsets between the adjacent frames in the x and y directions. The spatiotemporal convolution is used again for these offsets to integrate the motion information and timing information. Finally, the fully connected layer is used to merge the feature information of the two branches, and the probability values ​​of each category are output through the classified softmax layer. It is understandable that the spatiotemporal convolution modules in each branch of the present application are composed of several groups of 2D convolutions and 1D convolutions, wherein when the network is initialized, the 2D convolutions in each module use pre-trained model parameters. When training the network, a learning rate decay strategy is adopted, the initial learning rate is set to 0.01, and the learning rate is divided by 10 after every 10 epochs. The cross entropy loss function is used for 150 epochs of training. Such a training setting helps the network to better learn and capture the complex time, space, and motion information in bronchoscopy videos, and improve the generalization ability and robustness of the model.

[0084] In some optional embodiments, the determining that the test video in the test set is a video obtained by in vivo detection or in vitro detection collected by bronchoscope includes:

[0085] Get the test video;

[0086] Determining whether the continuous video frame images in the test video are in-vivo detection or out-of-vivo detection based on a preset window length;

[0087] When the continuous video frame images in the preset window length are in vivo detections and the number of continuous video frame images exceeds the preset threshold, it is determined that the current frame image of the test video is an in vivo detection image collected by a bronchoscope; when the continuous video frame images in the preset window length are in vitro detections and the number of continuous video frame images exceeds the preset threshold, it is determined that the current frame image of the test video is an in vitro detection image collected by a bronchoscope.

[0088] In some optional embodiments, obtaining the recognition result includes:

[0089] For videos with clear images, identify and output part recognition results;

[0090] For videos with blurred images, the part recognition results are not displayed.

[0091] It is understandable that when testing the part recognition model in the present application, the video is read first, and the in vivo and in vitro detection is performed first. In the present application, the window length is set to 20, and the preset threshold is 12. When 12 consecutive frames in the window are judged to be in vivo, the in vivo state is set from the current frame. After entering the in vivo state, a clear and blurry detection is performed, and the part recognition is performed for the clear video clips and the predicted results of the part recognition model are output. For video clips with blurred states, the results of part recognition are not displayed. When 12 consecutive frames in the window are judged to be in vitro, the in vitro state is set from the current frame, and the recognition ends.

[0092] In some optional embodiments, the bronchial site identification method provided in the present application further includes:

[0093] Determine whether the video in the training set is a clear image or a blurred image, and determine whether the video detected in the body is a clear image or a blurred image;

[0094] The videos in the training set are determined to be clear images and input into the convolutional neural network for training, and the videos detected in the body are labeled as clear images or blurred images and input into the part recognition model.

[0095] Because lens shaking, water spraying and other operations will cause the captured image to be blurred, and the blurred image will interfere with the algorithm's identification of the bronchial site. Therefore, in this application, whether it is a video that needs to be input into the convolutional neural network or a test video that needs to be input into the site recognition model, it is necessary to judge the clear image and the blurred image. In this application, the method of judging whether an image is a clear image or a blurred image can be through the method of discriminating the model, and it can also be judged by setting a clear threshold or a blurred threshold. Whether the image is a clear image or a blurred image is not limited in this application.

[0096] In some optional embodiments, the bronchial site provided in the present application includes:

[0097] One or more parts of the pharynx, glottis, trachea, carina, left main bronchus, left upper lobe, left upper lobe proper segment, left upper lobe lingular segment, left lower lobe, left lower lobe basal segment, left lower lobe dorsal segment, right main bronchus, right upper lobe, right intermediate bronchus, right middle lobe, right lower lobe, right lower lobe basal segment, and right lower lobe dorsal segment.

[0098] The present application provides a bronchial site recognition method, which realizes automatic recognition of bronchial sites by extracting the temporal and spatial features of bronchoscopic examination videos and combining dense optical flow calculation. The method flexibly responds to external interference information in images through clear and fuzzy recognition and in vivo and in vitro recognition, and is based on deep learning algorithm modeling to adapt to different bronchial morphologies. It enhances the perception of moving sites by modeling time dimension information and introducing dense optical flow calculation, and realizes real-time output of bronchial site recognition results.

[0099] like Figure 3 As shown, the embodiment of the present application provides a bronchial part identification device, characterized in that the device comprises:

[0100] An acquisition module 301 is used to acquire historical videos collected through a bronchoscope, annotate the historical videos as a data set, and divide the data set into a training set and a test set;

[0101] A training module 302 is used to input the training set into a convolutional neural network for training until the convolutional neural network converges to obtain a part recognition model;

[0102] The output module 303 is used to determine whether the test video in the test set is a video obtained by in vivo detection or in vitro detection collected by bronchoscope, and input the test video determined to be an in vivo detection into the part recognition model to obtain a recognition result.

[0103] Some optional embodiments further include:

[0104] The preprocessing module is used to preprocess the historical video to obtain the video to be processed.

[0105] In some optional embodiments, the preprocessing module includes:

[0106] A processing unit, used for scaling the picture size of the historical video to a preset size, and taking the historical video with the preset picture size as the first video to be processed;

[0107] The elimination unit is used to obtain the segment duration of the first video to be processed, eliminate the first video to be processed whose segment duration is less than a preset duration, and obtain the second video to be processed; and determine the second video to be processed as a data set.

[0108] In some optional embodiments, the convolutional neural network comprises:

[0109] A first branch, a second branch, a fully connected layer, and a classification layer, wherein the first branch and the second branch are both connected to the fully connected layer, and the fully connected layer is connected to the classification layer;

[0110] The first branch is used to extract time information and space information of video frame images of training videos;

[0111] The second branch is used to extract motion change information between adjacent video frame images of the training video;

[0112] The fully connected layer is used to combine the time information and spatial information extracted by the first branch and the motion change information extracted by the second branch to obtain bronchial site feature information;

[0113] The classification layer outputs the probability value of each bronchial part according to the bronchial part characteristic information.

[0114] In some optional embodiments, the first branch includes a first spatiotemporal convolution layer, a second spatiotemporal convolution layer, a third spatiotemporal convolution layer, a fourth spatiotemporal convolution layer, a fifth spatiotemporal convolution layer and a first pooling layer, which are sequentially connected from top to bottom;

[0115] The second branch includes a downsampling layer, a dense optical flow calculation layer, a sixth spatiotemporal convolution layer, a seventh spatiotemporal convolution layer, an eighth spatiotemporal convolution layer and a second pooling layer connected sequentially from top to bottom;

[0116] The first pooling layer and the second pooling layer are both connected to a fully connected layer, and the fully connected layer is connected to a classification layer.

[0117] In some optional embodiments, the output module includes:

[0118] An acquisition unit, used for acquiring a test video;

[0119] A detection unit, used for determining whether the continuous video frame images in the test video are in-vivo detection or out-of-vivo detection according to a preset window length;

[0120] A judgment unit is used to determine that the current frame image of the test video is an in vivo detection image collected by a bronchoscope when the continuous video frame images in a preset window length are in vivo detection and the number of continuous video frame images exceeds a preset threshold; and to determine that the current frame image of the test video is an in vitro detection image collected by a bronchoscope when the continuous video frame images in a preset window length are in vitro detection and the number of continuous video frame images exceeds a preset threshold.

[0121] Some optional embodiments further include:

[0122] Determine whether the video in the training set is a clear image or a blurred image, and determine whether the video detected in the body is a clear image or a blurred image;

[0123] The videos in the training set are determined to be clear images and input into the convolutional neural network for training, and the videos detected in the body are labeled as clear images or blurred images and input into the part recognition model.

[0124] In some optional embodiments, the output module includes:

[0125] The first output subunit is used to identify and output a part recognition result for a video of a clear image;

[0126] The second output subunit is used for not displaying the part recognition result for the video with blurred image.

[0127] In some optional embodiments, the bronchial site includes:

[0128] One or more parts of the pharynx, glottis, trachea, carina, left main bronchus, left upper lobe, left upper lobe proper segment, left upper lobe lingular segment, left lower lobe, left lower lobe basal segment, left lower lobe dorsal segment, right main bronchus, right upper lobe, right intermediate bronchus, right middle lobe, right lower lobe, right lower lobe basal segment, and right lower lobe dorsal segment.

[0129] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device and a readable storage medium.

[0130] Figure 4 A schematic block diagram of an example electronic device 400 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0131] like Figure 4 As shown, the device 400 includes a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 402 or a computer program loaded from a storage unit 408 into a random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the device 400 can also be stored. The computing unit 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0132] A number of components in the device 400 are connected to the I / O interface 405, including: an input unit 406, such as a keyboard, a mouse, etc.; an output unit 407, such as various types of displays, speakers, etc.; a storage unit 408, such as a disk, an optical disk, etc.; and a communication unit 409, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 409 allows the device 400 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0133] The computing unit 401 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 401 performs the various methods and processes described above, such as the bronchial site identification method. For example, in some embodiments, the bronchial site identification method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 400 via the ROM 402 and / or the communication unit 409. When the computer program is loaded into the RAM 403 and executed by the computing unit 401, one or more steps of the bronchial site identification method described above may be performed. Alternatively, in other embodiments, the computing unit 401 may be configured to perform the bronchial site identification method in any other appropriate manner (e.g., by means of firmware).

[0134] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0135] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0136] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0137] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0138] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0139] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0140] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0141] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present disclosure, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0142] The above is only a specific embodiment of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present disclosure, which should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A method for identifying bronchial parts, characterized in that: The method comprises: Obtaining historical videos collected through a bronchoscope, annotating the historical videos as a data set, and dividing the data set into a training set and a test set; Inputting the training set into a convolutional neural network for training until the convolutional neural network converges to obtain a part recognition model; It is determined that the test video in the test set is a video obtained by in vivo detection or in vitro detection collected by a bronchoscope, and the test video determined to be an in vivo detection is input into the part recognition model to obtain a recognition result.

2. The method according to claim 1, characterized in that After obtaining the historical video acquired via bronchoscopy, also include: The historical video is preprocessed to obtain a video to be processed.

3. The method according to claim 1, characterized in that Preprocessing the historical video includes: Scaling the picture size of the historical video to a preset size, and using the historical video with the preset picture size as the first video to be processed; The duration of the segments of the first video to be processed is obtained, and the first video to be processed whose segment duration is shorter than a preset duration is eliminated to obtain a second video to be processed; and the second video to be processed is determined as a data set.

4. The method according to claim 1, characterized in that: The training set includes training videos; the convolutional neural network includes: A first branch, a second branch, a fully connected layer, and a classification layer, wherein the first branch and the second branch are both connected to the fully connected layer, and the fully connected layer is connected to the classification layer; The first branch is used to extract time information and space information of video frame images of training videos; The second branch is used to extract motion change information between adjacent video frame images of the training video; The fully connected layer is used to combine the time information and spatial information extracted by the first branch and the motion change information extracted by the second branch to obtain bronchial site feature information; The classification layer outputs the probability value of each bronchial part according to the bronchial part characteristic information.

5. The method according to claim 4, characterized in that The first branch includes a first spatiotemporal convolution layer, a second spatiotemporal convolution layer, a third spatiotemporal convolution layer, a fourth spatiotemporal convolution layer, a fifth spatiotemporal convolution layer and a first pooling layer, which are sequentially connected from top to bottom; The second branch includes a downsampling layer, a dense optical flow calculation layer, a sixth spatiotemporal convolution layer, a seventh spatiotemporal convolution layer, an eighth spatiotemporal convolution layer and a second pooling layer connected sequentially from top to bottom; The first pooling layer and the second pooling layer are both connected to a fully connected layer, and the fully connected layer is connected to a classification layer.

6. The method according to claim 1, characterized in that The step of determining whether the test video in the test set is a video obtained by in vivo detection or in vitro detection collected by a bronchoscope comprises: Get the test video; Determining whether the continuous video frame images in the test video are in-vivo detection or out-of-vivo detection based on a preset window length; When the continuous video frame images in the preset window length are in vivo detections and the number of continuous video frame images exceeds the preset threshold, it is determined that the current frame image of the test video is an in vivo detection image collected by a bronchoscope; when the continuous video frame images in the preset window length are in vitro detections and the number of continuous video frame images exceeds the preset threshold, it is determined that the current frame image of the test video is an in vitro detection image collected by a bronchoscope.

7. The method according to claim 6, characterized in that Also includes: Determine whether the video in the training set is a clear image or a blurred image, and determine whether the video detected in the body is a clear image or a blurred image; The videos in the training set are determined to be clear images and input into the convolutional neural network for training, and the videos detected in the body are labeled as clear images or blurred images and input into the part recognition model.

8. The method according to claim 7, characterized in that The obtaining of the recognition result includes: For videos with clear images, identify and output part recognition results; For videos with blurred images, the part recognition results are not displayed.

9. The method according to claim 1, characterized in that: The bronchial sites include: One or more parts of the pharynx, glottis, trachea, carina, left main bronchus, left upper lobe, left upper lobe proper segment, left upper lobe lingular segment, left lower lobe, left lower lobe basal segment, left lower lobe dorsal segment, right main bronchus, right upper lobe, right intermediate bronchus, right middle lobe, right lower lobe, right lower lobe basal segment, and right lower lobe dorsal segment.

10. A bronchial part identification device, characterized in that: The device comprises: An acquisition module, used to acquire historical videos collected through a bronchoscope, annotate the historical videos as a data set, and divide the data set into a training set and a test set; A training module, used for inputting the training set into a convolutional neural network for training until the convolutional neural network converges to obtain a part recognition model; The output module is used to determine whether the test video in the test set is a video obtained by in vivo detection or in vitro detection collected by bronchoscope, and input the test video determined to be an in vivo detection into the part recognition model to obtain a recognition result.

Citation Information

Patent Citations

  • Behavior recognition method and device, electronic equipment and storage medium

    CN113177450A

  • Human body behavior recognition method and system based on extracted video spatio-temporal information

    CN113887419A

  • Method for training lung endoscope image recognition model and recognition method

    CN115393670A

  • Driver behavior recognition method and device based on space-time and motion deep learning

    CN115482519A