Method for collecting and processing video of respiratory endoscope, electronic device and storage medium

Abstract

The present invention discloses a method for video acquisition and processing of a respiratory endoscope, an electronic device, and a storage medium, belonging to the technical field of video processing. The method includes: acquiring a tracheal scan image; generating a bronchial tree diagram based on the tracheal scan image, and marking the lesion occurrence area on the bronchial tree diagram; generating a lesion navigation path based on the bronchial tree diagram and the lesion occurrence area; acquiring a real-time endoscope video based on the lesion navigation path; determining whether the endoscope reaches the airway bifurcation corresponding to the lesion occurrence area based on the real-time endoscope video; if it is determined that the endoscope reaches the airway bifurcation corresponding to the lesion occurrence area, then switching the video compression strategy of the real-time endoscope video from the first compression strategy to the second compression strategy. The present invention can reasonably arrange the video compression strategy during the diagnosis and treatment process of a respiratory endoscope, ensuring the timeliness of video transmission while meeting the high picture quality requirements of the lesion area, which is beneficial to improving the accuracy and reliability of respiratory endoscope examinations.

Description

Technical Field

[0001] The present invention belongs to the technical field of video processing, and particularly relates to a method for collecting and processing endoscopic video of respiration, an electronic device, and a storage medium. Background Art

[0002] Respiratory endoscopy is an important diagnosis and treatment technology, which plays an important role in the diagnosis and treatment of the entire respiratory system diseases and is an important part of respiratory interventional medicine. In the diagnosis and treatment process of respiratory endoscopy, high requirements are placed on the timeliness of video transmission and the clarity of image display. Therefore, it is necessary to achieve efficient and clear transmission of endoscopic video during the diagnosis and treatment process of respiratory endoscopy. Video compression technology is a technology that can reduce the amount of data transmitted or stored while providing high-quality video images, thus facilitating the improvement of data transmission efficiency. In the current process of respiratory interventional diagnosis and treatment, usually only a single video compression technology is used in the video processing and transmission process. This situation is likely to cause insufficient compression of low-value images before the endoscope reaches the lesion, and excessive compression of high-value images after the endoscope reaches the lesion, resulting in image quality loss.

[0003] In view of this, there is an urgent need to propose a method for collecting and processing endoscopic video of respiration, which can reasonably arrange video compression strategies during the diagnosis and treatment process of respiratory endoscopy, ensure the timeliness of video transmission, and meet the high image quality requirements of the lesion area, which is beneficial to improving the accuracy and reliability of respiratory endoscopy examination. Summary of the Invention

[0004] To overcome the problems existing in the related art, the present invention provides a method for collecting and processing endoscopic video of respiration, an electronic device, and a storage medium. The method for collecting and processing endoscopic video of respiration can reasonably arrange video compression strategies during the diagnosis and treatment process of respiratory endoscopy, ensure the timeliness of video transmission, and meet the high image quality requirements of the lesion area, which is beneficial to improving the accuracy and reliability of respiratory endoscopy examination.

[0005] The present invention provides a method for collecting and processing endoscopic video of respiration, including:

[0006] Obtaining a trachea scan image;

[0007] Generating a bronchial tree diagram based on the trachea scan image, and marking the lesion occurrence area on the bronchial tree diagram;

[0008] Generating a lesion navigation path based on the bronchial tree diagram and the lesion occurrence area;

[0009] Collecting real-time endoscopic video based on the lesion navigation path;

[0010] Determining whether the endoscope reaches the airway bifurcation corresponding to the lesion occurrence area based on the real-time endoscopic video;

[0011] If it is determined that the endoscope reaches the airway bifurcation corresponding to the lesion occurrence area, the video compression strategy of the real-time endoscopic video is switched from the first compression strategy to the second compression strategy; wherein, the data compression rate of the first compression strategy is greater than that of the second compression strategy.

[0012] Furthermore, the video compression strategy of the real-time endoscopic video is switched from the first compression strategy to the second compression strategy, where the first compression strategy includes:

[0013] Read each video frame of the real-time endoscopic video and obtain the moving speed of the endoscope;

[0014] Determine the key frame division interval duration based on the moving speed and the preset endoscopic step distance of the endoscope;

[0015] Determine key frames from each video frame based on the key frame division interval duration to obtain multiple key frames;

[0016] Determine multiple reference frames between each pair of adjacent key frames;

[0017] Perform motion compensation based on each reference frame to obtain residual data;

[0018] Perform video compression based on the residual data.

[0019] Furthermore, determining multiple reference frames between each pair of adjacent key frames includes:

[0020] Starting from the previous key frame in the current adjacent key frames, determine the next video frame separated by two video frames as the reference frame until the number of video frames between the last reference frame and the subsequent key frame in the current adjacent key frames is less than or equal to 2.

[0021] Furthermore, the video compression strategy of the real-time endoscopic video is switched from the first compression strategy to the second compression strategy, where the second compression strategy includes:

[0022] Perform lossless compression on the region of interest of the real-time endoscopic video.

[0023] Furthermore, determining whether the endoscope reaches the airway bifurcation corresponding to the lesion occurrence area based on the real-time endoscopic video includes:

[0024] Monitor the key frame division update status;

[0025] When it is monitored that an updated key frame is generated, extract the updated video image corresponding to the updated key frame;

[0026] The deviation direction of the main trunk of the lesion navigation path and the updated video image are input into the branch bifurcation classification model to obtain the branch bifurcation type result output by the branch bifurcation classification model; among them, the branch bifurcation classification model includes a main trunk bifurcation classification model, a first bronchial bifurcation classification model, and a second bronchial bifurcation classification model; among them, the main trunk bifurcation classification model is trained based on the main trunk tracheal bifurcation image training set, the first bronchial bifurcation classification model is trained based on the first bronchial bifurcation image training set, and the second bronchial bifurcation classification model is trained based on the second bronchial bifurcation image training set;

[0027] Based on the branch bifurcation type result, it is determined whether the endoscope reaches the airway branch bifurcation corresponding to the lesion occurrence area.

[0028] Furthermore, determining whether the endoscope reaches the airway branch bifurcation corresponding to the lesion occurrence area based on the branch bifurcation type result includes:

[0029] If the branch bifurcation type result matches the airway branch bifurcation corresponding to the lesion occurrence area, it is determined that the endoscope reaches the airway branch bifurcation corresponding to the lesion occurrence area.

[0030] Furthermore, generating a bronchial tree diagram based on the tracheal scan image includes:

[0031] Perform three-dimensional reconstruction on the tracheal scan image to obtain an initial bronchial three-dimensional image;

[0032] Preprocess the initial bronchial three-dimensional image to obtain a preprocessed three-dimensional image;

[0033] Perform segmentation and extraction on the preprocessed three-dimensional image through a 3D convolutional neural network to obtain a bronchial tree diagram.

[0034] Furthermore, generating a lesion navigation path based on the bronchial tree diagram and the lesion occurrence area includes:

[0035] Extract the centerline of the bronchial tree diagram through a topological refinement algorithm;

[0036] Through a path planning algorithm based on the Euclidean distance, with the entrance of the centerline as the starting point and the airway branch bifurcation corresponding to the lesion occurrence area as the end point, plan the lesion navigation path.

[0037] The present invention also provides an electronic device, including:

[0038] A processor; and a memory, on which executable code is stored, and when the executable code is executed by the processor, the processor executes the method described above.

[0039] The present invention further provides a non-transitory machine-readable storage medium, on which executable code is stored. When the executable code is executed by a processor of an electronic device, the processor is caused to execute the method as described above.

[0040] The technical solution provided by the present invention may include the following beneficial effects:

[0041] The method for collecting and processing a respiratory endoscope video, the electronic device and the storage medium provided by the present invention obtain a trachea scan image, and then generate a bronchial tree diagram based on the trachea scan image, mark the lesion occurrence area on the bronchial tree diagram, and then generate a lesion navigation path based on the bronchial tree diagram and the lesion occurrence area. Thereby, it can help the examiner efficiently and accurately insert the endoscope into the lesion occurrence area.

[0042] The present invention can collect a real-time endoscope video based on the lesion navigation path, and then determine whether the endoscope reaches the airway branch fork corresponding to the lesion occurrence area based on the real-time endoscope video. After passing through the airway branch fork corresponding to the lesion occurrence area, it can enter the lesion occurrence area. At this time, if it is determined that the endoscope reaches the airway branch fork corresponding to the lesion occurrence area, the video compression strategy of the real-time endoscope video is switched from the first compression strategy to the second compression strategy, where the data compression rate of the first compression strategy is greater than that of the second compression strategy. Thereby, it can achieve that the low-value pictures before the endoscope reaches the lesion are fully compressed, and the high-value pictures after the endoscope reaches the lesion are reasonably compressed while avoiding picture quality loss.

[0043] Generally speaking, the present invention can reasonably arrange the video compression strategy during the diagnosis and treatment process of the respiratory endoscope, ensure the timeliness of video transmission while guaranteeing the high picture quality requirements of the lesion area, and is beneficial to improving the accuracy and reliability of the respiratory endoscope examination. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is one of the schematic flowcharts of the method for collecting and processing a respiratory endoscope video shown in an embodiment of the present invention;

[0045] Figure 2 is another schematic flowchart of the method for collecting and processing a respiratory endoscope video shown in an embodiment of the present invention;

[0046] Figure 3 is yet another schematic flowchart of the method for collecting and processing a respiratory endoscope video shown in an embodiment of the present invention;

[0047] Figure 4 is the schematic structural diagram of the electronic device shown in an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The preferred embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the preferred embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to make the present invention more thorough and complete, and to fully convey the scope of the present invention to those skilled in the art.

[0049] The terms used in the present invention are for the purpose of describing specific embodiments only and are not intended to limit the present invention. The singular forms "a", "the" and "said" used in the present invention and the appended claims are also intended to include the plural forms unless the context clearly dictates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0050] It should be understood that although the terms "first", "second", "third", etc. may be used in the present invention to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the present invention, the meaning of "a plurality" is two or more unless otherwise specifically defined.

[0051] In the process of diagnosis and treatment using a respiratory endoscope, high requirements are placed on the timeliness of video transmission and the clarity of image display. Therefore, it is necessary to achieve efficient and clear transmission of endoscope video during the process of diagnosis and treatment using a respiratory endoscope. Video compression technology is a technology that can reduce the amount of data transmitted or stored while providing high-quality video images, thus facilitating the improvement of data transmission efficiency. Although existing video compression technologies have been relatively mature, if a single video compression technology is used in the video processing and transmission process during the entire diagnosis and treatment process, it is likely that low-value images before the endoscope reaches the lesion cannot be fully compressed, while high-value images after the endoscope reaches the lesion are overly compressed, resulting in image quality loss.

[0052] The present invention proposes a method for collecting and processing respiratory endoscope video, so as to be able to reasonably arrange video compression strategies during the process of diagnosis and treatment using a respiratory endoscope, ensure the timeliness of video transmission while meeting the high image quality requirements for the lesion area, and facilitate the improvement of the accuracy and reliability of respiratory endoscope examinations.

[0053] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0054] Example 1

[0055] Please refer to Figure 1 , the respiratory endoscope video acquisition and processing method shown in the embodiments of the present invention includes:

[0056] S101, Obtain a tracheal scan image, which refers to an image obtained by performing a CT scan on a lung region including the trachea, bronchi, and lung lobes. CT scan is of great value in the diagnosis of tracheal lesions, and can provide detailed image information of the trachea and its surrounding structures, helping doctors identify and evaluate lesions. CT scan can clearly show the position, size, wall thickness, and morphological changes of the trachea, which is helpful for judging whether there are organic lesions, such as stenosis, tumors, polyps, or other abnormalities.

[0057] S102, Generate a bronchial tree diagram based on the tracheal scan image, and mark the lesion occurrence area on the bronchial tree diagram. Import the tracheal scan image into a navigation system, such as Virtual Bronchoscopic Navigation (VBN), which is a navigation technology based on CT image data and is used to assist bronchoscopy and the diagnosis of lung lesions. Generate a virtual bronchial tree image through the navigation system and mark the lesion occurrence area of the target lesion.

[0058] S103, Generate a lesion navigation path based on the bronchial tree diagram and the lesion occurrence area. Use the tree branches in the bronchial tree diagram to plan the path to reach the lesion occurrence area. During the examination, the endoscope entering the respiratory tract will gradually reach the lesion occurrence area along the path obtained above.

[0059] S104, Acquire real-time endoscopic video based on the lesion navigation path. Use an endoscope (i.e., bronchoscope) for examining the trachea and bronchi to acquire real-time endoscopic video along the lesion navigation path. An endoscope is a medical device used for examining and treating the internal organs of the human body. It is inserted into the natural body cavity or a small incision of the human body, and uses optical or electronic imaging technology to transmit the image inside the body to an external display to help doctors for diagnosis and treatment.

[0060] S105, Determine whether the endoscope reaches the airway bifurcation corresponding to the lesion occurrence area based on the real-time endoscopic video. From the anatomical structure of the bronchial tree, it can be known that the overall tracheal structure is divided into the main trachea, the right main bronchus, and the left main bronchus, and there will be a bronchial bifurcation at the bifurcation of the right main bronchus and the left main bronchus.

[0061] On the one hand, the right main bronchus bifurcates into the right upper lobe, the right middle lobe, and the right lower lobe, and there is a right lobe bifurcation at the bifurcation of the right upper lobe, the right middle lobe, and the right lower lobe. Further, the right upper lobe contains the apical segment of the right upper lobe, the anterior segment of the right upper lobe, and the posterior segment of the right upper lobe, and there is a segmental bifurcation of the right upper lobe at the bifurcation of the apical segment of the right upper lobe, the anterior segment of the right upper lobe, and the posterior segment of the right upper lobe. In addition, the right middle lobe contains the lateral segment of the right middle lobe and the medial segment of the right middle lobe, and there is a segmental bifurcation of the right middle lobe at the bifurcation of the lateral segment of the right middle lobe and the medial segment of the right middle lobe. Moreover, the right lower lobe contains the dorsal segment of the right lower lobe, the anterior basal segment of the right lower lobe, the medial basal segment of the right lower lobe, the lateral basal segment of the right lower lobe, and the posterior basal segment of the right lower lobe, and there is a segmental bifurcation of the right lower lobe at the bifurcation of the dorsal segment of the right lower lobe, the anterior basal segment of the right lower lobe, the medial basal segment of the right lower lobe, the lateral basal segment of the right lower lobe, and the posterior basal segment of the right lower lobe.

[0062] On the other hand, the left main bronchus divides into the left upper lobe and the left lower lobe, and there is a left lobe bifurcation at the bifurcation of the left upper lobe and the left lower lobe. Further, the left upper lobe contains the apicoposterior segment of the left upper lobe, the anterior segment of the left upper lobe, the superior lingular segment of the left upper lobe, and the inferior lingular segment of the left upper lobe, and there is a segmental bifurcation of the left upper lobe at the bifurcation of the apicoposterior segment of the left upper lobe, the anterior segment of the left upper lobe, the superior lingular segment of the left upper lobe, and the inferior lingular segment of the left upper lobe. In addition, the left lower lobe contains the dorsal segment of the left lower lobe, the anterior basal segment of the left lower lobe, the lateral basal segment of the left lower lobe, and the posterior basal segment of the left lower lobe, and there is a segmental bifurcation of the left lower lobe at the bifurcation of the dorsal segment of the left lower lobe, the anterior basal segment of the left lower lobe, the lateral basal segment of the left lower lobe, and the posterior basal segment of the left lower lobe.

[0063] Among the various bifurcations mentioned above, there are differences in terms of the bifurcation shape, bifurcation size, bifurcation branch angle, bifurcation texture, etc. Therefore, the bifurcation reached by the endoscope can be identified to confirm whether the airway bifurcation corresponding to the lesion occurrence area has been reached. Suppose the lesion occurrence area is located in the anterior segment of the left upper lobe. Then it can be identified whether the endoscope has reached the segmental bifurcation of the left upper lobe. If so, it is determined that the endoscope has reached the airway bifurcation corresponding to the anterior segment of the left upper lobe.

[0064] S106. If it is determined that the endoscope has reached the airway bifurcation corresponding to the lesion occurrence area, the video compression strategy of the real-time endoscope video is switched from the first compression strategy to the second compression strategy. The data compression rate of the first compression strategy is greater than that of the second compression strategy. This is because before the endoscope reaches the lesion occurrence area, the endoscope moves in the trachea, and the images collected at this time do not have the significance for lesion examination. Therefore, the first compression strategy with a higher data compression rate can be used to compress the video data collected before the endoscope reaches the lesion occurrence area. After the endoscope reaches the lesion occurrence area, the endoscope is ready to enter the lesion occurrence area for examination. Therefore, it is necessary to ensure that the image quality is maintained at a high level and does not affect real-time transmission. Therefore, the second compression strategy with a lower data compression rate is used to compress the video data collected after the endoscope reaches the lesion occurrence area.

[0065] In the embodiment of the present invention, by obtaining the tracheal scan image, a bronchial tree diagram is generated based on the tracheal scan image, and the lesion occurrence area is marked on the bronchial tree diagram. Then, a lesion navigation path is generated based on the bronchial tree diagram and the lesion occurrence area. This can help the examiner efficiently and accurately insert the endoscope into the lesion occurrence area. The present invention can collect real-time endoscope video based on the lesion navigation path, and then determine whether the endoscope has reached the airway bifurcation corresponding to the lesion occurrence area based on the real-time endoscope video. After passing through the airway bifurcation corresponding to the lesion occurrence area, it can enter the lesion occurrence area. At this time, if it is determined that the endoscope has reached the airway bifurcation corresponding to the lesion occurrence area, the video compression strategy of the real-time endoscope video is switched from the first compression strategy to the second compression strategy, where the data compression rate of the first compression strategy is greater than that of the second compression strategy. Thus, it can be realized that the low-value images before the endoscope reaches the lesion are fully compressed, and while the high-value images after the endoscope reaches the lesion are reasonably compressed, the situation of image quality loss is avoided.

[0066] Generally speaking, the present invention can reasonably arrange the video compression strategy during the diagnosis and treatment process of the respiratory endoscope, ensure the timeliness of video transmission, and meet the high image quality requirements of the lesion area, which is beneficial to improving the accuracy and reliability of the respiratory endoscope examination.

[0067] In some embodiments, the first compression strategy and the second compression strategy can be further designed. Figure 2 This is the second flowchart of the respiratory endoscope video acquisition and processing method shown in the embodiment of the present invention. Please refer to Figure 2 The respiratory endoscope video acquisition and processing method shown in the embodiment of the present invention may include:

[0068] Before the endoscope reaches the airway bifurcation corresponding to the lesion occurrence area, a first compression strategy is adopted, where the first compression strategy includes:

[0069] S201, Read each video frame of the real-time endoscope video and obtain the moving speed of the endoscope. Determine the moving speed of the endoscope based on each video frame of the real-time endoscope video. First, receive the current video frame and obtain the previous adjacent video frame. Then, generate an optical flow map for the current video frame relative to the previous video frame (an optical flow algorithm in the OpenCV library can be used, for example, based on sparse optical flow (Lucas-Kanade method) and dense optical flow (Farneback method)). The vectors in the optical flow map represent the movement direction and magnitude of pixel points. Next, determine whether the endoscope is in a moving state according to the orientations of the vectors in the optical flow map (if the modulus of the vector sum is less than a preset threshold, it is determined that the device is in a moving state). Furthermore, when it is determined that the endoscope is in a moving state, calculate the moving speed corresponding to the current image frame using all the vectors in the optical flow map. For example, the moving speed of the endoscope can be estimated by calculating the average value of all the vectors. The method of obtaining the moving speed of the endoscope is diverse. For example, the moving speed of the endoscope can also be obtained through a speed detection sensor set on the endoscope. In practical applications, the method of obtaining the moving speed of the endoscope needs to be determined according to the actual application situation, and the present invention is not limited to a single method.

[0070] S202, Determine the key frame division interval duration based on the moving speed and the preset endoscopic step distance of the endoscope. Divide the preset endoscopic step distance of the endoscope by the moving speed to determine the key frame division interval duration. The moving speed at this time can also be the average speed of the endoscope during the endoscopic step distance, or the instantaneous moving speed at the midpoint of the endoscopic step distance, which needs to be determined according to the actual application situation and is not limited to a single method here.

[0071] S203, Determine key frames from each video frame based on the key frame division interval duration to obtain multiple key frames. Since the moving speed will fluctuate, the key frame division interval duration will also fluctuate, so the number of video frames between different adjacent key frames will also vary. Using this method is beneficial for setting key frames at each relatively fixed position in the trachea (because the endoscopic step distance is fixed), which can improve the decoding quality of the real-time endoscope video and the efficiency of random access.

[0072] S204. Determine multiple reference frames between each pair of adjacent key frames. Starting from the previous key frame in the current adjacent key frames, determine the next video frame separated by two video frames as the reference frame until the number of video frames between the last reference frame and the subsequent key frame in the current adjacent key frames is less than or equal to 2. Suppose there are 8 video frames between the current adjacent key frames, the previous key frame is the video frame numbered 1, and the subsequent key frame is the video frame numbered 10. Then, the video frames numbered 4 and 7 can be used as reference frames.

[0073] S205. Perform motion compensation based on each reference frame to obtain residual data. First, the current video frame can be segmented into multiple blocks of a fixed size, usually macroblocks of 16×16 pixels or 64×64 pixels. Then, select the reference frame adjacent to the current video frame, and then search for the reference block in the reference frame that is most similar to the current macroblock in the current video frame. Search methods such as Diamond Search and Hexagon Search can be used. Next, calculate the matching cost between the current macroblock and the corresponding reference block in the reference frame. Cost functions such as Sum of Absolute Differences (SAD) and Mean Squared Error (MSE) can be used for calculation. Then, select the macroblock with the minimum matching cost, and the position offset of it relative to its corresponding reference block is the motion vector. Furthermore, the calculated motion vector can be encoded and transmitted to the decoding end. The decoding end reconstructs the current video frame from the reference frame according to the motion vector, generates a prediction frame, and divides the prediction frame into multiple blocks of a fixed size. Then, calculate the residual between each macroblock and its corresponding prediction block respectively. After performing the subtraction between the macroblock and the prediction block, residual data is obtained after calculating the residuals for each macroblock.

[0074] S206. Perform video compression based on the residual data. Encode the residual data to reduce the amount of data, thereby achieving efficient data compression. Transform coding can be used, such as Discrete Cosine Transform (DCT), to convert the residual from the time domain to the frequency domain, and then quantize the coefficients to reduce high-frequency information. Entropy coding can also be used, such as Huffman coding, to perform lossless coding on the quantized coefficients.

[0075] If it is determined that the endoscope reaches the airway bifurcation corresponding to the lesion occurrence area, then switch the video compression strategy of the real-time endoscopic video from the first compression strategy to the second compression strategy. The second compression strategy includes:

[0076] Lossless compression of the region of interest in real-time endoscopic video. First, the region where the lesion occurs can be extracted as the region of interest (ROI) through the cv2.selectROI function of OpenCV. Then, lossless compression is performed on the region of interest to retain all details, while the background region (BG) can be compressed lossily to reduce the amount of data. Lossless compression can be performed, for example, using integer wavelet transform, embedded zero-tree coding (EZW), and Huffman coding. Lossy compression can be performed, for example, using the JPEG (Joint Photographic Experts Group) compression algorithm and the SPIHT (Set Partitioning in Hierarchical Trees) compression algorithm. Finally, the compressed region of interest and background region data are merged to form the final encoded stream.

[0077] In some embodiments, the method of planning the lesion navigation path can be further designed. Then, the updated video images corresponding to the key frames are extracted on the lesion navigation path and input into the trained branch bifurcation classification model to identify the type of the branch bifurcation reached by the endoscope, so as to determine whether the airway branch bifurcation corresponding to the lesion occurrence region is reached. Figure 3 This is the third schematic flowchart of the method for collecting and processing respiratory endoscopy video shown in the embodiments of the present invention. Please refer to Figure 3 The method for collecting and processing respiratory endoscopy video shown in the embodiments of the present invention includes:

[0078] S301, generating a bronchial tree diagram based on the tracheal scan image. The obtained tracheal scan image is saved in DICOM format, and then the tracheal scan image is three-dimensionally reconstructed. For example, the DICOM format tracheal scan image is imported into three-dimensional reconstruction software such as 3D Slicer, VTK, or Matlab to generate an initial three-dimensional bronchial image. The initial three-dimensional bronchial image is preprocessed, such as denoising, contrast enhancement, artifact removal, etc., to obtain a preprocessed three-dimensional image. Then, the preprocessed three-dimensional image can be segmented and extracted through a 3D convolutional neural network to obtain a bronchial tree diagram.

[0079] S302, generating a lesion navigation path based on the bronchial tree diagram and the lesion occurrence region. First, the center line of the bronchial tree diagram is extracted through a topological thinning algorithm to provide a basis for path planning. Then, a path planning algorithm based on the Euclidean distance can be used to plan the lesion navigation path with the entrance of the center line as the starting point and the airway branch bifurcation corresponding to the lesion occurrence region as the end point. In some implementation scenarios, the Bezier curve interpolation method can also be used to smooth the path, reduce path jitter, and improve the fluency of navigation. The three-dimensional lung model of the patient can also be combined to verify the feasibility of the path and ensure that the bronchoscope can pass smoothly.

[0080] S303. Collect real-time endoscopic video based on the lesion navigation path. The real-time endoscopic video is collected along the lesion navigation path by an endoscope (i.e., a bronchoscope) used for examining the trachea and bronchi. An endoscope is a medical device used for examining and treating the internal organs of the human body. It is inserted into the natural body cavity or a small incision of the human body, and uses optical or electronic imaging technology to transmit the image inside the body to an external display to assist doctors in diagnosis and treatment.

[0081] S304. Monitor the update status of key frame division, and when it is detected that an updated key frame is generated, extract the updated video image corresponding to the updated key frame. The key frames can be divided in real time along with the real-time collection of the real-time endoscopic video. When it is detected that a newly divided updated key frame is generated, the updated video image corresponding to the updated key frame can be extracted.

[0082] S305. Input the main trunk deviation direction of the lesion navigation path and the updated video image into the branch fork classification model to obtain the branch fork type result output by the branch fork classification model. The branch fork classification model includes a main trunk fork classification model, a first bronchial fork classification model, and a second bronchial fork classification model. Among them, the main trunk fork classification model is trained based on the main trunk tracheal fork image training set, the first bronchial fork classification model is trained based on the first bronchial fork image training set, and the second bronchial fork classification model is trained based on the second bronchial fork image training set. The first bronchial fork classification model can be used, for example, to identify the branch fork type result in the right main bronchus, and the second bronchial fork classification model can be used, for example, to identify the branch fork type result in the left main bronchus, which is not uniquely limited. Thus, based on the main trunk deviation direction of the lesion navigation path, the main trunk fork classification model, the first bronchial fork classification model, or the second bronchial fork classification model can be selected for identification and detection to improve the identification accuracy.

[0083] The above-mentioned main trunk fork classification model, first bronchial fork classification model, and second bronchial fork classification model can all be trained using supervised machine learning algorithms based on a large amount of labeled data. For example, the ResNet deep convolutional neural network classification model can be used for training.

[0084] First, in the training of the main trunk fork classification model, a large number of training images of bronchial bifurcations and non-bronchial bifurcations can be collected and the positions of the bifurcations can be labeled, so that the main trunk fork classification model has the ability to distinguish bronchial bifurcations and non-bronchial bifurcations.

[0085] Secondly, in the training of the first bronchial bifurcation classification model, a large number of training images of the right lobe bifurcation, the right upper lobe segment bifurcation, the right middle lobe segment bifurcation, and the right lower lobe segment bifurcation are collected, and each of the foregoing training images is labeled. Then, the labeled training images are input into the initial ResNet deep convolutional neural network classification model for training, and the predicted probabilities of the four types of results, namely the right lobe bifurcation, the right upper lobe segment bifurcation, the right middle lobe segment bifurcation, and the right lower lobe segment bifurcation, are output respectively. The cross-entropy loss function is used to determine whether the loss function converges. If it converges, the final first bronchial bifurcation classification model can be output. In actual operation, the bifurcation type with the highest predicted probability output is used as the result of the branch bifurcation type.

[0086] Furthermore, in the training of the second bronchial bifurcation classification model, a large number of training images of the left lobe bifurcation, the left upper lobe segment bifurcation, and the left lower lobe segment bifurcation can be collected, and each of the foregoing training images is labeled. Then, the labeled training images are input into the initial ResNet deep convolutional neural network classification model for training, and the predicted probabilities of the three types of results, namely the left lobe bifurcation, the left upper lobe segment bifurcation, and the left lower lobe segment bifurcation, are output respectively. The cross-entropy loss function is used to determine whether the loss function converges. If it converges, the final second bronchial bifurcation classification model can be output. In actual operation, the bifurcation type with the highest predicted probability output is used as the result of the branch bifurcation type.

[0087] S306. Determine whether the endoscope reaches the airway bifurcation corresponding to the lesion occurrence area based on the result of the branch bifurcation type. If the result of the branch bifurcation type matches the airway bifurcation corresponding to the lesion occurrence area, it is determined that the endoscope reaches the airway bifurcation corresponding to the lesion occurrence area. Suppose the result of the branch bifurcation type is the left upper lobe segment bifurcation, and the lesion occurrence area is in the superior lingular segment of the left upper lobe. Then it indicates that the result of the branch bifurcation type matches the airway bifurcation corresponding to the lesion occurrence area, and it can be determined that the endoscope reaches the airway bifurcation corresponding to the lesion occurrence area.

[0088] Embodiment 2

[0089] Corresponding to the foregoing application function implementation method, the present invention also provides an electronic device for executing a respiratory endoscope video acquisition and processing method and a corresponding embodiment.

[0090] See Figure 4 , the electronic device 400 includes a memory 410 and a processor 420.

[0091] The processor 420 may be a Central Processing Unit (CPU), or may also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0092] The memory 410 includes various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. Among them, the ROM can store static data or instructions required by the processor 420 or other modules of the computer. The permanent storage device may be a readable and writable storage device. The permanent storage device may be a non-volatile storage device that does not lose the stored instructions and data even when the computer is powered off. In some embodiments, the permanent storage device uses a mass storage device (such as a magnetic or optical disk, flash memory) as the permanent storage device. In other embodiments, the permanent storage device may be a removable storage device (such as a floppy disk, optical drive). The system memory may be a readable and writable storage device or a volatile readable and writable storage device, such as dynamic random access memory. The system memory can store some or all of the instructions and data required by the processor during operation. In addition, the memory 410 includes any combination of computer-readable storage media, including various types of semiconductor storage chips (DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and magnetic disks and / or optical disks may also be used. The memory 410 may include removable storage devices that are readable and / or writable, such as compact discs (CDs), read-only digital versatile discs (such as DVD-ROM, dual-layer DVD-ROM), read-only Blu-ray discs, ultra-density discs, flash memory cards (such as SD cards, min SD cards, Micro-SD cards, etc.), magnetic floppy disks, etc. Computer-readable storage media do not include carrier waves and instantaneous electronic signals transmitted wirelessly or by wire.

[0093] Executable code is stored on the memory 410, and when the executable code is processed by the processor 420, it can cause the processor 420 to execute some or all of the methods described above.

[0094] In addition, the method according to the present invention can also be implemented as a computer program or a computer program product, which includes computer program code instructions for performing some or all of the steps in the above-described method of the present invention.

[0095] Alternatively, the present invention can also be implemented as a non-transitory machine-readable storage medium (or computer-readable storage medium, or machine-readable storage medium) having executable code (or computer program, or computer instruction code) stored thereon. When the executable code (or computer program, or computer instruction code) is executed by a processor of an electronic device (or an electronic device, server, etc.), the processor is caused to execute some or all of the steps of the above-described method according to the present invention.

[0096] Those skilled in the art will also understand that the various logic blocks, modules, circuits, and algorithm steps described herein can be implemented as a combination of electronic hardware, computer software, or both.

[0097] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems and methods according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a part of code that contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than marked in the accompanying drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0098] The various embodiments of the present invention have been described above. The above description is exemplary and not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments.

Claims

1. A method for video acquisition and processing of a respiratory endoscope, characterized in that, Including: Obtaining a trachea scan image; Generating a bronchial tree diagram based on the trachea scan image, and marking the lesion occurrence area on the bronchial tree diagram; Generating a lesion navigation path based on the bronchial tree diagram and the lesion occurrence area; Collecting a real-time endoscopic video based on the lesion navigation path; Determining whether the endoscope reaches the airway branch fork corresponding to the lesion occurrence area based on the real-time endoscopic video; If it is determined that the endoscope reaches the airway branch fork corresponding to the lesion occurrence area, switching the video compression strategy of the real-time endoscopic video from a first compression strategy to a second compression strategy; wherein, the data compression rate of the first compression strategy is greater than the data compression rate of the second compression strategy; The first compression strategy includes: Reading each video frame of the real-time endoscopic video and obtaining the moving speed of the endoscope; Determining the key frame division interval duration based on the moving speed and the preset endoscopic stepping distance of the endoscope; Determining key frames from each video frame based on the key frame division interval duration to obtain a plurality of key frames; Determining a plurality of reference frames between each pair of adjacent key frames; Performing motion compensation based on each reference frame to obtain residual data; Performing video compression based on the residual data; The determining a plurality of reference frames between each pair of adjacent key frames includes: Starting from the previous key frame in the current adjacent key frames, determining the next video frame separated by two video frames as a reference frame until the number of video frames between the last reference frame and the subsequent key frame in the current adjacent key frames is less than or equal to 2; Determining whether the endoscope reaches the airway branch fork corresponding to the lesion occurrence area based on the real-time endoscopic video includes: Monitoring the key frame division update status; When an updated key frame is generated during monitoring, extracting the updated video image corresponding to the updated key frame; Inputting the main trunk deviation direction of the lesion navigation path and the updated video image into a branch fork classification model to obtain a branch fork type result output by the branch fork classification model; wherein, the branch fork classification model includes a main trunk fork classification model, a first bronchial fork classification model, and a second bronchial fork classification model; wherein, the main trunk fork classification model is trained based on a main trunk trachea fork image training set, the first bronchial fork classification model is trained based on a first bronchial fork image training set, and the second bronchial fork classification model is trained based on a second bronchial fork image training set; Determining whether the endoscope reaches the airway branch fork corresponding to the lesion occurrence area based on the branch fork type result; Determining whether the endoscope reaches the airway branch fork corresponding to the lesion occurrence area based on the branch fork type result includes: If the branch fork type result matches the airway branch fork corresponding to the lesion occurrence area, determining that the endoscope reaches the airway branch fork corresponding to the lesion occurrence area.

2. The method for collecting and processing respiratory endoscopy video according to claim 1, wherein Switching the video compression strategy of the real-time endoscopic video from a first compression strategy to a second compression strategy, wherein the second compression strategy includes: Performing lossless compression on the region of interest of the real-time endoscopic video.

3. The method for collecting and processing respiratory endoscope videos according to claim 1, wherein Generating a bronchial tree diagram based on the trachea scan image includes: Performing three-dimensional reconstruction on the trachea scan image to obtain an initial three-dimensional bronchial image; Performing preprocessing on the initial three-dimensional bronchial image to obtain a preprocessed three-dimensional image; Performing segmentation and extraction on the preprocessed three-dimensional image through a 3D convolutional neural network to obtain the bronchial tree diagram.

4. The method for collecting and processing respiratory endoscopy videos according to claim 1, wherein, Generating a lesion navigation path based on the bronchial tree diagram and the lesion occurrence area includes: Extracting the centerline of the bronchial tree diagram through a topological thinning algorithm; Planning the lesion navigation path by a path planning algorithm based on the Euclidean distance, starting from the entrance of the centerline and ending at the airway branch fork corresponding to the lesion occurrence area.

5. An electronic device, characterized in that, Including: A processor and a memory, with executable code stored on the memory. When the executable code is executed by the processor, the processor executes the method according to any one of claims 1-4.

6. A non-transitory machine-readable storage medium, characterized in that, With executable code stored thereon. When the executable code is executed by the processor of an electronic device, the processor executes the method according to any one of claims 1-4.

Images