Breathing endoscope video collecting and processing method, electronic equipment and storage medium

Abstract

The invention discloses a respiratory endoscope video collecting and processing method, electronic equipment and a storage medium, and belongs to the technical field of video processing. Generating a bronchial tree diagram based on the trachea scanning image, and marking a focus generation area on the bronchial tree diagram; generating a focus navigation path based on the bronchial tree diagram and the focus generation area; acquiring a real-time endoscope video based on the focus navigation path; based on the real-time endoscope video, determining whether the endoscope reaches an airway branch fork corresponding to the focus occurrence area; if it is determined that the endoscope reaches the airway branch fork corresponding to the focus occurrence area, the video compression strategy of the real-time endoscope video is switched from the first compression strategy to the second compression strategy. According to the method, the video compression strategy can be reasonably arranged in the diagnosis and treatment process of the respiratory endoscope, the high-definition requirement of a focus area is ensured while the video transmission timeliness is ensured, and the accuracy and reliability of respiratory endoscope examination are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of video processing, and in particular relates to a respiratory endoscope video acquisition and processing method, electronic equipment and storage medium. Background Art

[0002] Respiratory endoscopy is an important diagnostic and treatment technology. It plays an important role in the diagnosis and treatment of respiratory diseases and is one of the important components of respiratory interventional medicine. During the diagnosis and treatment of respiratory endoscopy, high requirements are placed on the timeliness of video transmission and the clarity of image display. Therefore, it is necessary to achieve efficient and clear transmission of endoscopic videos during the diagnosis and treatment of respiratory endoscopy. Video compression technology is a technology that can reduce the amount of data transmitted or stored while providing high-quality video images, thereby helping to improve data transmission efficiency. In the current respiratory interventional diagnosis and treatment process, usually only a single video compression technology is used in the video processing and transmission process. This situation can easily cause the low-value images before the endoscope reaches the lesion to be unable to be fully compressed, and the high-value images after the endoscope reaches the lesion to be over-compressed, resulting in image quality loss.

[0003] In view of this, it is urgent to propose a respiratory endoscopy video acquisition and processing method so as to reasonably arrange the video compression strategy in the process of respiratory endoscopy diagnosis and treatment, ensure the timeliness of video transmission while ensuring the high image quality requirements of the lesion area, which is conducive to improving the accuracy and reliability of respiratory endoscopy examination. Summary of the invention

[0004] In order to overcome the problems existing in the related art, the present invention provides a respiratory endoscopy video acquisition and processing method, an electronic device and a storage medium. The respiratory endoscopy video acquisition and processing method can reasonably arrange the video compression strategy in the diagnosis and treatment process of respiratory endoscopy, while ensuring the timeliness of video transmission, ensuring the high image quality requirements of the lesion area, which is conducive to improving the accuracy and reliability of respiratory endoscopy examination.

[0005] The present invention provides a respiratory endoscope video acquisition and processing method, comprising: Obtain tracheal scan images; Generate a bronchial tree diagram based on the tracheal scan image, and mark the lesion occurrence area on the bronchial tree diagram; Generate a lesion navigation path based on the bronchial tree diagram and the lesion occurrence area; Collect real-time endoscopic video based on the lesion navigation path; Determine whether the endoscope has reached the airway branch corresponding to the lesion area based on real-time endoscopic video; If it is determined that the endoscope has reached the airway branch corresponding to the lesion area, the video compression strategy of the real-time endoscopic video is switched from the first compression strategy to the second compression strategy; wherein the data compression rate of the first compression strategy is greater than the data compression rate of the second compression strategy.

[0006] Further, the video compression strategy of the real-time endoscopic video is switched from the first compression strategy to the second compression strategy, wherein the first compression strategy includes: Read each video frame of the real-time endoscopic video and obtain the moving speed of the endoscope; Determine the key frame division interval duration based on the moving speed and the preset endoscope stepping distance of the endoscope; Determine a key frame from each video frame based on the key frame division interval duration to obtain a plurality of key frames; determining a plurality of reference frames between each pair of adjacent key frames; Perform motion compensation based on each reference frame to obtain residual data; Video compression based on residual data.

[0007] Further, determining a plurality of reference frames between each pair of adjacent key frames comprises: Starting from the previous key frame among the current adjacent key frames, the next video frame separated by two video frames is determined as a reference frame until the number of video frames between the last reference frame and the subsequent key frame among the current adjacent key frames is less than or equal to 2.

[0008] Further, the video compression strategy of the real-time endoscopic video is switched from the first compression strategy to the second compression strategy, wherein the second compression strategy includes: Lossless compression of regions of interest in real-time endoscopic videos.

[0009] Further, determining whether the endoscope has reached the airway branch corresponding to the lesion area based on the real-time endoscopic video includes: Monitor the key frame division update status; When the generation of an update key frame is detected, an update video image corresponding to the update key frame is extracted; The trunk deviation direction and the updated video image of the lesion navigation path are input into the branch fork classification model to obtain the branch fork type result output by the branch fork classification model; wherein the branch fork classification model includes a trunk fork classification model, a first bronchial fork classification model and a second bronchial fork classification model; wherein the trunk fork classification model is trained based on the trunk tracheal fork image training set, the first bronchial fork classification model is trained based on the first bronchial fork image training set, and the second bronchial fork classification model is trained based on the second bronchial fork image training set; Based on the branch fork type results, determine whether the endoscope has reached the airway branch fork corresponding to the lesion area.

[0010] Further, determining whether the endoscope has reached the airway branch corresponding to the lesion occurrence area based on the branch fork type result includes: If the branch fork type result matches the airway branch fork corresponding to the lesion occurrence area, it is determined that the endoscope has reached the airway branch fork corresponding to the lesion occurrence area.

[0011] Further, generating a bronchial dendrogram based on the tracheal scan image includes: Perform three-dimensional reconstruction on the tracheal scan image to obtain an initial three-dimensional bronchial image; Preprocessing the initial bronchial three-dimensional image to obtain a preprocessed three-dimensional image; The preprocessed three-dimensional image was segmented and extracted using a 3D convolutional neural network to obtain a bronchial dendrogram.

[0012] Furthermore, generating a lesion navigation path based on the bronchial tree diagram and the lesion occurrence area includes: The centerline of the bronchial dendrogram was extracted using a topological thinning algorithm; The lesion navigation path is planned by using a path planning algorithm based on Euclidean distance, with the entrance of the centerline as the starting point and the airway branch corresponding to the lesion area as the end point.

[0013] The present invention also provides an electronic device, including: A processor; and a memory having executable codes stored thereon, which, when executed by the processor, cause the processor to execute the method as described above.

[0014] The present invention further provides a non-transitory machine-readable storage medium having executable codes stored thereon. When the executable codes are executed by a processor of an electronic device, the processor is caused to execute the method as described above.

[0015] The technical solution provided by the present invention may include the following beneficial effects: The respiratory endoscope video acquisition and processing method, electronic device and storage medium provided by the present invention obtains a tracheal scan image, generates a bronchial tree diagram based on the tracheal scan image, marks the lesion occurrence area on the bronchial tree diagram, and generates a lesion navigation path based on the bronchial tree diagram and the lesion occurrence area. This can help the examiner to efficiently and accurately probe the endoscope into the lesion occurrence area.

[0016] The present invention can collect real-time endoscopic video based on the lesion navigation path, and then determine whether the endoscope has reached the airway branch fork corresponding to the lesion area based on the real-time endoscopic video. After passing through the airway branch fork corresponding to the lesion area, the endoscope can enter the lesion area. At this time, if it is determined that the endoscope has reached the airway branch fork corresponding to the lesion area, the video compression strategy of the real-time endoscopic video is switched from the first compression strategy to the second compression strategy, wherein the data compression rate of the first compression strategy is greater than the data compression rate of the second compression strategy. Thereby, the low-value images before the endoscope reaches the lesion can be fully compressed, and the high-value images after the endoscope reaches the lesion can be reasonably compressed while avoiding the situation of image quality loss.

[0017] In general, the present invention can reasonably arrange the video compression strategy during the diagnosis and treatment process of respiratory endoscopy, while ensuring the timeliness of video transmission and ensuring the high image quality requirements of the lesion area, which is conducive to improving the accuracy and reliability of respiratory endoscopy. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is one of the flow charts of the respiratory endoscope video acquisition and processing method shown in the embodiment of the present invention; Figure 2 This is a second flow chart of a respiratory endoscope video acquisition and processing method according to an embodiment of the present invention; Figure 3 FIG3 is a flow chart of a respiratory endoscope video acquisition and processing method according to an embodiment of the present invention; Figure 4 It is a schematic diagram of the structure of an electronic device shown in an embodiment of the present invention. DETAILED DESCRIPTION

[0019] The preferred embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the preferred embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to make the present invention more thorough and complete, and to fully convey the scope of the present invention to those skilled in the art.

[0020] The terms used in the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The singular forms "a", "the" and "the" used in the present invention and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0021] It should be understood that although the terms "first", "second", "third", etc. may be used to describe various information in the present invention, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0022] During the diagnosis and treatment of respiratory endoscopy, high requirements are placed on the timeliness of video transmission and the clarity of image display. Therefore, it is necessary to achieve efficient and clear transmission of endoscopic videos during the diagnosis and treatment of respiratory endoscopy. Video compression technology is a technology that can reduce the amount of transmitted or stored data while providing high-quality video images, thereby improving data transmission efficiency. Although existing video compression technology is relatively mature, if a single video compression technology is used in the video processing and transmission process of the entire diagnosis and treatment process, it is easy to cause the low-value images before the endoscope reaches the lesion to be unable to be fully compressed, and the high-value images after the endoscope reaches the lesion to be over-compressed, resulting in image quality loss.

[0023] The present invention proposes a respiratory endoscopy video acquisition and processing method, so as to reasonably arrange the video compression strategy in the process of respiratory endoscopy diagnosis and treatment, ensure the timeliness of video transmission, and at the same time guarantee the high image quality requirements of the lesion area, which is beneficial to improve the accuracy and reliability of respiratory endoscopy examination.

[0024] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings and embodiments.

[0025] Example 1

[0026] See also Figure 1 The respiratory endoscope video acquisition and processing method shown in the embodiment of the present invention includes: S101, obtaining a tracheal scan image, which refers to an image obtained by performing a CT scan of the lung area including the trachea, bronchi and lung lobes. CT scanning is of great value in the diagnosis of tracheal lesions, and can provide detailed image information of the trachea and its surrounding structures to help doctors identify and evaluate lesions. CT scanning can clearly show the position, size, wall thickness and morphological changes of the trachea, which helps to determine whether there are organic lesions, such as stenosis, tumors, polyps or other abnormalities.

[0027] S102, generating a bronchial tree diagram based on the tracheal scan image, marking the lesion occurrence area on the bronchial tree diagram, and importing the tracheal scan image into a navigation system, such as Virtual Bronchoscopic Navigation (VBN), which is a navigation technology based on CT image data and is used to assist bronchoscopy and diagnose lung lesions. A virtual bronchial tree image is generated by the navigation system, and the lesion occurrence area of ​​the target lesion is marked.

[0028] S103, generating a lesion navigation path based on the bronchial tree diagram and the lesion occurrence area, using the tree branches in the bronchial tree diagram to plan a path to the lesion occurrence area. During the inspection, the endoscope entering the respiratory tract will gradually reach the lesion occurrence area along the aforementioned path.

[0029] S104, real-time endoscopic video is collected based on the lesion navigation path, and real-time endoscopic video is collected along the lesion navigation path through an endoscope (i.e., bronchoscope) used to examine the trachea and bronchi. An endoscope is a medical device used to examine and treat internal organs of the human body. It is inserted into the natural cavity or small incision of the human body, and uses optical or electronic imaging technology to transmit images in the body to an external display to help doctors diagnose and treat.

[0030] S105, based on the real-time endoscopic video, determine whether the endoscope has reached the airway branch corresponding to the lesion area. From the anatomical structure of the bronchial tree, it can be known that the overall tracheal structure is divided into the trunk trachea, the right main bronchus and the left main bronchus, and there is a bronchial bifurcation at the bifurcation of the right main bronchus and the left main bronchus.

[0031] On the one hand, the right main bronchus is divided into the right upper lobe, the right middle lobe and the right lower lobe, and there is a right lobe bifurcation at the bifurcation of the right upper lobe, the right middle lobe and the right lower lobe. Further, the right upper lobe includes the right upper lobe tip segment, the right upper lobe front segment and the right upper lobe rear end, and there is a right upper lobe segment bifurcation at the bifurcation of the right upper lobe tip segment, the right upper lobe front segment and the right upper lobe rear end. In addition, the right middle lobe includes the right middle lobe lateral segment and the right middle lobe medial segment, and there is a right middle lobe segment bifurcation at the bifurcation of the right middle lobe lateral segment and the right middle lobe medial segment. Furthermore, the right lower lobe includes the right lower lobe dorsal segment, the right lower lobe anterior basal segment, the right lower lobe inner basal segment, the right lower lobe outer basal segment and the right lower lobe posterior basal segment, and there is a right lower lobe segment bifurcation at the bifurcation of the right lower lobe dorsal segment, the right lower lobe anterior basal segment, the right lower lobe inner basal segment, the right lower lobe outer basal segment and the right lower lobe posterior basal segment.

[0032] On the other hand, the left main bronchus is divided into the left upper lobe and the left lower lobe, and there is a left lobe bifurcation at the bifurcation of the left upper lobe and the left lower lobe. Further, the left upper lobe includes the posterior segment of the left upper lobe tip, the anterior segment of the left upper lobe, the upper lingual segment of the left upper lobe, and the lower lingual segment of the left upper lobe, and there is a left upper lobe segment bifurcation at the bifurcation of the posterior segment of the left upper lobe tip, the anterior segment of the left upper lobe, the upper lingual segment of the left upper lobe, and the lower lingual segment of the left upper lobe. In addition, the left lower lobe includes the dorsal segment of the left lower lobe, the anterior basal segment of the left lower lobe, the outer basal segment of the left lower lobe, and the posterior basal segment of the left lower lobe, and there is a left lower lobe segment bifurcation at the bifurcation of the dorsal segment of the left lower lobe, the anterior basal segment of the left lower lobe, the outer basal segment of the left lower lobe, and the posterior basal segment of the left lower lobe.

[0033] There are differences between the various bifurcations mentioned above in terms of bifurcation shape, bifurcation size, bifurcation branch angle, bifurcation texture, etc. Therefore, the bifurcation reached by the endoscope can be identified to confirm whether it has reached the airway branch bifurcation corresponding to the lesion area. Assuming that the lesion area is located in the anterior segment of the left upper lobe, it can be identified whether the endoscope has reached the bifurcation of the left upper lobe segment. If so, it is determined that the endoscope has reached the airway branch bifurcation corresponding to the anterior segment of the left upper lobe.

[0034] S106, if it is determined that the endoscope reaches the airway branch fork corresponding to the lesion occurrence area, the video compression strategy of the real-time endoscopic video is switched from the first compression strategy to the second compression strategy. The data compression rate of the first compression strategy is greater than the data compression rate of the second compression strategy. This is because before the endoscope reaches the lesion occurrence area, the endoscope is moving in the trachea, and the image collected at this time does not have the significance for lesion inspection, so the first compression strategy with a higher data compression rate can be used to compress the video data collected before the endoscope reaches the lesion occurrence area. After the endoscope reaches the lesion occurrence area, the endoscope is ready to enter the lesion occurrence area for inspection, so it is necessary to ensure that the image quality remains at a high level and does not affect real-time transmission, so the second compression strategy with a lower data compression rate is used to compress the video data collected after the endoscope reaches the lesion occurrence area.

[0035] The embodiment of the present invention obtains a tracheal scanning image, and then generates a bronchial tree diagram based on the tracheal scanning image, and marks the lesion occurrence area on the bronchial tree diagram, and then generates a lesion navigation path based on the bronchial tree diagram and the lesion occurrence area. Thereby, it can help the examiner to efficiently and accurately probe the endoscope into the lesion occurrence area. The present invention can collect real-time endoscopic video based on the lesion navigation path, and then determine whether the endoscope reaches the airway branch fork corresponding to the lesion occurrence area based on the real-time endoscopic video. After passing through the airway branch fork corresponding to the lesion occurrence area, it can enter the lesion occurrence area. At this time, if it is determined that the endoscope reaches the airway branch fork corresponding to the lesion occurrence area, the video compression strategy of the real-time endoscopic video is switched from the first compression strategy to the second compression strategy, wherein the data compression rate of the first compression strategy is greater than the data compression rate of the second compression strategy. Thereby, the low-value picture before the endoscope reaches the lesion can be fully compressed, and the high-value picture after the endoscope reaches the lesion can be reasonably compressed while avoiding the situation of image quality loss.

[0036] In general, the present invention can reasonably arrange the video compression strategy during the diagnosis and treatment process of respiratory endoscopy, while ensuring the timeliness of video transmission and ensuring the high image quality requirements of the lesion area, which is conducive to improving the accuracy and reliability of respiratory endoscopy.

[0037] In some embodiments, the first compression strategy and the second compression strategy may be further designed. Figure 2 This is a second flow chart of the respiratory endoscope video acquisition and processing method shown in an embodiment of the present invention. Figure 2 The respiratory endoscopy video acquisition and processing method shown in the embodiment of the present invention may include: Before the endoscope reaches the airway branch corresponding to the lesion area, compression is performed using a first compression strategy, wherein the first compression strategy includes: S201, read each video frame of the real-time endoscopic video and obtain the moving speed of the endoscope. Determine the moving speed of the endoscope based on each video frame of the real-time endoscopic video. First, receive the current video frame and obtain the previous video frame adjacent to it. Then, generate an optical flow map relative to the previous video frame for the current video frame (the optical flow algorithm in the OpenCV library can be used, for example, based on sparse optical flow (Lucas-Kanade method) and dense optical flow (Farneback method)). The vectors in the optical flow map represent the movement direction and size of the pixel points. Then, according to the direction of each vector in the optical flow map, determine whether the endoscope is in a moving state (if the modulus of the vector sum is less than a preset threshold, it is determined that the device is in a moving state). Furthermore, when it is determined that the endoscope is in a moving state, all the vectors in the optical flow map are used to calculate the moving speed corresponding to the current image frame. For example, the moving speed of the endoscope can be estimated by calculating the average value of all vectors. There are various ways to obtain the moving speed of the endoscope. For example, the moving speed of the endoscope can be obtained by setting a speed detection sensor on the endoscope. In practical applications, the method for obtaining the moving speed of the endoscope needs to be determined according to the actual application situation, and the present invention does not make a sole limitation.

[0038] S202, based on the moving speed and the preset endoscope stepping distance of the endoscope, determine the key frame division interval duration. The preset endoscope stepping distance of the endoscope is divided by the moving speed to determine the key frame division interval duration. The moving speed at this time may also be the average speed of the endoscope during the endoscope stepping distance process, or the instantaneous moving speed at the midpoint of the endoscope stepping distance, which needs to be determined according to the actual application situation and is not limited here.

[0039] S203, determining a key frame from each video frame based on the key frame division interval duration, and obtaining a plurality of key frames. Since the moving speed may fluctuate, the key frame division interval duration may also fluctuate, and thus the number of video frames between different adjacent key frames may also differ. This method is advantageous for setting key frames at various relatively fixed positions in the trachea (because the endoscope stepping distance is fixed), and can improve the decoding quality of real-time endoscopic video and the efficiency of random access.

[0040] S204, determining multiple reference frames between each pair of adjacent key frames. Starting from the previous key frame in the current adjacent key frames, the next video frame separated by two video frames is determined as a reference frame until the number of video frames between the last reference frame and the next key frame in the current adjacent key frames is less than or equal to 2. Assuming that there are 8 video frames between the current adjacent key frames, the previous key frame is the video frame numbered 1, and the next key frame is the video frame numbered 10, then the video frames numbered 4 and 7 can be used as reference frames.

[0041] S205, motion compensation is performed based on each reference frame to obtain residual data. First, the current video frame can be divided into multiple fixed-size blocks, usually 16×16 pixel macroblocks or 64×64 pixel macroblocks, and then a reference frame adjacent to the current video frame is selected, and then the reference block most similar to the current macroblock in the current video frame is searched in the reference frame, and search methods such as diamond search and hexagon search can be used. Next, the matching cost between the current macroblock and the reference block corresponding to the current macroblock is calculated, and the cost function such as the sum of absolute differences (SAD) and mean square error (MSE) can be used for calculation. Next, the macroblock with the smallest matching cost is selected, and its position offset relative to its corresponding reference block is the motion vector. Furthermore, the calculated motion vector can be encoded and transmitted to the decoding end, and the decoding end reconstructs the current video frame from the reference frame according to the motion vector, generates a prediction frame and divides the prediction frame into multiple fixed-size prediction blocks. Then, the residual between each macroblock and its corresponding prediction block is calculated respectively, and the macroblock is subtracted from the prediction block. After the residual is calculated for each macroblock, residual data is obtained.

[0042] S206, video compression is performed based on the residual data. The residual data is encoded to reduce the amount of data, thereby achieving efficient data compression. Transform coding can be used: such as discrete cosine transform (DCT), the residual is converted from the time domain to the frequency domain, and then the coefficients are quantized to reduce high-frequency information. Entropy coding can also be used: such as Huffman coding, the quantized coefficients are losslessly encoded.

[0043] If it is determined that the endoscope has reached the airway branch corresponding to the lesion area, the video compression strategy of the real-time endoscopic video is switched from the first compression strategy to the second compression strategy. The second compression strategy includes: Lossless compression of the region of interest of real-time endoscopic video. First, the cv2.selectROI function of OpenCV can be used to extract the lesion area as the region of interest (ROI). Then, the region of interest is losslessly compressed to retain all details, while the background region (BG) can be lossily compressed to reduce the amount of data. Lossless compression can be performed by integer wavelet transform, embedded zerotree coding (EZW) and Huffman coding. Lossy compression can be performed by JPEG (Joint Photographic Experts Group) compression algorithm and SPIHT (SetPartitioning in Hierarchical Trees) compression algorithm. Finally, the compressed region of interest and background region data are merged to form the final coded stream.

[0044] In some embodiments, the planning method of the lesion navigation path can be further designed, and then the updated video image corresponding to the key frame on the lesion navigation path can be extracted and input into the trained branch fork classification model to identify the type of branch fork reached by the endoscope, so as to determine whether the airway branch fork corresponding to the lesion area has been reached. Figure 3 This is a flow chart of the respiratory endoscope video acquisition and processing method shown in the embodiment of the present invention. Figure 3 The respiratory endoscope video acquisition and processing method shown in the embodiment of the present invention includes: S301, generate a bronchial tree diagram based on the tracheal scan image. The acquired tracheal scan image is saved in DICOM format, and then the tracheal scan image is three-dimensionally reconstructed, for example, the tracheal scan image in DICOM format is imported into a three-dimensional reconstruction software, such as 3D Slicer, VTK or Matlab, so as to generate an initial bronchial three-dimensional image. The initial bronchial three-dimensional image is preprocessed, such as denoising, contrast enhancement, artifact removal, etc., to obtain a preprocessed three-dimensional image. Then, the preprocessed three-dimensional image can be segmented and extracted through a 3D convolutional neural network to obtain a bronchial tree diagram.

[0045] S302, generate a lesion navigation path based on the bronchial tree diagram and the lesion occurrence area. First, the center line of the bronchial tree diagram is extracted through a topological refinement algorithm to provide a basis for path planning. Then, the lesion navigation path can be planned by a path planning algorithm based on Euclidean distance, with the entrance of the center line as the starting point and the airway branch fork corresponding to the lesion occurrence area as the end point. In some implementation scenarios, the path can also be smoothed by a Bezier curve interpolation method to reduce the jitter of the path and improve the fluency of navigation. The feasibility of the path can also be verified in combination with the patient's three-dimensional lung model to ensure that the bronchoscope can pass smoothly.

[0046] S303, real-time endoscopic video is collected based on the lesion navigation path. Real-time endoscopic video is collected along the lesion navigation path through an endoscope (i.e., bronchoscope) used to examine the trachea and bronchi. An endoscope is a medical device used to examine and treat internal organs of the human body. It is inserted into the natural cavity or small incision of the human body and uses optical or electronic imaging technology to transmit images in the body to an external display to help doctors diagnose and treat.

[0047] S304, monitoring the key frame division update status, and when monitoring the generation of the updated key frame, extracting the updated video image corresponding to the updated key frame. The key frame can be divided in real time with the real-time acquisition of the real-time endoscopic video. When monitoring the generation of the newly divided updated key frame, the updated video image corresponding to the updated key frame can be extracted.

[0048] S305, input the main deviation direction of the lesion navigation path and the updated video image into the branch fork classification model, and obtain the branch fork type result output by the branch fork classification model. The branch fork classification model includes a main fork classification model, a first bronchial fork classification model and a second bronchial fork classification model. Among them, the main fork classification model is trained based on the main tracheal fork image training set, the first bronchial fork classification model is trained based on the first bronchial fork image training set, and the second bronchial fork classification model is trained based on the second bronchial fork image training set. The first bronchial fork classification model can be used to identify the branch fork type result in the right main bronchus, and the second bronchial fork classification model can be used to identify the branch fork type result in the left main bronchus, without making a unique limitation. Therefore, the main fork classification model, the first bronchial fork classification model or the second bronchial fork classification model can be selected based on the main deviation direction of the lesion navigation path for identification and detection, thereby improving the recognition accuracy.

[0049] The above-mentioned trunk fork classification model, first bronchial fork classification model and second bronchial fork classification model can all be trained based on a large amount of labeled data using a supervised machine learning algorithm, for example, a ResNet deep convolutional neural network classification model can be used for training.

[0050] First, in the training of the trunk bifurcation classification model, a large number of training images of bronchial bifurcations and non-bronchial bifurcations can be collected and the bifurcation positions can be marked, so that the trunk bifurcation classification model has the ability to distinguish between bronchial bifurcations and non-bronchial bifurcations.

[0051] Secondly, in the training of the first bronchial fork classification model, a large number of training images of right lobe fork, right upper lobe segment fork, right middle lobe segment fork and right lower lobe segment fork are collected, and each of the aforementioned training images is annotated respectively, and then the annotated training images are input into the initial ResNet deep convolutional neural network classification model for training, and the prediction probabilities of the four types of results of right lobe fork, right upper lobe segment fork, right middle lobe segment fork and right lower lobe segment fork are output respectively, and the cross entropy loss function is used to determine whether the loss function converges. If converged, the final first bronchial fork classification model can be output. In actual operation, the fork type with the highest prediction probability is output as the branch fork type result.

[0052] Furthermore, in the training of the second bronchial fork classification model, a large number of training images of the left lobe fork, the left upper lobe segment fork and the left lower lobe segment fork can be collected, and each of the aforementioned training images can be annotated respectively, and then the annotated training images are input into the initial ResNet deep convolutional neural network classification model for training, and the prediction probabilities of the three types of results of the left lobe fork, the left upper lobe segment fork and the left lower lobe segment fork are output respectively, and the cross entropy loss function is used to determine whether the loss function converges. If converged, the final second bronchial fork classification model can be output. In actual operation, the fork type with the highest prediction probability is output as the branch fork type result.

[0053] S306, based on the branch fork type result, determine whether the endoscope has reached the airway branch fork corresponding to the lesion occurrence area. If the branch fork type result matches the airway branch fork corresponding to the lesion occurrence area, it is determined that the endoscope has reached the airway branch fork corresponding to the lesion occurrence area. If the branch fork type result is a left upper lobe segment bifurcation, and the lesion occurrence area is in the upper lingular segment of the left upper lobe, then it means that the branch fork type result matches the airway branch fork corresponding to the lesion occurrence area, and it can be determined that the endoscope has reached the airway branch fork corresponding to the lesion occurrence area.

[0054] Example 2

[0055] Corresponding to the aforementioned application function implementation method, the present invention also provides an electronic device for executing a respiratory endoscopy video acquisition and processing method and corresponding embodiments.

[0056] See also Figure 4 , the electronic device 400 includes a memory 410 and a processor 420.

[0057] The processor 420 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0058] The memory 410 includes various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. Among them, ROM can store static data or instructions required by the processor 420 or other modules of the computer. The permanent storage device can be a readable and writable storage device. The permanent storage device can be a non-volatile storage device that does not lose the stored instructions and data even if the computer is powered off. In some embodiments, the permanent storage device uses a large-capacity storage device (such as a magnetic or optical disk, flash memory) as a permanent storage device. In some other embodiments, the permanent storage device can be a removable storage device (such as a floppy disk, an optical drive). The system memory can be a readable and writable storage device or a volatile readable and writable storage device, such as a dynamic random access memory. The system memory can store some or all instructions and data required by the processor at runtime. In addition, the memory 410 includes a combination of any computer-readable storage media, including various types of semiconductor memory chips (DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks can also be used. The memory 410 may include a readable and / or writable removable storage device, such as a laser disc (CD), a read-only digital versatile disc (such as a DVD-ROM, a double-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (such as an SD card, a mini SD card, a Micro-SD card, etc.), a magnetic floppy disk, etc. The computer-readable storage medium does not contain carrier waves and transient electronic signals transmitted wirelessly or wired.

[0059] The memory 410 stores executable codes, and when the executable codes are processed by the processor 420 , the processor 420 can execute part or all of the methods described above.

[0060] In addition, the method according to the present invention may also be implemented as a computer program or a computer program product, which includes computer program code instructions for executing part or all of the steps in the above method of the present invention.

[0061] Alternatively, the present invention may also be implemented as a non-transitory machine-readable storage medium (or computer-readable storage medium, or machine-readable storage medium) on which executable code (or computer program, or computer instruction code) is stored. When the executable code (or computer program, or computer instruction code) is executed by a processor of an electronic device (or electronic device, server, etc.), the processor executes part or all of the steps of the above-mentioned method according to the present invention.

[0062] Those skilled in the art will also appreciate that the various logical blocks, modules, circuits, and algorithm steps described in conjunction with this description may be implemented as electronic hardware, computer software, or a combination of both.

[0063] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system and method according to multiple embodiments of the present invention. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or code, and the part of the module, program segment or code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0064] The embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments.

Claims

1. A respiratory endoscope video acquisition and processing method, characterized in that: include: Obtain tracheal scan images; generating a bronchial tree diagram based on the tracheal scan image, and marking the lesion occurrence area on the bronchial tree diagram; generating a lesion navigation path based on the bronchial tree diagram and the lesion occurrence area; collecting real-time endoscopic video based on the lesion navigation path; Determining whether the endoscope has reached the airway branch corresponding to the lesion occurrence area based on the real-time endoscopic video; If it is determined that the endoscope has reached the airway branch corresponding to the lesion area, the video compression strategy of the real-time endoscopic video is switched from the first compression strategy to the second compression strategy; wherein the data compression rate of the first compression strategy is greater than the data compression rate of the second compression strategy.

2. The respiratory endoscope video acquisition and processing method according to claim 1, characterized in that: The video compression strategy of the real-time endoscopic video is switched from a first compression strategy to a second compression strategy, wherein the first compression strategy includes: Read each video frame of the real-time endoscopic video and obtain the moving speed of the endoscope; Determining the key frame division interval duration based on the moving speed and a preset endoscope stepping distance of the endoscope; Determine a key frame from each video frame based on the key frame division interval duration to obtain a plurality of key frames; determining a plurality of reference frames between each pair of adjacent key frames; Perform motion compensation based on each reference frame to obtain residual data; Video compression is performed based on the residual data.

3. The respiratory endoscope video acquisition and processing method according to claim 2, characterized in that: Determining a plurality of reference frames between each pair of adjacent key frames comprises: Starting from the previous key frame among the current adjacent key frames, the next video frame separated by two video frames is determined as a reference frame until the number of video frames between the last reference frame and the subsequent key frame among the current adjacent key frames is less than or equal to 2.

4. The respiratory endoscope video acquisition and processing method according to claim 1, characterized in that: The video compression strategy of the real-time endoscopic video is switched from a first compression strategy to a second compression strategy, wherein the second compression strategy includes: The real-time endoscopic video is subjected to lossless compression of the region of interest.

5. The respiratory endoscope video acquisition and processing method according to claim 1, characterized in that: Determining whether the endoscope has reached the airway branch corresponding to the lesion occurrence area based on the real-time endoscopic video includes: Monitor the key frame division update status; When the generation of an update key frame is detected, an update video image corresponding to the update key frame is extracted; The main trunk deviation direction of the lesion navigation path and the updated video image are input into a branch fork classification model to obtain a branch fork type result output by the branch fork classification model; wherein the branch fork classification model comprises a main trunk fork classification model, a first bronchial fork classification model and a second bronchial fork classification model; wherein the main trunk fork classification model is trained based on a main trunk tracheal fork image training set, the first bronchial fork classification model is trained based on a first bronchial fork image training set, and the second bronchial fork classification model is trained based on a second bronchial fork image training set; Based on the branch fork type result, it is determined whether the endoscope has reached the airway branch fork corresponding to the lesion occurrence area.

6. The respiratory endoscope video acquisition and processing method according to claim 5, characterized in that: Determining whether the endoscope has reached the airway branch corresponding to the lesion occurrence area based on the branch fork type result includes: If the branch fork type result matches the airway branch fork corresponding to the lesion occurrence area, it is determined that the endoscope has reached the airway branch fork corresponding to the lesion occurrence area.

7. The respiratory endoscope video acquisition and processing method according to claim 1, characterized in that: Generating a bronchial tree diagram based on the tracheal scan image includes: Performing three-dimensional reconstruction on the tracheal scan image to obtain an initial bronchial three-dimensional image; Preprocessing the initial bronchial three-dimensional image to obtain a preprocessed three-dimensional image; The pre-processed three-dimensional image is segmented and extracted by a 3D convolutional neural network to obtain the bronchial dendrogram.

8. The respiratory endoscope video acquisition and processing method according to claim 1, characterized in that: Generating a lesion navigation path based on the bronchial tree diagram and the lesion occurrence area includes: Extracting the centerline of the bronchial dendrogram by a topological refinement algorithm; The lesion navigation path is planned by a path planning algorithm based on Euclidean distance, with the entrance of the center line as the starting point and the airway branch corresponding to the lesion occurrence area as the end point.

9. An electronic device, characterized in that: include: A processor and a memory, wherein an executable code is stored in the memory, and when the executable code is executed by the processor, the processor executes the method according to any one of claims 1 to 8.

10. A non-transitory machine-readable storage medium, characterized in that: An executable code is stored thereon, and when the executable code is executed by a processor of an electronic device, the processor executes the method as claimed in any one of claims 1 to 8.

Images