Bronchoscope biopsy bleeding level prediction method based on dynamic multi-path deep learning model

Abstract

The invention provides a bronchoscope biopsy bleeding level prediction method based on a dynamic multi-path deep learning model. The method comprises the following steps: acquiring a video clip; feature extraction is carried out, and global features and local features are identified; fusion is carried out to obtain fusion features, and bleeding risk grade classification is carried out; performing dynamic weighting; accumulating the weighted probability distribution; and screening a bleeding risk prediction grade. According to the method for predicting the bleeding level before biopsy based on the dynamic multi-path model, through combination of multiple types of deep learning models, processized video clip analysis is achieved, through combination of the deep learning models and multi-frame fusion analysis, the accuracy and reliability of bleeding risk prediction before biopsy are remarkably improved, and the prediction efficiency of the bleeding level before biopsy is improved. Powerful support is provided for clinical decision making, and the method has important application value.

Description

Technical Field

[0001] The present invention relates to the technical field of video image recognition, and particularly to a method for predicting the bleeding grade of bronchoscopic biopsy based on a dynamic multi-channel deep learning model. Background Art

[0002] Endoscopic technology is an important branch in the medical field. For example, bronchoscopy is an important medical technique. By inserting a bronchoscope into the airway of a patient, the internal conditions of the trachea and bronchi can be directly observed. It is mainly used for the diagnosis and treatment of respiratory diseases such as lung cancer, pulmonary tuberculosis, chronic obstructive pulmonary disease, etc. Bronchoscopy can not only help doctors clarify the lesion site, but also perform biopsy, remove foreign bodies, stop bleeding and other treatment operations.

[0003] Bronchoscopic biopsy is the process of obtaining a sample of diseased tissue through a bronchoscope, which is widely used in the diagnosis of lung and airway diseases such as tumors, interstitial lung diseases, granulomatous diseases and certain infectious diseases. Biopsy can help clarify the nature of the lesion and provide an important basis for clinical diagnosis and treatment. Although bronchoscopic biopsy is a common and important diagnostic method, there are still certain risks during its operation. Common complications include bleeding, pneumothorax, airway spasm and hypoxemia, etc. Especially for patients with a bleeding tendency or cardiopulmonary insufficiency, the risk of biopsy operation will increase significantly. Bleeding during bronchoscopic biopsy is one of the most common complications. During the biopsy process, an experienced bronchoscopic doctor is required to record and evaluate the bleeding grade. Usually, according to existing clinical studies, the bleeding risk can be divided into the following grades:

[0004] Grade 0: No or very little bleeding;

[0005] Grade 1: Minor bleeding, which can stop bleeding by itself with negative pressure aspiration without other hemostatic measures;

[0006] Grade 2: Moderate bleeding, which requires treatment with diluted adrenaline, injection of hemocoagulase or perfusion with pre-cooled normal saline, and the intracavitary treatment can continue;

[0007] Grade 3: Massive bleeding, which requires intervention by using thrombin freeze-dried powder intracavitary, balloon compression, intravenous application of pituitrin, intubation or insertion of a rigid endoscope, etc., and the intracavitary treatment is forced to stop;

[0008] Grade 4: Larger bleeding volume or hemodynamic disorder, requiring blood transfusion, emergency treatment such as surgery and vascular intervention. The assessment of bleeding risk needs to comprehensively consider the patient's medical history, hematological examination results and imaging examination results. For high-risk patients, a detailed risk assessment should be carried out before the operation, and corresponding preventive measures should be taken during the operation, such as local application of hemostatic drugs and mechanical compression hemostasis, etc.

[0009] Deep learning models are increasingly widely used in bronchoscopy, mainly for image recognition and diagnostic assistance. Through deep learning algorithms such as convolutional neural networks (CNNs), the recognition accuracy of bronchoscopy images can be improved to assist doctors in diagnosing and classifying lesions. Research shows that deep learning models have high accuracy and sensitivity in distinguishing between benign and malignant lesions.

[0010] Using a deep learning model to process bronchoscopy images before biopsy can predict the bleeding risk that may occur during the biopsy. By training on a large number of bronchoscopy images, the deep learning model can identify potential high-risk areas and provide a bleeding risk assessment. This method can not only improve the safety of the biopsy operation but also help doctors develop more precise treatment plans.

[0011] However, in the current existing technologies, only a preliminary bleeding risk warning can be provided through simple image recognition, and it is not possible to comprehensively consider the spatial scale (including both overall and local images) and the time scale to give a relatively accurate and reliable bleeding risk warning, which does not significantly help in assisting doctors to improve their bronchoscopy operation level. Summary of the Invention

[0012] Aiming at the deficiencies of the existing technologies, the purpose of the present invention is to provide a method for predicting the bleeding grade of bronchoscopy biopsy based on a dynamic multi-path deep learning model.

[0013] To achieve the aforementioned invention purpose, the technical solutions adopted by the present invention include:

[0014] In a first aspect, the present invention provides a method for predicting the bleeding grade of bronchoscopy biopsy based on a dynamic multi-path deep learning model, which includes:

[0015] Obtain a video segment, the video segment includes a plurality of bronchial images sorted by time, and identify local lesion areas in the bronchial images;

[0016] Use a dynamic multi-path deep learning model to extract features from the bronchial images, the dynamic multi-path deep learning model includes a global branch and a local branch, the global branch identifies the global features of the bronchial images, and the local branch identifies the local features of the local lesion areas;

[0017] Fuse the global features and local features to obtain fused features, and classify the bleeding risk grades based on the fused features to obtain a risk grade probability distribution;

[0018] Dynamically weight each probability value in the risk grade probability distribution to obtain a weighted probability distribution, and the weight coefficient of the dynamic weighting has a positive correlation with the risk level corresponding to the highest probability grade in the risk grade probability distribution;

[0019] For the video clip, accumulate the weighted probability values corresponding to each risk level in the weighted probability distribution to obtain a fused probability distribution.

[0020] Based on the magnitudes of the probability values of each risk level in the fused probability distribution, screen to obtain the bleeding risk prediction level corresponding to the video clip.

[0021] In a second aspect, the present invention also provides a bronchoscope biopsy bleeding level prediction system based on a dynamic multi-path deep learning model, which includes:

[0022] A data acquisition module, configured to acquire a video clip, where the video clip includes a plurality of bronchial images sorted by time, and identify local lesion regions in the bronchial images;

[0023] A feature extraction module, configured to extract features from the bronchial images by using a dynamic multi-path deep learning model, where the dynamic multi-path deep learning model includes a global branch and a local branch, the global branch identifies global features of the bronchial images, and the local branch identifies local features of the local lesion regions;

[0024] A feature fusion module, configured to fuse the global features and local features to obtain fused features, and perform bleeding risk level classification based on the fused features to obtain a risk level probability distribution;

[0025] A feature weighting module, configured to dynamically weight each probability value in the risk level probability distribution to obtain a weighted probability distribution, where the weight coefficient of the dynamic weighting is positively correlated with the risk level corresponding to the highest probability level in the risk level probability distribution;

[0026] A probability fusion module, configured to, for the video clip, accumulate the weighted probability values corresponding to each risk level in the weighted probability distribution to obtain a fused probability distribution;

[0027] A level prediction module, configured to screen to obtain the bleeding risk prediction level corresponding to the video clip based on the magnitudes of the probability values of each risk level in the fused probability distribution.

[0028] In a third aspect, the present invention also provides a readable storage medium, in which a computer program is stored, and when the computer program is run, it executes the steps of the above-mentioned bronchoscope biopsy bleeding level prediction method.

[0029] Based on the above technical solutions, compared with the prior art, the biopsy pre-bleeding level prediction method based on a dynamic multi-path model of the present invention realizes a process-based video clip analysis through the combination of multiple types of deep learning models, and brings the following beneficial effects:

[0030] First, by constructing a bleeding grade prediction process based on bronchoscope video clips and combining deep learning network models of different types and functions, the bleeding risk can be predicted more accurately. This method of combining multiple models can make full use of the advantages of each model and improve the accuracy of overall prediction.

[0031] Secondly, a bronchoscope lesion detection model based on deep learning object detection algorithm can extract local lesion areas with potential biopsy significance from bronchoscope images and obtain the confidence of the lesion areas, which helps doctors accurately locate the lesion areas before biopsy and reduce the risk of misoperation. In addition, through a deep learning image classification model with a two-way parallel backbone network, the whole image information and local information of the image are integrated. One backbone network is used to extract the whole image feature vector, and the other backbone network is used to extract the image features of the local area and fuse them. Finally, the feature vectors generated by the two backbone models are fused again as the prediction feature vector of a single-frame image. This method can analyze the image information more comprehensively and improve the reliability of prediction.

[0032] Furthermore, a classification algorithm that accumulates multi-frame classification probabilities is adopted to fuse the prediction results of multiple frames in the video clip sequence, and the preset bleeding risk grade category weights are used to obtain more accurate and reasonable prediction results, effectively utilizing the global and local information of each frame in the video segment and improving the stability and accuracy of prediction.

[0033] Finally, by accurately predicting the bleeding risk, doctors can take corresponding preventive measures before biopsy, reduce the bleeding risk during the biopsy operation, and improve the safety of patients.

[0034] In summary, the method provided by the present invention significantly improves the accuracy and reliability of bleeding risk prediction before biopsy through the combination of deep learning models and multi-frame fusion analysis, provides strong support for clinical decision-making, and has important application value.

[0035] The above description is only an overview of the technical solution of the present invention. In order to enable those skilled in the art to understand the technical means of the present application more clearly and implement it according to the content of the specification, the following is a detailed description with reference to the preferred embodiments of the present invention and the accompanying drawings. Brief Description of the Drawings

[0036] Figure 1 It is a schematic diagram of the process of the first step of the bronchoscope biopsy bleeding grade prediction method provided by a typical embodiment of the present invention;

[0037] Figure 2 It is a schematic diagram of the process of the second step of the bronchoscope biopsy bleeding grade prediction method provided by a typical embodiment of the present invention;

[0038] Figure 3 It is a schematic diagram of the process of the third step of the bronchoscope biopsy bleeding grade prediction method provided by a typical embodiment of the present invention;

[0039] Figure 4 It is a schematic diagram of the process of the fourth step of the bronchoscope biopsy bleeding grade prediction method provided by a typical embodiment of the present invention;

[0040] Figure 5 It is a schematic diagram of the process of the fifth step of the bronchoscope biopsy bleeding grade prediction method provided by a typical embodiment of the present invention;

[0041] Figure 6 It is a schematic diagram of the structure of the dynamic multi-channel deep learning model provided by a typical embodiment of the present invention. Detailed implementation manners

[0042] In view of the deficiencies in the prior art, the inventors of this case have, through long-term research and a large number of practices, been able to propose the technical solution of the present invention. The following will further explain the technical solution, its implementation process, principle, etc.

[0043] In the following description, many specific details are set forth in order to fully understand the present invention. However, the present invention can also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited by the specific embodiments disclosed below.

[0044] Moreover, relational terms such as "first" and "second" are only used to distinguish one component or method step with the same name from another, and do not necessarily require or imply any such actual relationship or order between these components or method steps.

[0045] The main purpose of the present invention is to design and implement a dynamic multi-channel bleeding grade classification model using local pictures and whole pictures of lesions before biopsy, and predict the bleeding grade by a parallel analysis method. In the bleeding risk assessment of the present invention, a bleeding grade classification model of several local pictures and whole picture pictures before operation is used for parallel analysis, and the probability of the predicted bleeding risk grade within a certain time period is statistically analyzed. Through dynamic weight probability integration, the accuracy of bleeding risk prediction can be improved. Through comprehensive analysis of the local area of the lesion and the whole image sequence, the deep learning model can more comprehensively evaluate the bleeding risk grade and provide strong support for clinical decision-making. This method can not only reduce the bleeding risk in biopsy operations, but also improve the accuracy and reliability of diagnosis.

[0046] Based on the above purposes and technical concepts, an embodiment of the present invention provides a bronchoscope biopsy bleeding grade prediction method based on a dynamic multi-channel deep learning model, which includes the following steps:

[0047] Obtain a video clip, where the video clip includes a plurality of bronchial images sorted by time, and identify local lesion regions in the bronchial images;

[0048] Use a dynamic multi-path deep learning model to extract features from the bronchial images. The dynamic multi-path deep learning model includes a global branch and a local branch. The global branch identifies the global features of the bronchial images, and the local branch identifies the local features of the local lesion regions;

[0049] Fuse the global features and local features to obtain fused features, and classify the bleeding risk level based on the fused features to obtain a risk level probability distribution;

[0050] Perform dynamic weighting on each probability value in the risk level probability distribution to obtain a weighted probability distribution. The weight coefficient of the dynamic weighting is positively correlated with the risk level corresponding to the highest probability level in the risk level probability distribution;

[0051] For the video clip, accumulate the weighted probability values corresponding to each risk level in the weighted probability distribution to obtain a fused probability distribution;

[0052] Based on the magnitude of the probability values of each risk level in the fused probability distribution, screen to obtain the bleeding risk prediction level corresponding to the video clip.

[0053] In order to improve the practical application performance of the deep learning model in predicting the bleeding level of images before bronchoscopy biopsy, the present invention invented a classification method based on a dynamic multi-path deep learning model. The present invention is applied to the bleeding level analysis of bronchoscopy video clips before actual biopsy. Compared with single-model prediction and single-frame image analysis, it can greatly improve the rationality and accuracy of bleeding level prediction.

[0054] As some typical application examples, the method provided by the embodiments of the present invention for predicting the bleeding grade of biopsy based on a dynamic multi-channel model is a process-based video clip analysis method based on a combination of multiple types of deep learning models. Its technical implementation includes constructing a bleeding grade prediction process based on the pre-biopsy video clip of the bronchoscope, as well as different types and functions of deep learning network models relied on in each step of this process. Its technical implementation includes adopting a bronchoscope lesion detection model based on the deep learning object detection algorithm. This object detection model uses the current general neural network image detection models (such as YOLO, SSD, Faster-Rcnn, FCOS, etc.). This model extracts several lesion areas with potential biopsy significance locally from the bronchoscope pictures and obtains the confidence of the lesion areas. Its technical implementation also includes a deep learning image classification model with a two-way parallel backbone network. The backbone model of this classification model uses the current general neural network image classification models (such as Resnet, EfficientNet, ConvNext, Swin Transformer, etc.). The purpose of constructing this classification model is to effectively fuse the whole image information and the local information of the image. One backbone network is used for extracting the feature vector of the whole picture, and the other parallel backbone network is used for extracting and fusing the image features of several local areas. The feature vectors generated by the two backbone models will be fused again as the prediction feature vector of the current single-frame image, and the probability of each bleeding category predicted by this frame image is generated.

[0055] Its technical implementation also includes a classification algorithm for accumulating the classification probabilities of multiple frames. This algorithm fuses the prediction results of multiple frames in the video clip sequence and uses the preset weights of the bleeding risk grade categories to obtain a more accurate and reasonable prediction result.

[0056] In some embodiments, the process of obtaining the video clip specifically includes the following process:

[0057] Obtain the video stream;

[0058] Specify a time segment in the video stream and intercept the images in the time segment as the video clip.

[0059] In some embodiments, the process of obtaining the local lesion area may specifically include:

[0060] Perform image recognition on the bronchial image to obtain an initial local area;

[0061] Perform size screening on the initial local area to obtain the local lesion area. The size screening is based on the pixel size of the initial local area, and the initial local areas with pixel sizes lower than the threshold are removed.

[0062] In some embodiments, the process of obtaining the local lesion area can be expressed as:

[0063] Calculate the area ratio of the initial local area to the bronchial image where area(·) represents the pixel area, l i represents the initial local area l i =(x i , y i , w i , h i , p i ), x i , y i represent the pixel center coordinates of the initial local area, w i , h i represent the pixel width and pixel length of the initial local area, p i represents the confidence that the initial local area is classified as a lesion, x t represents the bronchial image, i represents the index number of the initial local area, and t represents the index number of the bronchial image;

[0064] Set the ratio threshold of the local image to the whole image as

[0065] If the initial local area l i is retained in the set I t of the local lesion area, otherwise filter out the initial local area l i and its confidence value p t in the lesion confidence set P i .

[0066] In some embodiments, the process of feature extraction is expressed as:

[0067]

[0068]

[0069] where represents the global feature, F 1 (·) represents the global branch, represents the local feature, represents the set of local lesion areas, represents the corresponding lesion confidence set, F 2 (·) represents the local branch, Represents the initial fusion of the local branches, f represents the fusion feature, F(·) represents the feature decoder, f j Represents the eigenvalue corresponding to each risk level in the local feature, n represents the number of local lesion regions, p t Represents the risk level probability distribution, Represents the probability value corresponding to each risk level in the risk level probability distribution, o represents the highest level number of the risk levels.

[0070] In some embodiments, the process of dynamic weighting is expressed as:

[0071] 1 = w 0 + w 1 + w 2 + …… + w o ,

[0072] W o = α o * w0,

[0073] ……

[0074] W 3 = α 3 * W 0 ,

[0075] W 2 = α 2 * W 0 ,

[0076] W 1 = α 1 * W 0

[0077]

[0078] Wherein, w. represents the weight coefficient, α. represents the dynamic coefficient and its value increases with the increase of the subscript value o, W represents the set of weight coefficients, Represents the weighted probability distribution, Represents the probability value corresponding to each risk level in the weighted probability distribution, k = 0, 1, 2 …… o, represents the index of the risk level, Represents the set of weighted probability distributions, P represents the fusion probability distribution, pk represents the probability value corresponding to each risk level in the fusion probability distribution, l represents the index of the risk level corresponding to the maximum value of the probability value, t represents the index of the bronchial image, and m represents the number of bronchial images in the video segment.

[0079] In some embodiments, the screening process of the bleeding risk prediction level is expressed as:

[0080]

[0081] Among them, Level represents the predicted bleeding risk level.

[0082] The bronchoscopic biopsy bleeding grade prediction method according to claim 1, wherein the global branch and the local branch are located in the backbone network of the dynamic multi-way deep learning model, and the architecture of the backbone network adopts any one of Resnet, EfficientNet, ConvNext, and Swin Transformer.

[0083] As a typical application example of the above technical solution, aiming at the problem of low accuracy and rationality of the bronchial bleeding risk prediction of a single model in a single-frame image, the following three main innovation points are proposed in the present invention:

[0084] First, aiming at the problem of predicting the bleeding grade according to the video segment before biopsy, the present invention designs and implements a complete prediction process based on the object detection model and the dynamic multi-way classification model. Exemplarily, the process mainly includes the following steps:

[0085] Step 1. Access the real-time bronchoscope video stream, use the lesion target detection model to be biopsied, detect the lesion targets in each frame of the video stream, and record the coordinate positions and confidence levels of several local lesion areas on each frame of the image in real time. The detection target in the image can be empty;

[0086] Step 2. Obtain the video segment to be predicted. In actual operation, for example, this step requires the user to input the start signal of the video segment to be predicted through the combination keys of the keyboard. Typing control + s represents the start signal of the prediction segment, and control + e represents the end signal of the prediction segment. Of course, the specific method of how to operate and intercept the segment is not limited to the specific method here;

[0087] Step 3. Filter and optimize the set of local lesion areas. This step is only executed for each frame of the image between the two time points of the start and end described in Step 2. Each frame of the image within the predicted video segment can be detected by the lesion target detector to have 0 or more local lesion areas with biopsy significance. In this step, a minimum local target ratio is preset to filter out the local lesion areas with a relatively small ratio, because the local images with a relatively small ratio have lower bleeding risk details and will affect the final prediction effect;

[0088] Step 4. Use the dynamic multi-way classification model and the iterative input algorithm to predict the bleeding grade for each frame. This step is only executed for each frame image between the start and end time points described in Step 2. The model takes each frame image and the local lesion regions selected in Step 3 as inputs, and generates the probability of each bleeding category predicted by this frame image.

[0089] Step 5. Perform multi-frame dynamic weight probability cumulative fusion. This step is only executed at the end time point described in Step 2. In this step, the category probabilities predicted for each frame image generated in Step 4 are cumulatively weighted, where the weights are preset values, and the weights for higher bleeding grades are larger. Here, considering that the bleeding risk brought by higher bleeding grades is greater, larger weights are set. After cumulative weighting, the category with the maximum probability in the cumulative probabilities is taken as the output of the final bleeding risk grade.

[0090] Second, for the problem of single-frame bleeding grade prediction, the present invention designs and implements a dynamic multi-way deep learning network model. The purpose of this model is to use a single-frame image and several corresponding local images in the image to predict the bleeding grade. This model can effectively integrate the global and local information of the image. This model can process different numbers of local images through the corresponding iterative algorithm, and has higher classification accuracy compared to the original classification model. This model is applied to Step 4 of Innovation Point (1).

[0091] Third, for the problem of obtaining bleeding grade prediction in the prediction segment, the present invention designs and implements a multi-frame dynamic weight cumulative fusion algorithm. This algorithm is applied to Step 5 of Innovation Point (1). This algorithm includes the following three steps:

[0092] Set weights for the category probabilities corresponding to each frame;

[0093] Perform within-category accumulation on the probabilities after setting weights;

[0094] Take the category with the highest probability as the final prediction grade.

[0095] The second aspect of the embodiment of the present invention also provides a bronchoscope biopsy bleeding grade prediction system based on a dynamic multi-way deep learning model, which includes:

[0096] A data acquisition module for acquiring a video segment, where the video segment includes a plurality of bronchial images sorted by time, and identifying local lesion regions in the bronchial images;

[0097] A feature extraction module for extracting features from the bronchial images using a dynamic multi-way deep learning model. The dynamic multi-way deep learning model includes a global branch and a local branch. The global branch identifies the global features of the bronchial images, and the local branch identifies the local features of the local lesion regions;

[0098] A feature fusion module, configured to fuse the global feature and the local feature to obtain a fused feature, and perform bleeding risk level classification based on the fused feature to obtain a risk level probability distribution;

[0099] A feature weighting module, configured to dynamically weight each probability value in the risk level probability distribution to obtain a weighted probability distribution, where the weight coefficient of the dynamic weighting is positively correlated with the risk level corresponding to the highest probability level in the risk level probability distribution;

[0100] A probability fusion module, configured to accumulate the weighted probability values corresponding to each risk level in the weighted probability distribution for the video segment to obtain a fused probability distribution;

[0101] A level prediction module, configured to screen and obtain the bleeding risk prediction level corresponding to the video segment based on the probability values of each risk level in the fused probability distribution.

[0102] A third aspect of the embodiments of the present invention further provides a readable storage medium, in which a computer program is stored, and when the computer program is run, it executes the steps of the bronchoscope biopsy bleeding level prediction method provided in any of the above embodiments.

[0103] As some typical examples of the above overall technical solution, the bronchoscope biopsy pre-bleeding level prediction method provided by the present invention mainly includes 5 steps. As shown in Fig. 1- Figure 5 shown:

[0104] Step 1: Refer to Figure 1 , obtain each frame of image from a real-time bronchoscope, and use a lesion detector to detect the lesion area with biopsy significance in each frame;

[0105] Step 2: Refer to Figure 2 Manually input start and end signals to determine the starting point of the video segment to be graded in the video stream;

[0106] Step 3: Refer to Figure 3 Optimize the local lesion area set, and filter out local lesion areas with a relatively low image area ratio;

[0107] Step 4: Refer to Figure 4 Input each frame of image in the video segment to be graded into a dynamic multi-path deep model for classification;

[0108] Step 5: Refer to Figure 5 Accumulate the weights of the classification sequence to obtain the final segment grading prediction.

[0109] To improve the effect of the deep learning model in real-time predicting the bleeding grade during bronchoscopy, the automated prediction process based on the multi-neural network model and the classification method of the dynamic multi-channel deep learning model in this embodiment are used to predict the bleeding grade of the bronchoscopy video segment before biopsy. After the method described in the present invention is applied to the bleeding grade prediction of bronchoscopy, compared with the single-frame classification model prediction method, it can more accurately and reasonably predict the bleeding risk of the biopsy site during the bronchoscopy operation.

[0110] The method for predicting the bleeding grade before biopsy based on the dynamic multi-channel model is a process-based video segment analysis method based on the combination of multiple types of deep learning models. Its technical implementation includes constructing a bleeding grade prediction process for the video segment before bronchoscopy biopsy and an object detection model for detecting the lesion position with biopsy significance included in this process, and a dynamic multi-channel deep learning classification model for single-frame risk grade prediction, as well as a weight accumulation algorithm for fusing the grade prediction probability sequences of each frame within the video segment. This process can effectively utilize the global and local information and sequence information of each picture in the video segment before biopsy to predict the bleeding risk of the lesion biopsy in the video segment. Its technical implementation includes using a bronchoscopy lesion detection model D(·) based on the deep learning object detection algorithm. This object detection model uses the current general neural network image detection model (such as YOLO, SSD, Faster-Rcnn, FCOS, etc.). This model can process a single-frame bronchoscopy picture x t , where t represents the frame number, and predicts the position l i =(x i , y i , w i , h i , p i ) of the local potentially biopsy-significant lesion area in the predicted picture, where x i , y i represent the pixel center coordinates of the detection target in the image, w i , h i represent the pixel width and pixel length of the detection target in the image, and pi represents the confidence of the lesion area.

[0111] Its technical implementation also includes a two-way parallel backbone deep learning image classification model F(·) with dynamic input. The backbone model of this classification model uses the current general neural network image classification model (such as Resnet, EfficientNet, ConvNext, Swin Transformer, etc.). The purpose of constructing this classification model is to integrate the whole-image information and the local information of the image. This classification model includes two parallel backbone networks including F 1 (·) and F2 (·), where F 1 (·) is used to extract the feature vector of the whole image F 2 (·) is used to extract the feature vectors of several detected local lesion regions in the image n is the number of detected local lesions, which can be an empty set, and each feature vector is accumulated and fused according to the local lesion confidence weight into a single feature vector After accumulation, it is then combined with the global feature vector for fusion. The obtained global-local feature fusion vector f is used as the feature vector for the final grade classification, and the probability value for each category is obtained where represents the probability that the predicted bleeding grade of the t-th frame is k (k = 0, 1, 2, 3, 4). Of course, in this embodiment, the risk grade is 0-4, and a risk grade of 0 indicates no bleeding risk. However, depending on different applications, the possible risk grades are not limited to these 5 grades. For example, it can be a three-level classification, or even more levels of classification, and each level can be further subdivided, etc., which can be adjusted according to the specific application

[0112] Its technical implementation also includes a classification algorithm for cumulative classification probability of multiple frames. This algorithm accumulates and fuses the category probabilities of the prediction results of all frames in the video clip sequence according to the category preset weights {W 0 , W 1 , W 2 , W 3 , W 4}, where w 0 represents the weight for grade 0, and the magnitude of the preset weight is proportional to the risk grade

[0113] The technical solution of the present invention will be further described in detail below through several embodiments in conjunction with the accompanying drawings. However, the selected embodiments are only used to illustrate the present invention and do not limit the scope of the present invention

[0114] Embodiment 1

[0115] In this embodiment, in response to the problem of predicting the bleeding grade of the video clip before biopsy, a complete prediction process based on an object detection model and a dynamic multi-way classification model is designed and implemented. This process mainly includes the following 5 steps

[0116] 1. Access the real-time bronchoscope video stream, and use a single-frame bronchoscope image x t as the input. Using the target detection model D(·) for the lesion to be biopsied, perform lesion target detection L t = D(x t) In a single-frame image, 0 or several lesion regions can be detected, where n represents the number of detected lesion regions, and the location information of lesion region i is l i =(x i , y i , w i , h i , p i );

[0117] 2. This step defines the range of video segments to be predicted in a real-time video stream, that is, the start time and end time of the video segment. To obtain the video segment to be predicted, this step requires manual input of the start signal of the video segment to be predicted through the combination keys of the keyboard. Typing control + s represents the start signal of the predicted segment, and control + e represents the end signal of the predicted segment. This step obtains the set of video frames to be predicted Frames = {x 1 ,..., x m}, where x defines the image to be predicted, and m represents the length of this segment, that is, the total number of frames. The lesion regions L t corresponding to each frame image in step a are intercepted to obtain the local image set I t = {l 1 ,..., l n}, and the corresponding confidence set P t = {p 1 ,..., p n};

[0118] 3. This step will filter out the local lesion regions with a relatively small area ratio intercepted in step 2. This step is only executed for each frame image between the start and end time points described in step 2. The ratio of the local image to the whole image is Set the ratio threshold of the local image to the whole image as If l i is retained in I t , otherwise filter out l i and p t in the confidence set P i . As shown in step three of Figure 1 , after this step, the optimized local lesion region set and the confidence set

[0119] 4. This step uses the single-frame image x t in the prediction segment represented in step 2 and the optimized local lesion image set and the confidence set As the input, where the single-frame image is x t As the global feature extraction backbone network F 1 The input of (·), and As the local feature extraction backbone network F 2 The input of (·), and finally obtain the probability distribution of each level of the bleeding level of the frame image This step also includes an iterative input method commonly used in the field. This iterative method enables F 2 (@) to dynamically handle a variable number of local pictures;

[0120] 5. This step accumulates the weights of the bleeding level probabilities of each frame in the video clip to be detected obtained in step 4 It is planned to set the weight of each category as W = {w 0 , w 1 , w 2 , w 3 , w 4}, where the higher-level weights are set higher, and the total weight accumulates to 1 = w 0 + w 1 + w 2 + w 3 + w 4 , w 4 = 5 * w 0 , w 3 = 4 * w 0 , w 2 = 3 * w 0 , w 1 = 2 * w 0 . When using the weighted cumulative probability, select the weight with the highest probability in the current frame for weighted accumulation. Finally, obtain the probability P of the bleeding level of the video clip lesion biopsy = (p 0 , p 1 , p 2 , p 3 , p 4 ). Select the category with the highest probability in the cumulative probability distribution as the final bleeding risk level output.

[0121] In the above steps, for the problem of single-frame bleeding level prediction, a dynamic multi-channel deep learning network model is designed and implemented. This model can process multiple picture inputs with variable numbers, fuse the features of all input pictures, and obtain classification probabilities using the fused feature vectors. As Figure 2 shown, the model has two inputs. One of the inputs is the original image x t . After passing through the global image feature extraction network, the global feature vector is obtained The other input is multiple local images corresponding to the original image and the confidence of the local image Since the local feature extraction network needs to handle a variable number of local images, the present invention particularly designs to extract features from each local image in the in an iterative manner to obtain a local feature set After weighted averaging of these features, a local fusion feature vector is obtained When it is empty, it is a zero vector; the global vector and the local feature vector are fused using the average value method to obtain the final feature vector of the single-frame image After passing through the fully connected layer and Softmax, the class probability distribution of the current frame is obtained The specific structure of this model is as shown in Figure 6 When classifying the bleeding level, the information of the global feature vector and several local feature vectors of the lesion regions is fused. Among them, the local feature vectors are used to capture the details of the lesion regions, while the global feature vector provides the overall background comparison, enabling the model to more accurately predict the level. This model is applied in step 4

[0122] To address the problem of obtaining the bleeding level prediction in the prediction segment, a multi-frame dynamic weight cumulative fusion algorithm is designed and implemented. This algorithm is applied in step 5 of claim (1) and includes the following three steps

[0123] Using the following conditional equation, the weight of each class W = {w 0 , w 1 , w 2 , w 3 , w 4} is preset to satisfy the following conditions

[0124] 1 = w 0 + w 1 + w 2 + w 3 + w 4 ,

[0125] w 4 = 5 * w 0 ,

[0126] w 3 = 4 * w 0 ,

[0127] w 2 = 3 * w 0 ,

[0128] w 1 = 2 * w 0

[0129] The probability distribution of each frame needs to be weighted with the weight of the corresponding highest-probability category:

[0130]

[0131] Here calculate the highest-probability category l corresponding to this frame, and use this category as the index for selecting the weight to obtain one of the preset weights w l weight pt to obtain the weighted probability distribution as The set of weighted vectors for all frames within the segment is

[0132] Sum each category in the set to obtain the predicted category distribution of the bleeding grade in the biopsy within the segment to be predicted:

[0133]

[0134] Here P=(p 0 , p 1 , p 2 , p 3 , p 4 ), representing the classification probability of the bleeding grade corresponding to this segment;

[0135] Take the category corresponding to the highest probability as the final predicted grade:

[0136]

[0137] The present invention has conducted a certain number of application tests based on the biopsy bleeding grade prediction method provided in Embodiment 1. During bronchoscopic examination and biopsy operations, the bleeding risk grades prompted by the system show extremely strong consistency with the subjective judgments of experienced doctors, demonstrating excellent auxiliary functions. Compared with existing bleeding risk prediction models, it shows a higher prediction ability, greatly reducing the workload of doctors.

[0138] It should be understood that the above embodiments are only used to illustrate the technical concept and features of the present invention, and their purpose is to enable those familiar with this technology to understand the content of the present invention and implement it accordingly, and should not be used to limit the protection scope of the present invention. Any equivalent changes or modifications made according to the spirit and essence of the present invention should be covered within the protection scope of the present invention.

Claims

1. A method for predicting bronchoscopic biopsy bleeding grade based on a dynamic multi-channel deep learning model, characterized in that: include: Acquire a video clip, the video clip comprising a plurality of bronchial images ordered in time, and identify a local lesion area in the bronchial images; A dynamic multi-channel deep learning model is used to extract features from the bronchial image, wherein the dynamic multi-channel deep learning model includes a global branch and a local branch, wherein the global branch identifies the global features of the bronchial image, and the local branch identifies the local features of the local lesion area; The global feature and the local feature are fused to obtain a fused feature, and the bleeding risk level is classified based on the fused feature to obtain a risk level probability distribution; Dynamically weighting each probability value in the risk level probability distribution to obtain a weighted probability distribution, wherein the weight coefficient of the dynamic weighting is positively correlated with the risk level corresponding to the highest probability level in the risk level probability distribution; For the video clip, accumulating the weighted probability value corresponding to each risk level in the weighted probability distribution to obtain a fused probability distribution; Based on the probability value of each risk level of the fused probability distribution, the bleeding risk prediction level corresponding to the video clip is screened and obtained.

2. The method for predicting bronchoscopic biopsy bleeding grade according to claim 1, characterized in that: The process of obtaining the video clip specifically includes: Get the video stream; A time segment is specified in the video stream, and images in the time segment are captured as the video segment.

3. The method for predicting bronchoscopic biopsy bleeding grade according to claim 1, characterized in that: The process of obtaining the local lesion area specifically includes: Performing image recognition on the bronchial image to obtain an initial local area; The initial local area is subjected to size screening to obtain the local lesion area, wherein the size screening is performed based on the pixel size of the initial local area, and the initial local area whose pixel size is lower than a threshold is removed.

4. The method for predicting bronchoscopic biopsy bleeding grade according to claim 3, characterized in that: The acquisition process of the local lesion area is expressed as: Calculate the area ratio of the initial local area to the bronchial image Among them, area(·) represents the pixel area, l i represents the initial local area, l i =(x i ,y i , w i ,h i , p i ), x i ,y i represents the pixel center coordinates of the initial local area, w i ,h i represents the pixel width and pixel length of the initial local area, p i represents the confidence that the initial local area is classified as a lesion, x t represents the bronchial image, i represents the index number of the initial local area, and t represents the index number of the bronchial image; Set the threshold of the proportion of the local image to the whole image like Initial local area l u The set I retained in the local lesion area t Otherwise, filter and remove the initial local area l i And its confidence set P t The confidence value p in i .

5. The method for predicting bronchoscopic biopsy bleeding grade according to claim 1, characterized in that: The feature extraction process is expressed as: in, represents the global feature, F1(·) represents the global branch, represents the local feature, represents the set of local lesion areas, represents the corresponding lesion confidence set, F2(·) represents the local branch, represents the preliminary fusion of the local branches, f represents the fusion feature, F(·) represents the feature decoder, and f j represents the characteristic value corresponding to each risk level in the local feature, n represents the number of the local lesion area, p t represents the risk level probability distribution, represents the probability value corresponding to each risk level in the risk level probability distribution, and o represents the highest level number of the risk level.

6. The method for predicting bronchoscopic biopsy bleeding grade according to claim 1, characterized in that: The dynamic weighting process is expressed as: 1=in o +w1+w2+……+w o , In o =α o *w0, w3=α3*w0, w2=α2*w0, w1=α1*w0 Wherein, w. represents the weight coefficient, α. represents the dynamic coefficient and its value varies with the subscript value. o increases with the increase of , W represents the set of weight coefficients, represents the weighted probability distribution, represents the probability value corresponding to each risk level in the weighted probability distribution, k=0, 1, 2…o, represents the index of the risk level, represents the set of weighted probability distributions, P represents the fusion probability distribution, p k represents the probability value corresponding to each risk level in the fused probability distribution, l represents the index of the risk level corresponding to the maximum probability value, and m represents the number of bronchial images in the video clip.

7. The method for predicting bronchoscopic biopsy bleeding grade according to claim 1, characterized in that: The screening process of the bleeding risk prediction level is expressed as: Wherein, Level represents the predicted level of bleeding risk.

8. The method for predicting bronchoscopic biopsy bleeding grade according to claim 1, characterized in that: The global branch and the local branch are located in the backbone network in the dynamic multi-path deep learning model, and the architecture of the backbone network adopts any one of Resnet, EfficientNet, ConvNext, and Swin Transformer.

9. A bronchoscopic biopsy bleeding grade prediction system based on a dynamic multi-channel deep learning model, characterized in that: include: A data acquisition module, configured to acquire a video clip, the video clip comprising a plurality of bronchial images ordered in time, and identify a local lesion area in the bronchial image; A feature extraction module, used to extract features from the bronchial image using a dynamic multi-channel deep learning model, wherein the dynamic multi-channel deep learning model includes a global branch and a local branch, wherein the global branch identifies the global features of the bronchial image, and the local branch identifies the local features of the local lesion area; A feature fusion module, used for fusing the global features and local features to obtain fused features, and classifying the bleeding risk level based on the fused features to obtain a risk level probability distribution; A feature weighting module, used for dynamically weighting each probability value in the risk level probability distribution to obtain a weighted probability distribution, wherein the weight coefficient of the dynamic weighting is positively correlated with the risk level corresponding to the highest probability level in the risk level probability distribution; A probability fusion module, configured to accumulate, for the video clip, the weighted probability value corresponding to each risk level in the weighted probability distribution to obtain a fused probability distribution; The level prediction module is used to screen and obtain the bleeding risk prediction level corresponding to the video clip based on the probability value of each risk level of the fused probability distribution.

10. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed, the steps of the bronchoscopic biopsy bleeding grade prediction method according to any one of claims 1 to 8 are executed.

Images