Bronchoscopy biopsy bleeding grade prediction method based on dynamic multi-path deep learning model

Abstract

The application provides a bronchoscopy biopsy bleeding grade prediction method based on a dynamic multi-path deep learning model. The method comprises the following steps: acquiring a video segment; performing feature extraction to identify global features and local features; performing fusion to obtain fused features and performing bleeding risk grade classification; performing dynamic weighting; accumulating weighted probability distribution; and screening a bleeding risk prediction grade. The method provided by the application is a biopsy pre-bleeding grade prediction method based on a dynamic multi-path model. Through the combination of multiple types of deep learning models, the method realizes process-based video segment analysis. Through the combination of deep learning models and multi-frame fusion analysis, the accuracy and reliability of the pre-biopsy bleeding risk prediction are significantly improved, which provides strong support for clinical decision-making and has important application value.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video image recognition, and particularly relates to a bronchoscopy biopsy bleeding grade prediction method based on a dynamic multi-path deep learning model. BACKGROUND

[0002] Endoscopy is an important branch of medicine, and bronchoscopy is an important medical technology, for example, by inserting a bronchoscope into the airway of a patient, the inside of the trachea and bronchi can be directly observed. It is mainly used for the diagnosis and treatment of respiratory system diseases such as lung cancer, tuberculosis, chronic obstructive pulmonary disease, etc. Bronchoscopy not only helps doctors to determine the lesion site, but also can perform biopsy, remove foreign bodies, hemostasis and other treatment operations.

[0003] Bronchoscopy biopsy is a process of obtaining lesion tissue samples through a bronchoscope, and is widely used in the diagnosis of lung and airway diseases such as tumors, interstitial lung disease, granulomatous disease and some infectious diseases. Biopsy can help determine the nature of the lesion and provide important basis for clinical diagnosis and treatment. Although bronchoscopy biopsy is a common and important diagnostic method, there are still certain risks in the operation process. Common complications include bleeding, pneumothorax, airway spasm and hypoxemia, etc. In particular, for patients with bleeding tendency or cardiopulmonary insufficiency, the risk of biopsy operation will significantly increase. Bleeding during bronchoscopy biopsy is one of the most common complications. During the biopsy process, experienced bronchoscopists need to record and evaluate the bleeding grade, and under normal circumstances, the bleeding risk can be divided into the following grades according to existing clinical research:

[0004] Grade 0: No or minimal bleeding;

[0005] Grade 1: Small amount of bleeding, which can be self-stopped by negative pressure suction without the need for other hemostatic measures;

[0006] Grade 2: Moderate bleeding, which needs to be treated with adrenalin dilution, hemocoagulase infusion or pre-cooled saline perfusion, and endoscopic treatment can continue;

[0007] Grade 3: Large amount of bleeding, which needs to be intervened by endoscopic use of thrombin lyophilized powder, balloon compression, intravenous use of pituitrin, intubation or insertion of a rigid scope, etc., and endoscopic treatment is forced to stop;

[0008] Grade 4: Large amount of bleeding or hemodynamic disorder, which needs blood transfusion, emergency treatment such as surgery and vascular intervention. The assessment of bleeding risk needs to consider the patient's medical history, hematology test results and imaging test results. For high-risk patients, preoperative risk assessment should be carried out in detail, and appropriate preventive measures should be taken during the operation, such as local application of hemostatic drugs and mechanical compression hemostasis, etc.

[0009] Deep learning models are increasingly being used in bronchoscopy, mainly for image recognition and diagnostic assistance. Through deep learning algorithms such as convolutional neural networks (CNN), the accuracy of bronchoscopic image recognition can be improved, assisting doctors in diagnosing and classifying lesions. Studies have shown that deep learning models have high accuracy and sensitivity in distinguishing between benign and malignant lesions.

[0010] Using deep learning models to process bronchoscopic images before biopsy can predict the risk of bleeding during the biopsy process. Through training on a large number of bronchoscopic images, deep learning models can identify potential high-risk areas and provide bleeding risk assessment. This method not only improves the safety of biopsy operations, but also helps doctors develop more precise treatment plans.

[0011] However, current existing technology can only provide preliminary bleeding risk warnings through simple image recognition, and cannot comprehensively consider spatial scales (including overall and local images) and time scales to provide more accurate and reliable bleeding risk warnings, which is not significantly helpful in assisting doctors to improve their bronchoscopic operation level. SUMMARY

[0012] To address the shortcomings of the prior art, the present invention aims to provide a bronchoscopic biopsy bleeding grade prediction method based on a dynamic multi-path deep learning model.

[0013] To achieve the aforementioned invention purposes, the technical solutions adopted by the present invention include:

[0014] In a first aspect, the present invention provides a bronchoscopic biopsy bleeding grade prediction method based on a dynamic multi-path deep learning model, which includes:

[0015] Obtaining a video segment, which includes multiple bronchoscopic images sorted by time, and identifying local lesion areas in the bronchoscopic images;

[0016] Using a dynamic multi-path deep learning model to extract features from the bronchoscopic images, the dynamic multi-path deep learning model includes a global branch and a local branch, the global branch identifies global features of the bronchoscopic images, and the local branch identifies local features of the local lesion areas;

[0017] Fusing the global features and local features to obtain fused features, and classifying bleeding risk grades based on the fused features to obtain a risk grade probability distribution;

[0018] Dynamically weighting each probability value in the risk grade probability distribution to obtain a weighted probability distribution, the weight coefficient of the dynamic weighting has a positive correlation with the risk level corresponding to the highest probability grade in the risk grade probability distribution;

[0019] For the video segment, the weighted probability values corresponding to each risk level in the weighted probability distribution are accumulated to obtain a fusion probability distribution;

[0020] Based on the probability value size of each risk level of the fusion probability distribution, the bleeding risk prediction level corresponding to the video segment is obtained through screening.

[0021] In a second aspect, the present application also provides a bronchoscopy biopsy bleeding grade prediction system based on a dynamic multi-path deep learning model, which comprises:

[0022] A data acquisition module is configured to acquire a video segment, wherein the video segment comprises a plurality of bronchus images sorted by time, and identify a local lesion area in the bronchus image;

[0023] A feature extraction module is configured to extract features of the bronchus image by using a dynamic multi-path deep learning model, wherein the dynamic multi-path deep learning model comprises a global branch and a local branch, the global branch is configured to identify global features of the bronchus image, and the local branch is configured to identify local features of the local lesion area;

[0024] A feature fusion module is configured to fuse the global features and the local features to obtain fusion features, and perform bleeding risk level classification based on the fusion features to obtain a risk level probability distribution;

[0025] A feature weighting module is configured to dynamically weight each probability value in the risk level probability distribution to obtain a weighted probability distribution, wherein the weight coefficient of the dynamic weighting is positively correlated with the risk level corresponding to the highest probability in the risk level probability distribution;

[0026] A probability fusion module is configured to, for the video segment, accumulate the weighted probability values corresponding to each risk level in the weighted probability distribution to obtain a fusion probability distribution;

[0027] A grade prediction module is configured to, based on the probability value size of each risk level of the fusion probability distribution, obtain the bleeding risk prediction level corresponding to the video segment through screening.

[0028] In a third aspect, the present application also provides a readable storage medium, wherein the readable storage medium stores a computer program, and the computer program is configured to execute the steps of the above-mentioned bronchoscopy biopsy bleeding grade prediction method when running.

[0029] Compared with the prior art, the biopsy pre-bleeding grade prediction method based on the dynamic multi-path model of the present application realizes the video segment analysis in a streamlined manner through the combination of multiple types of deep learning models, and has the following advantages:

[0030] Firstly, by constructing a bleeding grade prediction process based on bronchoscope video clips and combining different types and functions of deep learning network models, the bleeding risk can be more accurately predicted. This multi-model combination method can fully utilize the advantages of each model to improve the overall prediction accuracy.

[0031] Secondly, a bronchoscope lesion detection model based on a deep learning target detection algorithm can extract local lesions with potential biopsy significance from bronchoscope images and obtain the confidence of the lesion area, which helps doctors accurately locate the lesion area before biopsy and reduces the risk of misoperation. In addition, a deep learning image classification model with two parallel backbone networks integrates the whole image information and local image information, one backbone network is used to extract the whole image feature vector, and the other backbone network is used to extract the image features of the local area and fuse them, and the feature vectors generated by the two backbone models are fused again as the prediction feature vector of a single image. This method can more comprehensively analyze image information and improve the reliability of prediction.

[0032] Furthermore, a classification algorithm that accumulates the classification probabilities of multiple frames is used to fuse the prediction results of multiple frames in the video clip sequence, and a preset bleeding risk level category weight is used to obtain a more accurate and reasonable prediction result, effectively utilizing the global and local information of each frame in the video segment to improve the stability and accuracy of prediction.

[0033] Finally, by accurately predicting the bleeding risk, doctors can take appropriate preventive measures before biopsy to reduce the risk of bleeding during biopsy and improve patient safety.

[0034] In summary, the method provided by the present application significantly improves the accuracy and reliability of bleeding risk prediction before biopsy by combining and multi-frame fusion analysis of deep learning models, providing strong support for clinical decision-making and having important application value.

[0035] The above description is only a summary of the technical solutions of the present application. In order to enable those skilled in the art to more clearly understand the technical means of the present application and can be implemented according to the content of the description, the following is a preferred embodiment of the present application with detailed description of the drawings as follows. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 is a process schematic diagram of the first step of the bronchoscope biopsy bleeding grade prediction method provided by a typical embodiment of the present application;

[0037] Figure 2 is a process schematic diagram of the second step of the bronchoscope biopsy bleeding grade prediction method provided by a typical embodiment of the present application;

[0038] Figure 3 Figure 4 is a process schematic diagram of the third step of the bronchoscopy biopsy bleeding grade prediction method provided by an exemplary embodiment of the present application;

[0039] Figure 4 Figure 5 is a process schematic diagram of the fourth step of the bronchoscopy biopsy bleeding grade prediction method provided by an exemplary embodiment of the present application;

[0040] Figure 5 Figure 6 is a process schematic diagram of the fifth step of the bronchoscopy biopsy bleeding grade prediction method provided by an exemplary embodiment of the present application;

[0041] Figure 6 Figure 7 is a structural schematic diagram of the dynamic multi-path deep learning model provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0042] In view of the deficiencies in the prior art, the present inventors have long studied and practiced to come up with the technical solution of the present application. The technical solution, its implementation process and principles will be further explained as follows.

[0043] In the following description, many specific details are set forth in order to provide a thorough understanding of the present application, however, the present application can also be implemented in other ways different from those described herein, and therefore, the scope of protection of the present application is not limited by the specific embodiments disclosed below.

[0044] Moreover, relational terms such as "first" and "second" and the like are used only to distinguish one from another of a similar entity or method step, and do not necessarily require or imply any actual relationship or order between or among such entities or method steps.

[0045] The main purpose of the present application is to design and implement a dynamic multi-path bleeding grade classification model using pre-biopsy lesion local pictures and whole pictures, and to use parallel analysis methods for bleeding grade prediction. In the bleeding risk assessment, the present application uses several bleeding grade classification models of local pictures and whole pictures before operation for parallel analysis, and statistically predicts the bleeding risk grade probability within a certain period of time. Through dynamic weight probability integration, the accuracy of bleeding risk prediction can be improved. Through comprehensive analysis of the local area of the lesion and the whole image sequence, the deep learning model can more comprehensively evaluate the bleeding risk grade, and provide strong support for clinical decision-making. This method not only can reduce the bleeding risk in biopsy operation, but also can improve the accuracy and reliability of diagnosis.

[0046] Based on the above purposes and technical concepts, the present application provides a bronchoscopy biopsy bleeding grade prediction method based on a dynamic multi-path deep learning model, which includes the following steps:

[0047] obtaining a video clip, the video clip comprising a plurality of bronchoscopic images sorted in time, and identifying a local lesion region in the bronchoscopic images;

[0048] extracting features of the bronchoscopic images using a dynamic multi-path deep learning model, the dynamic multi-path deep learning model comprising a global branch and a local branch, the global branch identifying global features of the bronchoscopic images, and the local branch identifying local features of the local lesion region;

[0049] fusing the global features and the local features to obtain fused features, and classifying the risk grade based on the fused features to obtain a risk grade probability distribution;

[0050] dynamically weighting each probability value in the risk grade probability distribution to obtain a weighted probability distribution, the weight coefficient of the dynamic weighting being positively correlated with the risk level corresponding to the highest probability level in the risk grade probability distribution;

[0051] for the video clip, accumulating the weighted probability value corresponding to each risk grade in the weighted probability distribution to obtain a fused probability distribution;

[0052] based on the probability value of each risk grade in the fused probability distribution, screening to obtain a bleeding risk prediction grade corresponding to the video clip.

[0053] In order to improve the practical application performance of the deep learning model in the bleeding grade prediction of the bronchoscopic biopsy picture, a classification method based on a dynamic multi-path deep learning model is invented. The application is applied to the bleeding grade analysis of the actual bronchoscopic video clip before biopsy. Compared with single model prediction and single frame image analysis, the rationality and accuracy of the bleeding grade prediction can be greatly improved.

[0054] As some typical application examples, the biopsy bleeding grade prediction method based on the dynamic multi-path model provided by the embodiment of the present application is a process video segment analysis method based on a combination of multiple types of deep learning models. The technical implementation includes constructing a bleeding grade prediction process based on the video segment before bronchoscopy biopsy, and different types and different functions of deep learning network models relied on by each step in the process. The technical implementation includes using a bronchoscopic lesion detection model based on a deep learning object detection algorithm. The object detection model uses a current general neural network image detection model (such as YOLO, SSD, Faster-Rcnn, FCOS, etc.). The model extracts several lesion regions with potential biopsy significance from the bronchoscopic image and obtains the confidence of the lesion region. The technical implementation also includes a deep learning image classification model with a two-way parallel backbone network. The backbone model of the classification model uses a current general neural network image classification model (such as Resnet, EfficientNet, ConvNext, Swin Transformer, etc.). The purpose of constructing the classification model is to effectively fuse the whole image information and the local information of the image. One of the backbone networks is used for extracting the feature vector of the whole image, and the other parallel backbone network is used for extracting and fusing the image features of several local regions. The feature vectors generated by the two backbone models are fused again as the prediction feature vector of the current single frame image, and the probability of each bleeding category predicted by the frame image is generated.

[0055] The technical implementation also includes a multi-frame classification probability accumulation classification algorithm. The algorithm fuses the prediction results of multiple frames in the video segment sequence, and uses a preset bleeding risk level category weight to obtain a more accurate and reasonable prediction result.

[0056] In some embodiments, the video segment acquisition process specifically includes the following process:

[0057] Acquire a video stream;

[0058] Specify a time segment in the video stream, and cut the image in the time segment as the video segment.

[0059] In some embodiments, the local lesion region acquisition process specifically can include:

[0060] Perform image recognition on the bronchoscopic image to obtain an initial local region;

[0061] Perform size screening on the initial local region to obtain the local lesion region. The size screening is based on the pixel size of the initial local region, and removes the initial local region with a pixel size below a threshold.

[0062] In some embodiments, the process of obtaining the local lesion regions can be represented as:

[0063] calculating the area ratio of the initial local region to the bronchus image wherein area(·) represents the pixel area, l i represents the initial local region l i = (x i , y i , w i , h i , p i , x i , y i represents the pixel center coordinates of the initial local region, w i , h i represents the pixel width and pixel length of the initial local region, p i represents the confidence of the initial local region being classified as a lesion, x t represents the bronchus image, i represents the index number of the initial local region, and t represents the index number of the bronchus image.

[0064] Setting the proportion threshold of the local image to the whole image as

[0065] If the initial local region l i is retained in the set of local lesion regions I t , otherwise, the initial local region l i and its confidence value p t in the lesion confidence set P i are filtered out.

[0066] In some embodiments, the process of feature extraction can be represented as:

[0067]

[0068]

[0069] wherein, represents the global feature, F1(·) represents the global branch, represents the local feature, represents the set of local lesion regions, represents the corresponding lesion confidence set, F2(·) represents the local branch, represents the preliminary fusion of the local branch, f represents the fusion feature, F(·) represents the feature decoder, and f jwherein, n represents the number of the local lesion regions, p represents the feature value corresponding to each risk level in the local feature, and t wherein, p represents the probability value corresponding to each risk level in the risk level probability distribution, and o represents the highest level number of the risk level. wherein, p represents the probability value corresponding to each risk level in the risk level probability distribution, and o represents the highest level number of the risk level.

[0070] In some embodiments, the process of the dynamic weighting is represented as:

[0071] 1 = w0+ w1+ w2+ … + w o ,

[0072] w o = a o * w0,

[0073] …

[0074] w3= a3* w0,

[0075] w2= a2* w0,

[0076] w1= a1* w0

[0077]

[0078] wherein, w represents the weight coefficient, a represents the dynamic coefficient and takes a value increasing with the increase of the subscript value o, and W represents the set of the weight coefficient. · wherein, p represents the probability value corresponding to each risk level in the weighted probability distribution, and k = 0, 1, 2, … o represents the index of the risk level. · wherein, P represents the set of the weighted probability distribution, and P represents the fusion probability distribution. wherein, p represents the probability value corresponding to each risk level in the weighted probability distribution, and k = 0, 1, 2, … o represents the index of the risk level. wherein, P represents the set of the weighted probability distribution, and P represents the fusion probability distribution. k wherein, p represents the probability value corresponding to each risk level in the fusion probability distribution, l represents the index of the risk level corresponding to the maximum probability value, t represents the index of the bronchus image, and m represents the number of the bronchus images in the video segment. In some embodiments, the screening process of the bleeding risk prediction level is represented as:

[0079]

[0080] wherein, Level represents the bleeding risk prediction level.

[0081]

[0082] ​In some embodiments, the global branch and the local branch are located in a backbone network in the dynamic multi-path deep learning model, and an architecture of the backbone network adopts any one of Resnet, EfficientNet, ConvNext, and SwinTransformer.

[0083] As a typical application example of the above technical solution, in view of the low accuracy and rationality of bronchial bleeding risk prediction of a single model on a single frame image, the following three main innovations are proposed in the present application:

[0084] Firstly, the present application designs and implements a complete prediction process based on a target detection model and a dynamic multi-path classification model for predicting the bleeding grade according to the video clip before biopsy. The process mainly includes the following steps:

[0085] Step 1. Access the real-time bronchoscopy video stream, use the lesion target detection model to detect the lesion target in each frame of the video stream, and record the coordinate position and confidence of each local lesion area in each frame of the image in real time. The detected target in the image can be empty.

[0086] Step 2. Obtain the video clip to be predicted. In actual operation, for example, this step needs to input the starting signal of the video clip to be predicted through the combination of the keyboard. The input of control+s represents the starting signal of the prediction clip, and the input of control+e represents the ending signal of the prediction clip. Of course, the specific operation of cutting the clip is not limited to the specific method described here.

[0087] Step 3. Filter and optimize the local lesion area set. This step is only performed on each frame of image between the starting and ending time points described in step 2. Each frame of image in the prediction video clip can be detected by the lesion target detector to have 0 or more local lesion areas with biopsy significance. In this step, a minimum local target proportion is preset to filter out local lesion areas with small proportions. Because the local image with small proportion has low bleeding risk details, it will affect the final prediction effect.

[0088] Step 4. Dynamic multi-path classification model and iterative input algorithm for single frame bleeding grade prediction. This step is only performed on each frame of image between the starting and ending time points described in step 2. The model takes each frame of image and the local lesion area filtered out in step 3 as input to generate the probability of each bleeding category predicted by the frame of image.

[0089] Step 5. Multi-frame dynamic weight probability accumulation fusion, which is only performed at the end time point described in step 2. In this step, the class probability predicted by each frame image generated in step 4 is accumulated by weight, and the weight is a preset value. The weight of a higher bleeding grade is larger. Here, a larger weight is set considering that a higher bleeding grade brings a greater bleeding risk. After accumulation, the class with the maximum probability in the accumulated probability is taken as the final bleeding risk grade output.

[0090] Secondly, the present application aims at the problem of single-frame bleeding grade prediction, and designs and implements a dynamic multi-path deep learning network model. The purpose of the model is to predict the bleeding grade by using a single frame image and a plurality of corresponding local images in the image. The model can effectively integrate the global and local information of the image. The model can process different numbers of local images through a corresponding iterative algorithm. Compared with the original classification model, the model has higher classification accuracy. The model is applied to step 4 of the innovative point (1).

[0091] Thirdly, the present application aims at the problem of obtaining bleeding grade prediction in a video segment, and designs and implements a multi-frame dynamic weight accumulation fusion algorithm. The algorithm is applied to step 5 of the innovative point (1). The algorithm includes the following three steps:

[0092] Setting a weight for the class probability corresponding to each frame;

[0093] Accumulating the probabilities after setting the weight within the class;

[0094] Taking the class with the highest probability as the final prediction grade.

[0095] The second aspect of the embodiment of the present application also provides a bronchoscopy biopsy bleeding grade prediction system based on a dynamic multi-path deep learning model, which comprises:

[0096] A data acquisition module is configured to acquire a video segment, wherein the video segment comprises a plurality of bronchus images sorted by time, and identify a local lesion area in the bronchus image;

[0097] A feature extraction module is configured to extract features of the bronchus image by using a dynamic multi-path deep learning model, wherein the dynamic multi-path deep learning model comprises a global branch and a local branch. The global branch identifies global features of the bronchus image, and the local branch identifies local features of the local lesion area.

[0098] A feature fusion module is configured to fuse the global features and the local features to obtain fused features, and perform bleeding risk grade classification based on the fused features to obtain a risk grade probability distribution.

[0099] The feature weighting module is configured to dynamically weight each probability value in the risk level probability distribution to obtain a weighted probability distribution, wherein a weight coefficient of the dynamic weighting is in positive correlation with a risk level corresponding to a highest probability in the risk level probability distribution.

[0100] The probability fusion module is configured to accumulate a weighted probability value corresponding to each risk level in the weighted probability distribution to obtain a fusion probability distribution for the video segment.

[0101] The level prediction module is configured to filter a bleeding risk prediction level corresponding to the video segment based on a probability value of each risk level in the fusion probability distribution.

[0102] The third aspect of the embodiment of the present application also provides a readable storage medium, wherein the readable storage medium stores a computer program, and the computer program is used to execute the steps of the bronchoscopic biopsy bleeding level prediction method provided by any of the above embodiments.

[0103] As some typical examples of the above-mentioned overall technical scheme, the bronchoscopic biopsy pre-bleeding level prediction method based on the dynamic multi-path deep learning model provided by the present application mainly includes five steps. Figures 1-5 As shown in the following table:

[0104] Step 1: Referring to Figure 1 Each frame of picture is obtained from real-time bronchoscopy, and a lesion detector is used to detect a lesion area with biopsy significance in each frame;

[0105] Step 2: Referring to Figure 2 An artificial input start and end signal is used to determine a starting point of the video segment to be graded in the video stream;

[0106] Step 3: Referring to Figure 3 The local lesion area set is optimized, and the local lesion area with a low image area ratio is filtered;

[0107] Step 4: Referring to Figure 4 Each frame of picture in the video segment to be graded is input into the dynamic multi-path deep model for classification;

[0108] Step 5: Referring to Figure 5 The classification sequence is accumulated by weight to obtain a final segment grading prediction.

[0109] To improve the effectiveness of deep learning models in real-time prediction of biopsy bleeding levels during bronchoscopy, this embodiment employs an automated prediction process based on a multi-neural network model and a classification method using a dynamic multi-path deep learning model to predict bleeding levels in pre-biopsy bronchoscopy video clips. The method described in this invention, when applied to bronchoscopy bleeding level prediction, can more accurately and reasonably predict the bleeding risk at the biopsy site during bronchoscopy compared to single-frame classification models.

[0110] This method, based on a dynamic multipath model for predicting the bleeding level before bronchoscopy biopsy, is a streamlined video segment analysis method that combines multiple types of deep learning models. Its technical implementation includes constructing a bleeding level prediction process for pre-bronchoscopy biopsy video segments, including a target detection model for detecting lesion locations of biopsy significance, a dynamic multipath deep learning classification model for single-frame risk level prediction, and a weighted cumulative algorithm that fuses the level prediction probability sequences of each frame within the video segment. This process effectively utilizes global and local information and sequence information from each image in the pre-biopsy video segment to predict the risk of bleeding during lesion biopsy. Its technical implementation includes employing a bronchoscopic lesion detection model D(·) based on a deep learning target detection algorithm. This target detection model uses currently popular neural network image detection models (such as YOLO, SSD, Faster-Rcnn, FCOS, etc.), and can process single-frame bronchoscopy images x. t Where t represents the frame number, and l represents the location of the lesion region in the predicted image that has potential biopsy significance. i =(x i ,y i ,w i ,h i ,p i ), where x i ,y i w represents the pixel center coordinates of the detected target in the image. i ,h i p represents the pixel width and pixel length of the detected target in the image. i The confidence level represents the area of ​​the lesion.

[0111] The technical implementation also includes a two-way parallel backbone deep learning image classification model F(·) with dynamic input, the backbone model of the classification model adopts a commonly used neural network image classification model (such as Resnet, EfficientNet, ConvNext, Swin Transformer, etc.), and the purpose of constructing the classification model is to integrate the whole image information and the local information of the image. The classification model includes two parallel backbone networks F1(·) and F2(·), wherein F1(·) is used to extract the feature vector of the whole image F2(·) is used to extract the feature vector of the detected local lesion area in the image n is the number of detected local lesions, may be an empty set, and each feature vector is accumulated and fused into a feature vector according to a local lesion confidence weight After accumulation, the global feature vector is fused to obtain a global-local feature fusion vector f as a final grade classification feature vector, and a probability value of each class is obtained wherein represents the probability that the t-th frame is predicted to be of grade k (k = 0, 1, 2, 3, 4), of course, in the embodiment, the risk grade is 0-4, and 0 grade indicates no bleeding risk. However, according to different applications, the possible risk grades are not limited to these five grades, for example, it can be a three-grade classification, or more, and each grade can also be subdivided, etc. The application can be adjusted according to the specific application.

[0112] The technical implementation also includes a classification algorithm for multi-frame classification probability accumulation. The algorithm accumulates and fuses the class probabilities of all frames in the video segment sequence The prediction result is accumulated and fused according to the preset weight {w0, w1, w2, w3, w4} of the class, wherein w0 represents the weight of grade 0, and the size of the preset weight is proportional to the risk grade.

[0113] The technical solutions of the present application will be further described in detail by means of several embodiments in combination with the accompanying drawings. However, the selected embodiments are only used to illustrate the present application, and do not limit the scope of the present application.

[0114] Embodiment 1

[0115] According to the problem of bleeding grade prediction for pre-biopsy video segments, a complete prediction process based on a target detection model and a dynamic multi-path classification model is designed and implemented, which mainly includes the following five steps:

[0116] 1. Access real-time bronchoscopy video stream, single-frame bronchoscopy image x tAs input, the lesion detection model D(·) is used to detect lesions in each frame of the video stream. t =D(x) t In a single frame image, zero or several lesion areas can be detected, among which... n represents the number of detected lesion regions, and the location information of lesion region i is l. i =(x i ,y i ,w i ,h i ,p i ;

[0117] 2. This step defines the range of video segments to be predicted in the real-time video stream, i.e., the start and end times of the video segments. To obtain the video segments to be predicted, this step requires manual input of the start signal of the predicted video segment using keyboard shortcuts. Typing control+s represents the start signal of the predicted segment, and control+e represents the end signal of the predicted segment. This step obtains the set of video frames to be predicted: Frames = {x1,…,x…} m}, where x defines the image to be predicted, m represents the length of the segment, i.e., the total number of frames, and the lesion region L corresponding to each frame image in step a. t By cropping, a set of local images I for each frame is obtained. t ={l1,…,l n}, and the corresponding confidence set P t ={p1,…,p n};

[0118] 3. This step filters out localized lesion areas with a small area ratio extracted in step 2. This step is performed only on each frame of the image between the start and end time points described in step 2. The ratio of the local image to the entire image is... Set the threshold for the proportion of a local image relative to the whole image. ,like l i Retained in I t Otherwise, filter out l. i And in the confidence set P t p in i .like Figure 1 As shown in step three, after this step, an optimized set of local lesion areas is obtained. and confidence set

[0119] 4. This step uses a single frame image x within the predicted segment represented in step 2. t and the optimized set of local lesion images generated in step 3 and confidence set as input, where single frame image x t as the input of the global feature extraction backbone network F1(·), and as the input of the local feature extraction backbone network F2(·), finally obtaining the probability distribution of each level of the bleeding level of the frame image This step also contains an iterative input method commonly used in the field, which can enable F2(·) to dynamically respond to a variable number of local pictures;

[0120] 5, this step obtains the bleeding level probability distribution of each frame in the video segment to be detected obtained in step 4 Weight accumulation is performed, and the weight of each category is set as W={w0,w1,w2,w3,w4}, where the weight of the higher level is set higher, and the total weight is accumulated as 1=w0+w1+w2+w3+w4,w4=5*w0,w3=4*w0,w2=3*w0,w1=2*w0, When using weight accumulation probability, the highest weight of the current frame probability is selected as the weight for accumulation. Finally, the video segment lesion biopsy bleeding level probability P=(p0,p1,p2,p3,p4) is obtained, and the category with the maximum probability in the cumulative probability distribution is taken as the final bleeding risk level output.

[0121] In the above steps, for the problem of single frame bleeding level prediction, a dynamic multi-path deep learning network model is designed and implemented, which can process a variable number of picture inputs, fuse the features of all input pictures, and obtain classification probability using the fused feature vector. As shown in Figure 2 The model has two inputs, one of which is the original image x t After the original image is input into the global image feature extraction network, the global feature vector The other input is a plurality of local images corresponding to the original image and the confidence of the local image Since the local feature extraction network needs to respond to a variable number of local images, the present application specially designs to extract features from each local image in to obtain a local feature set The local feature set is weighted and averaged to obtain a local fusion feature vector When the local feature set is empty, it is a 0 vector; the global vector and the local feature vector are fused using the average value method to obtain the final feature vector of the single frame picture After the feature vector passes through the full connection layer and Softmax, the class probability distribution of the current frame is obtained The specific structure of the model is shown in Figure 6 When performing the bleeding grade classification, the information of the global feature vector and several lesion region local feature vectors is fused, wherein the local feature vector is used to capture the details of the lesion region, and the global feature vector provides the overall background comparison, so that the model can more accurately predict the grade. The model is applied to step 4.

[0122] For the problem of obtaining bleeding grade prediction in the prediction segment, a multi-frame dynamic weight accumulation fusion algorithm is designed and implemented, which is applied to step 5. The algorithm includes the following three steps:

[0123] The weight of each category W={w0, w1, w2, w3, w4} is preset by using the following conditional equation, which satisfies the following conditions:

[0124] 1=w0+w1+w2+w3+w4,

[0125] w4=5*w0,

[0126] w3=4*w0,

[0127] w2=3*w0,

[0128] w1=2*w0

[0129] The probability distribution of each frame needs to be weighted with the weight of the corresponding highest probability category:

[0130]

[0131] Here The highest probability category l corresponding to the frame is calculated, and the preset weight w of one of W is obtained by using the category as the index of the weight l Weighting p t , the weighted probability distribution is The weighted vector set of all frames in the segment is

[0132] Each category in the set is accumulated to obtain the bleeding grade prediction category distribution of the lesion biopsy in the to-be-predicted segment:

[0133]

[0134] Here P=(p0, p1, p2, p3, p4) represents the biopsy bleeding grade classification probability corresponding to the segment;

[0135] The category corresponding to the highest probability is taken as the final prediction grade:

[0136]

[0137] Based on the biopsy bleeding grade prediction method provided in embodiment 1, a certain number of application tests are carried out. During the bronchoscopy and biopsy operation, the bleeding risk level prompted by the system shows a strong consistency with the subjective judgment of experienced doctors, embodies excellent auxiliary function, shows higher prediction ability than the existing bleeding risk prediction model, and greatly reduces the work burden of doctors.

[0138] It should be understood that the above embodiments are only to illustrate the technical concepts and characteristics of the present application, and the purpose is to enable those skilled in the art to understand the content of the present application and to implement it, and cannot limit the protection scope of the present application. Any equivalent changes or modifications made in accordance with the spirit and essence of the present application shall be covered within the protection scope of the present application.

Claims

1. A method for predicting a bronchoscopic biopsy bleeding grade based on a dynamic multi-path deep learning model, characterized in that, The method comprises: obtaining a video clip, the video clip comprising a plurality of bronchial images sorted by time, and identifying a local lesion area in the bronchial images; extracting features of the bronchial images by using a dynamic multi-path deep learning model, the dynamic multi-path deep learning model comprising a global branch and a local branch, the global branch identifying global features of the bronchial images, and the local branch identifying local features of the local lesion area; fusing the global features and the local features to obtain fused features, and classifying a risk level based on the fused features to obtain a risk level probability distribution; the process of feature extraction and fusion and obtaining the risk level probability distribution is represented as: wherein, denotes the global feature, F1(·) denotes the global branch, denotes the local feature, denotes the set of local lesion regions, denotes the corresponding set of lesion confidence, F2(·) denotes the local branch, denotes the preliminary fusion of the local branch, f denotes the fused feature, F(·) denotes the feature decoder, f j denotes the feature value corresponding to each risk level in the local feature, n denotes the number of local lesion regions, p t denotes the risk level probability distribution, denotes the probability value corresponding to each risk level in the risk level probability distribution, o denotes the highest number of risk levels; dynamically weighting each probability value in the risk level probability distribution to obtain a weighted probability distribution, the weight coefficient of the dynamic weighting being positively correlated with the risk level corresponding to the highest probability in the risk level probability distribution; for the video clip, accumulating the weighted probability value corresponding to each risk level in the weighted probability distribution to obtain a fused probability distribution; the process of dynamic weighting and obtaining the fused probability distribution is represented as: 1 = w0+ w1+ w2+... + w o , w o = a o *w0, …… w3 = a3 * w0, w2 = a2 * w0, w1 = a1 * w0 wherein w · denotes the weight coefficient, a · denotes a dynamic coefficient and takes a value increasing with the increase of the subscript value o, W denotes a set of the weight coefficients, denotes the weighted probability distribution, denotes a probability value corresponding to each risk level in the weighted probability distribution, k = 0, 1, 2, … o, denotes an index of the risk level, denotes a set of the weighted probability distributions, P denotes the fusion probability distribution, p k denotes a probability value corresponding to each risk level in the fusion probability distribution, l denotes an index of the risk level corresponding to the maximum probability value, m denotes the number of bronchial images in the video segment; based on the probability value of each risk level in the fused probability distribution, screening to obtain a bleeding risk prediction level corresponding to the video clip.

2. The bronchoscopic biopsy bleeding grade prediction method according to claim 1, characterized by, The process of obtaining the video clip specifically comprises: obtaining a video stream; specifying a time segment in the video stream, and cutting the images in the time segment as the video clip.

3. The bronchoscopic biopsy bleeding grade prediction method of claim 1, wherein, The process of obtaining the local lesion area specifically comprises: performing image recognition on the bronchial images to obtain an initial local area; performing size screening on the initial local area to obtain the local lesion area, the size screening being based on the pixel size of the initial local area, and removing the initial local area with a pixel size below a threshold.

4. The bronchoscopic biopsy bleeding grade prediction method according to claim 3, characterized by, The process of obtaining the local lesion area is represented as: computing an area ratio of the initial local region to the bronchus image wherein area(·) represents pixel area, l i represents the initial local region, l i = (x i ,y i ,w i ,h i ,p i ), x i ,y i represent pixel center coordinates of the initial local region, w i ,h i represent pixel width and pixel length of the initial local region, p i represents confidence that the initial local region is classified as a lesion, x t represents the bronchus image, i represents an index number of the initial local region, and t represents an index number of the bronchus image; The threshold of the proportion of the local image to the whole image is set as If initial local region l i retained in the set of local lesion regions I t , otherwise filtered out i and its confidence value p t in the set of lesion confidences P i .

5. The bronchoscopic biopsy bleeding grade prediction method of claim 1, wherein, The screening process of the bleeding risk prediction level is represented as: wherein Level represents the bleeding risk prediction level.

6. The bronchoscopic biopsy bleeding grade prediction method of claim 1, wherein, The global branch and the local branch are located in a backbone network in the dynamic multi-path deep learning model, and the architecture of the backbone network adopts any one of Resnet, EfficientNet, ConvNext, and Swin Transformer.

7. A bronchoscopic biopsy bleeding grade prediction system based on a dynamic multi-path deep learning model, characterized by, The bronchoscopic biopsy bleeding level prediction system is used to perform the bronchoscopic biopsy bleeding level prediction method of any one of claims 1-6, comprising: a data acquisition module for obtaining a video clip, the video clip comprising a plurality of bronchial images sorted by time, and identifying a local lesion area in the bronchial images; a feature extraction module for extracting features of the bronchial images by using a dynamic multi-path deep learning model, the dynamic multi-path deep learning model comprising a global branch and a local branch, the global branch identifying global features of the bronchial images, and the local branch identifying local features of the local lesion area; The feature fusion module is configured to fuse the global feature and the local feature to obtain a fusion feature, perform bleeding risk level classification based on the fusion feature, and obtain a risk level probability distribution; The feature weighting module is configured to dynamically weight each probability value in the risk level probability distribution to obtain a weighted probability distribution, wherein a weight coefficient of the dynamic weighting is in a positive correlation with a risk level corresponding to a highest probability in the risk level probability distribution; The probability fusion module is configured to accumulate a weighted probability value corresponding to each risk level in the weighted probability distribution to obtain a fusion probability distribution for the video segment; The level prediction module is configured to filter a risk prediction level corresponding to the video segment based on a probability value of each risk level in the fusion probability distribution.

8. A readable storage medium, characterized by, The readable storage medium stores a computer program, and the computer program is executed to perform the steps of the bronchoscopy biopsy bleeding level prediction method in any one of claims 1-6.

Images