Method and system for assisting bronchus endoscope microscopic examination, electronic equipment and storage medium

Abstract

The invention discloses a method and system for assisting bronchus endoscope microscopic examination, electronic equipment and a storage medium. The method comprises the following steps: acquiring a bronchus endoscope examination image in real time; operating a first pre-trained classifier to classify the bronchus endoscopic examination image, wherein the bronchus endoscopic examination image comprises mark identification information used for identifying at least two inner cavities; in response to the classified bronchus endoscope examination image, obtaining an identification result of the bronchus endoscope examination image, and calling a state request for obtaining the microscopic examination condition of the bronchus endoscope; and feeding back the real-time microscopic examination condition of the bronchus endoscope according to the state request. According to the scheme, the current position of the bronchoscope can be intelligently identified by utilizing the bronchus tree model, so that a doctor is assisted in judging and updating the position of the bronchoscope, missing detection of a key position or wrong detection of other positions can be prevented, meanwhile, the microscopic examination operation level, the examination image quality and the like are evaluated in real time, and the operation efficiency is improved. Therefore, the reliability and the accuracy of bronchoscopy results are improved.

Description

Technical Field

[0001] The present disclosure generally relates to the field of medical intelligence technology. More specifically, the present disclosure relates to a method, system, electronic device and computer-readable storage medium for assisting bronchoendoscopic examination. Background Art

[0002] Bronchoscopy has always been considered the gold standard for diagnosing and screening lung diseases, especially early lung cancer. However, the quality of bronchoscopy is easily affected by many factors. First, the degree of perfection of the patient's preoperative preparation, such as fasting, sedation, local anesthesia, etc., will affect the patient's cooperation and the effect of endoscopic observation. Secondly, human factors such as the operating doctor's technical level, experience accumulation, and energy state are crucial to the comprehensiveness, meticulousness and examination time of the endoscopic examination. Furthermore, the internal structure of the bronchi is intricate, and the anatomical morphology of different parts such as the left and right main bronchi and each lobe segment of the bronchi is highly similar, which increases the difficulty of comprehensive examination in clinical practice. Traditional bronchoscopy may lead to uneven quality of bronchoscopy. Improper operation during the examination may cause damage, bleeding, edema, etc. to the local mucosa, and the risk of missed diagnosis of abnormal parts is high.

[0003] Prior art CN112614566A discloses a method for identifying parts based on a deep learning model. Its image recognition method relies on recognizing a part image and then considering that the part is covered. The system fails to distinguish between the left and right lungs, which may confuse these anatomical features, resulting in incorrect part recognition, and at the same time reduce the doctor's experience and increase the difficulty of understanding image annotation. Furthermore, the system involves multiple models that are interdependent. The output of model one directly affects the input of model two. If the misjudgment rate of model one is high, the image received by model two may contain a large number of unqualified images, which will seriously affect the accuracy and efficiency of subsequent recognition. Similarly, the output of model two affects model three. If model two does not accurately classify in vivo / in vitro, model three may receive wrong images, resulting in errors in part recognition. In real-time applications, the data flow is inefficient and the real-time performance is poor. In addition, there is a lack of monitoring and feedback mechanisms between multiple models. Model output verification relies only on manual reliability and stability, which is difficult to control. The poor generalization ability of the model and the lack of feedback mechanisms make the entire system unreliable and unable to interact with doctors.

[0004] At present, the acquisition of real-time feedback and information identification of bronchoendoscopic examination conditions, such as the completeness of site identification and the reliability of site identification, mainly rely on the doctor's own skills to carry out the inspection operation and the judgment of the inspection results. There is a lack of quantitative quality assessment methods for the operation quality and inspection completeness of bronchoscopy in the real-time inspection process. The standardization and systematicness of endoscopic examinations lack reliable auxiliary guidance and evaluation, and the quality and reliability of the inspections are difficult to guarantee, which in turn affects the accuracy of the diagnosis. Therefore, there is an urgent need for an intelligent system that can monitor the bronchoendoscopic examination process in real time, objectively evaluate the operation quality, and assist doctors in comprehensive and detailed inspections, so as to provide more reliable and intelligent auxiliary tools for bronchoendoscopic examinations and improve the level of bronchoscopic diagnosis and treatment. Summary of the invention

[0005] In order to at least solve one or more of the technical problems mentioned above, the present disclosure proposes a solution for bronchoendoscopic assisted examination, which uses artificial intelligence technology to realize intelligent monitoring and quality analysis of the entire process of endoscopic examination, so as to standardize operation behavior and improve the diagnosis level. To this end, the present disclosure proposes specific solutions in the following aspects.

[0006] In a first aspect, the present disclosure provides a method for assisting bronchoendoscopy, comprising:

[0007] Real-time acquisition of bronchoscopic images;

[0008] operating a first pre-trained classifier to perform location classification on a bronchoscopic examination image, the bronchoscopic examination image including landmark identification information for identifying at least two lumens;

[0009] In response to the bronchoscopic examination image after positioning and classification, obtaining a recognition result of the bronchoscopic examination image;

[0010] In response to the recognition result, calling a status request for obtaining a microscopic examination status of the bronchial endoscope;

[0011] In response to the status request, the real-time endoscopic status of the bronchial endoscope is fed back.

[0012] In some embodiments, in response to a classified bronchoscopic examination image, obtaining a recognition result of the bronchoscopic examination image includes: obtaining bronchial landmark recognition information; using the landmark recognition information, obtaining at least one inner cavity for determining the bronchoscopic examination image; in response to identifying at least one inner cavity, triggering execution of a branch structure of at least one inner cavity; using a pre-trained feature fusion classification network to perform feature fusion on image features and sequence features of multiple first continuous image frames collected within a first predetermined time to obtain fused image features; and inputting the fused image features into a preset multi-dimensional fully connected layer to obtain a classification result of the last frame of the multiple first continuous image frames as the recognition result of the specific part.

[0013] In some embodiments, feature extraction is performed on multiple first continuous image frames to obtain image features; the input image features are compressed based on global averaging to obtain a multidimensional feature vector; the multidimensional feature vector is generated as an attention weight coefficient based on an activation function, and the attention weight coefficient is multiplied element-by-element with each image feature and then fused to obtain a fused image feature.

[0014] In some embodiments, in response to the recognition result, calling a status request for obtaining the endoscopic examination status of the bronchoscope includes: performing real-time detection on the bronchial endoscopy examination image based on a target detection model trained to determine abnormal images in the bronchial examination image, and obtaining the position coordinates corresponding to the abnormal image; and in response to the abnormal area associated with the position coordinates, sending a status request to feedback the abnormal condition of the abnormal area.

[0015] In some embodiments, the bronchoscopic inspection image is detected in real time based on a target detection model trained to determine abnormal images in the bronchoscopic inspection image, and obtaining the position coordinates of the abnormal image includes: performing preprocessing of scaling and pixel value normalization on the bronchoscopic inspection image, and inputting it into the target detection network to obtain a detection frame of the abnormal image; performing de-redundancy processing on the detection frame of the abnormal image corresponding to the abnormal data, and optimizing the final abnormal detection result by merging or suppressing overlapping detection frames to obtain a target detection frame including the best result; and performing an inverse coordinate transformation on the target detection frame to obtain the position coordinates of the abnormal area of ​​the bronchoscopic inspection image.

[0016] In some embodiments, in response to the recognition result, calling a status request for obtaining the endoscopic status of the bronchoscope includes: performing preprocessing including scaling and pixel value normalization on the bronchial endoscopic inspection image; using a second pre-trained classifier to perform validity classification on the preprocessed bronchial endoscopic inspection image to obtain the corresponding category with the highest confidence in the output node as the valid image in the validity classification result, and the second pre-trained classifier is used to at least determine whether the bronchial endoscopic inspection image is a valid image based on the input image; in response to identifying that the current bronchial endoscopic inspection image is a valid image, starting a timer for obtaining the operation time for cumulative timing.

[0017] In some embodiments, in response to the recognition result, calling a status request for obtaining the endoscopic status of the bronchoscope includes: using a third pre-trained classifier to perform clarity classification on the bronchial endoscopy image to obtain a blurred image in the clarity classification result, the third pre-trained classifier being used to at least determine whether the bronchial endoscopy image is a blurred image based on the input image; obtaining a proportion value of blurred frames in a predetermined number of second consecutive image frames in the blurred image; in response to the blurred image having a proportion value not less than a preset proportion being a blurred image, issuing an adjustment request for adjusting the bronchoscope.

[0018] In some embodiments, obtaining the ratio value of blurred frames in a predetermined number of second consecutive image frames in a blurred image includes: collecting multiple second consecutive image frames in the current blurred image; performing clarity classification on the multiple second consecutive image frames through a third pre-trained classifier, and obtaining all blurred frames in the clarity classification results to obtain the ratio value.

[0019] In some embodiments, in response to a status request, feeding back the real-time endoscopic status of the bronchoscope also includes: determining a first display sub-style of at least one feedback graphic based on the recognition result of the bronchial endoscopy image, and displaying a second display sub-style of at least one interactive feedback graphic based on the request category of the status request; determining a display style of at least one feedback graphic based on the first display sub-style and the second display sub-style, and displaying at least one feedback graphic in an associated area of ​​the operation instruction based on the display style.

[0020] In some embodiments, determining a first display sub-pattern of at least one feedback graphic according to the recognition result of the bronchoendoscopic examination image includes:

[0021] Based on the feature elements in the recognition result, at least an observation area after the feature elements are emphasized is indicated, and the observation area includes a route that has been located or an area that has been located.

[0022] In a second aspect, the present disclosure provides a system for assisting bronchoscopic examination, which implements the method of the aforementioned first aspect and multiple embodiments thereof, and the system includes: an image acquisition unit, used to acquire bronchoscopic examination images in real time; an operation classification unit, used to operate a first pre-trained classifier to locate and classify the bronchoscopic examination images; an image recognition unit, used to obtain a recognition result of the bronchoscopic examination image in response to the bronchoscopic examination image after positioning and classification, and the bronchoscopic examination image includes at least two cavities; a request unit, in response to the recognition result, calling a status request for obtaining the endoscopic examination status of the bronchoscope; a status feedback unit, in response to the status request, feedback the real-time endoscopic examination status of the bronchoscope.

[0023] In a third aspect, the present disclosure provides an electronic device, comprising: a processor; and a memory storing executable program instructions, which, when executed by the processor, enables the device to implement the method of the aforementioned first aspect and its multiple embodiments.

[0024] In a fourth aspect, the present disclosure provides a computer-readable storage medium having executable program instructions stored thereon, which, when executed by one or more processors, implement the method of the aforementioned first aspect and its multiple embodiments.

[0025] By using the bronchoscopic assisted microscopy scheme disclosed in this invention, real-time monitoring and quality assessment of the bronchoscopic examination process can be achieved. Specifically, the scheme disclosed in this invention uses computer vision and artificial intelligence technology to automatically obtain the status information of the bronchoscopic examination, the site recognition results of the microscopic examination process, the cumulative effective operation time, the microscopic examination feedback under blurred vision, the abnormal site recognition feedback, etc. The status request acquisition of multiple microscopic examination conditions can be run at the same time, and the real-time status can be independently fed back, reducing the delay of the system inspection process and maintaining a high system real-time and reliability. The final result is usually integrated or fused to the display interface in the form of feedback images, so that the doctor can pay attention to the real-time display feedback of the operation quality assessment, image quality, abnormal sites, etc. during the operation, and provide intelligent assistance and feedback for bronchoscopic diagnosis and treatment operations. This helps to standardize the endoscopic examination process, improve the operation level, and ultimately indirectly improve the effect of disease diagnosis and treatment.

[0026] Furthermore, the disclosed solution adopts a hierarchical recognition strategy for image features and the idea of ​​establishing independent recognition models for different anatomical regions of the human body, further improving the accuracy and reliability of identifying different parts of the bronchi. At the same time, multi-dimensional information such as abnormal areas of bronchial mucosa, clarity of endoscopic field of view, and examination time can be obtained to comprehensively evaluate the operation of the bronchoscope. This allows the entire bronchoscopic examination system to maintain a certain degree of redundancy. For example, when evaluating the doctor's operating level, multiple models are used in multiple steps to complete the evaluation of the doctor's operating level through statistics of effective operating time and adjustment prompts under blurred field of view. More information can help doctors promptly discover deficiencies in their operations and optimize inspection strategies, thereby improving overall quality.

[0027] It should be noted that the bronchoendoscopic assisted examination method disclosed herein aims to improve the standardization and effectiveness of the endoscopic examination itself, and is an intelligent assistance to the doctor's diagnosis and treatment behavior, but it does not directly involve disease diagnosis and treatment. The results of the endoscopic examination still need to be professionally judged and diagnosed and treated by the doctor based on clinical experience. The method disclosed herein is intended to provide more objective and comprehensive reference information for this process, rather than replacing the doctor in making a diagnosis and formulating a treatment plan. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] By reading the detailed description below with reference to the accompanying drawings, the above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:

[0029] Figure 1 The anatomical divisions of the bronchial tree are schematically shown;

[0030] Figure 2 A flow chart schematically illustrates a method for assisting bronchoendoscopy according to the present disclosure;

[0031] Figure 3 is a schematic diagram showing a module for bronchoscopic site identification according to the present disclosure;

[0032] Figure 4 An exemplary flow chart showing the processing performed by the feature fusion classification network according to an embodiment of the present disclosure;

[0033] Figure 5-1 A flowchart schematically illustrates a method for obtaining real-time bronchoscopic status according to an embodiment of the present disclosure;

[0034] Figure 5-2 A flowchart schematically illustrates a method for obtaining abnormal areas of bronchial real-time microscopy according to an embodiment of the present disclosure;

[0035] Figure 6 A flowchart schematically illustrates a method for obtaining real-time bronchoscopic status according to another embodiment of the present disclosure;

[0036] Figure 7 A flowchart schematically illustrates a method for obtaining real-time bronchoscopic status according to yet another embodiment of the present disclosure;

[0037] Figure 8 A block diagram schematically illustrates a system for assisting bronchoendoscopy according to an embodiment of the present disclosure; and

[0038] Fig. 9 The block diagram schematically shows an electronic device for assisting bronchoendoscopy according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0040] It should be understood that the terms "include" and "comprising" used in the specification and claims of the present disclosure indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0041] It should also be understood that the terms used in this disclosure are only for the purpose of describing specific embodiments and are not intended to limit the disclosure. As used in this disclosure and claims, the singular forms of "a", "an", and "the" are intended to include the plural forms unless the context clearly indicates otherwise. It should also be further understood that the term "and / or" used in this disclosure and claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations.

[0042] As used in this specification and claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0043] Bronchoscopy is an important means of screening and diagnosing respiratory diseases, especially lung tumors. However, due to the complexity of the bronchial anatomical structure, the differences in patient preparation and cooperation, and the heterogeneity of doctors' operating experience and technology, the quality of bronchoscopy is difficult to stabilize at a high level, and it is difficult to monitor and objectively evaluate doctors' operations in real time. At present, there is a lack of intelligent systems specifically used to assist bronchoscopy in clinical practice. The realization of the auxiliary endoscopy function in clinical practice often depends on the doctor's experience, estimating the effective operation time of the bronchoscope in the body, relying on personal subjective experience to achieve the identification of bronchial sites, the prompt of blurred vision, the judgment of image clarity, the identification / prompt of abnormal sites, etc. When doctors with insufficient experience operate bronchoscopes for examination, the systematicness and integrity of the examination are difficult to ensure, the risk of missed diagnosis is high, the safety of patients is difficult to ensure, and the overall clinical examination effect is poor. These problems seriously limit the effectiveness of bronchoscopy technology in the early screening and diagnosis and treatment of lung diseases. Therefore, the intelligent operation level evaluation and operation assistance functions during bronchoscopy are crucial for assisting doctors' operations and visually checking the reliability of doctors' examination results.

[0044] In view of the above situation, the disclosed embodiment provides a bronchoendoscopic assisted microscopic examination method and system. The method uses computer vision and machine learning algorithms to perform intelligent image analysis on the real-time video stream collected by the bronchoscope. By analyzing multiple image frames or multiple continuous images in the video stream, the microscopic examination conditions of different dimensions during the bronchoscopic examination are obtained, which is crucial for the intelligent auxiliary operation of bronchial examination and the evaluation of operation quality. First, the bronchial endoscopy examination image is anatomically positioned through the image recognition positioning classification model, and the integrity and systematicity of the examination are monitored in real time, effectively preventing the omission and misdetection that may occur during the doctor's operation; secondly, the pre-trained target detection model is used to identify and prompt the abnormal mucosal area; thirdly, the effective operation time of the endoscope in the examination process is counted to evaluate the effectiveness of the operation; at the same time, the clarity of the endoscopic field of view is evaluated, and the doctor is prompted whether the direction of the bronchoscope needs to be adjusted or the lens needs to be cleaned by confirming the blurred field of view. The above information is fed back to the doctor in real time in an intuitive and friendly manner to guide standardized operations. By comprehensively applying the above functions, this disclosure can evaluate the quality of bronchoscopic examination in multiple dimensions, standardize the operation process, reduce the risk of missed diagnosis, improve diagnostic efficiency, and enhance diagnostic reliability. Compared with the existing technology, this disclosure introduces intelligent assistance and microscopic examination mechanism in the process of bronchoscopic examination, which makes up for the limitations of individual doctor's experience and ability, and is expected to significantly improve the level of bronchoscopic diagnosis and treatment.

[0045] The specific implementation of the present disclosure is described in detail below with reference to the accompanying drawings.

[0046] Figure 1The anatomical structure of the bronchial tree is schematically shown, including the three regions of the trunk, left lung, and right lung, as well as the specific bronchial branches in each region.

[0047] As mentioned above, an important goal of bronchoscopy is to comprehensively and systematically observe all the main branches of the bronchial tree to detect possible lesion areas. In order to achieve a quantitative assessment of the completeness of bronchoscopy, the present disclosure divides the bronchial tree into multiple standardized sites according to the anatomical structure. These multiple sites are first divided into three major regions: the trunk, left lung, and right lung. The trunk region includes the pharynx, glottis, trachea, and carina; the left lung region includes the left main bronchus and all segmental bronchi of the left upper lobe and the left lower lobe, such as the posterior segment of the left upper lobe apex and the posterior basal segment of the left lower lobe; the right lung region includes the right main bronchus and all segmental bronchi of the right upper lobe, right middle lobe, and right lower lobe, such as the anterior segment of the right upper lobe and the outer basal segment of the right lower lobe.

[0048] During bronchoscopy, doctors can manipulate the endoscope to move slowly in the tracheal cavity to carefully observe the mucosal surface and branch openings. The path of the lens can traverse all of the above-mentioned standardized sites, and each site can be carefully observed and evaluated. However, due to the complexity and numerous branches of the bronchial tree, and the very similar anatomical morphology of many branches, it is difficult for doctors to accurately determine the specific location of the lens based on experience, and it is difficult to ensure that all important branches have been examined. This leads to the blindness and uncertainty of bronchoscopy, and the risk of missed diagnosis is very high.

[0049] In order to solve the above problems, the present disclosure provides a bronchoscopy-assisted microscopy technology, which can track and confirm the specific position of the endoscope lens in the tracheal cavity in real time, objectively record and evaluate the complete inspection of the bronchial tree, and reduce human judgment bias. Furthermore, the present disclosure provides a bronchoscopy-assisted microscopy method, which automatically analyzes the video images collected by the endoscope through computer vision technology and artificial intelligence algorithms, accurately identifies the real-time position of the bronchial branch where the head of the scope is located, locates and classifies specific parts or abnormalities that need to be identified in the image, and judges the integrity and systematicness of the inspection accordingly. At the same time, the feedback of the microscopy results can provide doctors with real-time prompts and feedback, effectively avoiding inspection blind spots and the risk of missed diagnosis. This method is expected to significantly improve the quality of bronchoscopy and assist in the screening and diagnosis of early lung diseases.

[0050] Figure 2 The flowchart of the method 200 for assisting bronchoendoscopy according to the present disclosure is schematically shown. It should be understood that the method 200 can be implemented by hardware, firmware, software or a combination thereof.

[0051] like Figure 2As shown, first, at step S202, a bronchoscopic examination image is acquired in real time; then, at step 204, a first pre-trained classifier is operated to locate and classify the bronchoscopic examination image, and the bronchoscopic examination image includes at least landmark identification information that can be used to identify two inner cavities corresponding to the left main bronchus and the right main bronchus; then, at step S206, in response to the classified bronchoscopic examination image, a recognition result of the bronchoscopic examination image is acquired; then, at step S208, in response to the recognition result, a status request for obtaining the microscopic examination status of the bronchoscope is called; finally, at step 210, in response to the status request, the real-time microscopic examination status of the bronchoscope is fed back.

[0052] In some embodiments, the bronchoendoscopic examination images may be real-time video images acquired through an endoscopic lens, including video stream images within a certain preset time period, or may be key frame images extracted from a video.

[0053] In some embodiments, in the aforementioned step S204, the first pre-trained classifier is operated to locate and classify the bronchoscopic examination image, the bronchoscopic examination image includes at least two cavities, the bronchial system includes the left main bronchus, the right main bronchus and the internal space thereof, and the left main bronchus and the right main bronchus in the bronchi are further branched into a plurality of secondary bronchi, followed by smaller bronchial branches and terminal bronchioles. The first pre-trained classifier can use machine learning or deep learning algorithms to identify the current pre-processed bronchial examination image and correctly locate and classify these structures according to the shape, size, position, texture, color, etc. of the tracheal branches in the bronchoscopic examination image, and the specific training method is described in detail later.

[0054] In some embodiments, the recognition result in the aforementioned step S206 may be the position of the airway / airway branch where the endoscope lens is currently located, such as the posterior segment of the left upper lobe tip, the posterior basal segment of the right lower lobe, etc.

[0055] In some embodiments, in response to the recognition result in the aforementioned step 208, a status request for obtaining the endoscopic examination status of the bronchoscope is called, and the request will be used to query or update the current status of the endoscopic examination, and the status request can be sent to the bronchoscope and its related medical information system, etc. Specifically, the status request can include obtaining a status request that emphasizes the processing of the current airway position according to the position of the airway / airway branch where the endoscope lens is currently located, adjusting the bronchoscope action including the bronchoscope forward and backward, rotation, image data analysis request, abnormal position prompt / display request, bronchoscope operation time timing update request, etc. The status request for the endoscopic examination status can quickly and efficiently transmit the real-time endoscopic examination status of the bronchi to the main interface of the medical system, the bronchoscope, etc. to assist the doctor in taking relevant treatment, and at the same time help the doctor quickly evaluate the level and quality of the endoscopic operation, so as to flexibly adjust according to actual needs.

[0056] In some embodiments, in response to the status request in the aforementioned step S210, the feedback of the real-time microscopic status of the bronchoscope may include displaying processed images and important information such as the microscopic status that the user expects to obtain in the status request, the real-time position of the bronchoscope, etc. to the user in real time, so as to assist the user to adjust the operation in real time according to the microscopic quality, the real-time reliability of the microscopy, etc., or to assist in teaching and scientific research, etc.

[0057] In some embodiments, the real-time microscopic examination status may include the inspection completeness score of each area of ​​the bronchus, the display of the current real-time position, the display of the emphasized position, the number and distribution of abnormal areas, the effective duration of the bronchoscopic operation, the clarity of the field of view, the operation adjustment prompts under blurred field of view, and other information in multiple dimensions.

[0058] In some embodiments, the results of the real-time microscopy condition can be recorded and documented for future reference or as part of the patient's medical record.

[0059] In some implementations, if any exception or error is found during the status request sending or the status request response process, an exception handling process is executed, which may include alarms, logging, or emergency response measures.

[0060] use Figure 2 The method shown in the figure can objectively quantify the integrity and standardization of bronchoscopy, reduce blind spots and missed diagnoses; can remind doctors of the position of the lens in real time, and timely optimize and adjust the endoscopic operation path; can automatically identify suspicious lesions and provide reference for diagnosis; can evaluate the quality of the endoscopic field of view, and prompt timely cleaning of the lens and other operations. In short, the disclosed solution provides intelligent assistance and microscopic examination in the whole process of bronchoscopy, effectively avoiding the limitations of relying entirely on the doctor's personal experience, and comprehensively improving the diagnosis and treatment effect of bronchoscopy.

[0061] Figure 3 is a schematic diagram showing a module 300 for bronchoscopic site identification according to the present disclosure.

[0062] like Figure 3 As shown, the present disclosure proposes a module 300 for bronchoscope site identification, which can at least realize the identification of the main trunk site, left lung branch site, and right lung branch site in the three submodules when using the first pre-trained classifier for positioning and identification. Among them, the trunk module 301 can identify the pharynx, glottis, trachea, carina, left main bronchus and right main bronchus in the airway; the left lung module 302 can identify the bronchial segments of the left lung, the carina and the right main bronchus, and the right lung module 303 can identify the bronchial segments of the right lung, the carina and the left main bronchus. During the actual bronchoscopy, the module 300 dynamically switches between the three modules based on the algorithm and according to the recognition results. For example, when the trunk module recognizes the left main bronchus, it switches to the left lung module; when the right main bronchus is recognized, it switches to the image recognition of the right lung module. When the right lung module 303 recognizes the left main bronchus, it switches to the image recognition of the left lung module 302, and when the left lung module 302 recognizes the right main bronchus, it switches to the image recognition of the right lung module 303. Due to the symmetry of the anatomical structure of the left and right lungs, the left and right lungs are modeled separately, and it is clearly stated whether the recognition result corresponds to the left lung or the right lung, ensuring the accuracy of the results and that each recognition module can recognize the left and right main bronchus, which is conducive to improving the overall recognition rate and avoiding confusion in the recognition of the left and right lungs.

[0063] Specifically, in response to the classified bronchoscopic examination image, obtaining the recognition result of the bronchoscopic examination image includes: obtaining the landmark recognition information of the bronchus; using the landmark recognition information, obtaining at least one inner cavity for determining the bronchoscopic examination image, and in response to identifying at least one inner cavity, triggering execution of the branch structure of at least one inner cavity; using a pre-trained feature fusion classification network to fuse the image features and sequence features in multiple first continuous image frames collected within a first predetermined time to obtain a fused image feature; and inputting the fused image feature into a preset multi-dimensional fully connected layer to obtain the classification result of the last frame of the multiple first continuous image frames as the recognition result of the specific part.

[0064] As an example, the above-mentioned landmark recognition information can be a feature image of tracheal cartilage ring, carina structure, left and right main bronchial openings, etc., which represents the local texture, shape, color, edge, corner point, etc. of different anatomical parts in the bronchoscopic examination image as a non-background image. During initialization, the voting counts of the three modules are set to 0, and the three modules simultaneously perform n times (here "n" is a pre-set number of iterations, which can be adjusted according to actual needs) part recognition on the current same bronchoscopic examination image. The specific content of recognition can be a specific area in the image that includes landmark recognition information, or a specific feature in the image that includes landmark recognition information. When the model recognizes a non-background image, the corresponding model voting count is increased by 1. After n times of recognition, the voting count results of the three models are compared, and the one with the largest voting count is selected as the initial module, and the selected initial module is entered. Thus, the specific part corresponding to the bronchoscopic examination image recognized by the endoscope lens can be determined. The recognition result can further clarify the specific anatomical position of the lens, such as the left main bronchus, the right upper lobe bronchus, etc. The specific part recognition results output by the left lung module and / or the right lung module are more precise positioning within the corresponding anatomical region, such as the posterior segment of the left upper lobe apex and the posterior basal segment of the right lower lobe. By comprehensively utilizing the landmark recognition information, recognition results, and specific parts, the algorithm can achieve layer-by-layer positioning of the bronchial tree from coarse to fine, from global to local, and provide detailed anatomical location information for subsequent diagnosis and treatment operations.

[0065] Furthermore, if there is a tie among the above three modules (i.e., the vote counts of two or more modules are the same), additional strategies may be needed to determine the final model, such as performing another round of recognition or using other indicators (such as recognition accuracy, speed, etc.) for comparison to determine the final selected module.

[0066] In some embodiments, at least one identified inner cavity may be at least one of the trachea, left and right main bronchi, and smaller bronchial branches. In response to identifying the left main bronchus, a switching operation instruction is triggered to execute to the left lung module; the feature fusion classification network of the left lung module is used to identify the specific location of the current bronchoscope in the left lung; in response to identifying the right main bronchus, a switching operation instruction is triggered to execute to the right lung module; the feature fusion classification network of the right lung module is used to identify the specific location of the right lung.

[0067] The first pre-trained classifier mentioned above includes three recognition modules, all of which are trained based on a large amount of labeled data using a supervised machine learning algorithm. First, the existing bronchoscopy videos that have been collected are annotated frame by frame, and the anatomical parts corresponding to each video clip are marked in detail. Taking the trunk module as an example, images of six parts, namely the pharynx, glottis, trachea, carina, left main bronchus, and right main bronchus, are extracted from the annotated video, and some images of other parts are additionally extracted as background classes to form multiple categories (for example, 7 categories) of samples. The samples are divided into training set, validation set, and test set in a ratio of 8:1:1, and care is taken to ensure that the distribution of each anatomical part in the three data sets is consistent, and images of the same case cannot appear in different data sets at the same time, so as not to affect the generalization performance evaluation due to the similarity between cases.

[0068] In some embodiments, the acquisition of the six parts can also be annotated using existing images of the various parts that have been collected. These images should at least include the internal space of the trachea and its branches, including the trachea, left and right main bronchi, and smaller bronchial branches, such as the upper and lower lobe bronchi and their segmental and subsegmental bronchi, etc. The background image can be any other category of images that may be identified during bronchoscopy. The training data set includes normal images of all age groups, abnormal images, and other non-background images.

[0069] After the training set, validation set, and test set are completed, a deep convolutional neural network classification model is then constructed, such as GoogLeNet, ResNet, EfficientNet, etc. The output layer is set to 7 neurons, giving the prediction probabilities of 7 categories respectively. The network weights are randomly initialized, the loss function is defined as cross entropy, and the optimizer is Adam. During model training, the input image is uniformly scaled to 256×256, and data enhancement operations such as flipping and noise addition are performed to enhance the robustness of the model. The image pixel values ​​are normalized by dividing by 255 and then sent to the network training. The training termination time is determined by monitoring the loss changes of the training set and validation set, and the generalization performance is evaluated on the test set. If the accuracy of the test set is not ideal, it is necessary to analyze the characteristics of the error samples and expand the training set in a targeted manner for iterative model optimization.

[0070] Compared with the trunk module, the left lung module and the right lung module face greater recognition challenges. The left and right lungs each contain more than 10 segmental bronchi, and their anatomical structures are more similar to bronchoscopic images, making it difficult to accurately distinguish them based on a single frame of image. Therefore, these two modules adopt a sequence learning strategy, using a pre-trained feature fusion classification network to fuse the image features and sequence features of multiple first continuous image frames collected within the first predetermined time to obtain fused image features; the fused image features are input into a preset multi-dimensional fully connected layer to obtain the classification result of the last frame of the multiple first continuous image frames as the recognition result of the specific part.

[0071] In some embodiments, when using a pre-trained feature fusion classification network to perform feature fusion on image features and sequence features collected within a first predetermined time to obtain fused image features, the process includes: compressing the input image features based on global averaging to obtain a multidimensional feature vector; generating the multidimensional feature vector as an attention weight coefficient based on an activation function, performing an element-by-element multiplication operation on the attention weight coefficient and each image feature, and fusing them to obtain a fused image feature.

[0072] As an example, the context information of the video clip is used to improve the recognition accuracy. Figure 4 As shown, taking the left lung module as an example, when the feature fusion classification network of the left lung module is used for identification, the processing flow includes: when processing the image frame sequence of the bronchoscopic examination image, a first predetermined number (such as 20 frames / 24 frames, etc.) of first continuous image frames are collected from the video stream within a first predetermined time, and key features in the image frames are identified, such as global features or local features such as edges, shapes, textures, structural functions, etc., and feature fusion is performed based on the image features and the sequence features with dynamic information in the video stream to obtain fused image features.

[0073] The attention mechanism is used to output the attention weight coefficients of multiple first continuous image frames; each image feature and the attention weight coefficient are used for weighted fusion to obtain a fused image feature; the fused image feature is connected to a preset multi-dimensional fully connected classification layer (FC) of the feature fusion classification network to obtain the classification result of the last image frame (i.e., the current frame) in the first predetermined number of first continuous image frames, and the recognition result of the specific part is obtained.

[0074] In an exemplary specific operation, 20 consecutive frames of images (sampled at a frequency of 4 frames / second within 5 seconds, represented as M1…M2) may be first taken out during recognition. n), and sent to the convolutional neural network ("CNN") to extract features, and 20 image features were obtained. However, not every frame of the image is equally important for part recognition. In order to optimize feature fusion, the model designs an attention weight coefficient mechanism. The 20 image features are first subjected to global average pooling ("GlobalPooling"), and feature extraction and global average pooling are performed on these key features to obtain a multi-dimensional feature vector. After global average pooling, the feature dimension is reduced and important feature information can be retained. Then, through a 20-dimensional fully connected layer of 20 neurons and a Sigmoid activation function, 20 weight coefficients between 0 and 1 are output to represent the importance of each frame of the image. The Sigmoid function formula is as follows:

[0075]

[0076] For example, the above 20 image features are multiplied by the corresponding weight coefficients and then summed for flattening to obtain the fused image features. Finally, the fused features are passed through a preset dimension fully connected layer (20-dimensional fully connected layer) to give the predicted probability distribution of the segmental bronchus location as the recognition result of the 20th frame image (current frame). The entire network adopts an end-to-end training method, using the cross entropy of the current frame label and the predicted result as the loss function, randomly initialized and using the Adam optimizer to feedback the weights. The processing process of the right lung module is similar to that of the left lung module, which will not be repeated here.

[0077] When the actual system is running, the disclosed solution first obtains the backbone module in the first pre-trained classifier to perform preliminary recognition and positioning of the real-time image in the bronchial examination image. The first pre-selected trained classifier is initialized, and the input image is scaled, normalized, and pre-processed in the same way as the input image in the model training stage, and then sent to the first pre-selected trained classifier after initialization. Each module in the first pre-trained classifier performs n-time part recognition on the same bronchial examination image. Here, "n" is a pre-set number of iterations, which can be adjusted according to actual needs. When any module recognizes a non-background image, the voting count of its corresponding model is increased by 1. After n recognitions, the voting count results of the three models are compared, and the one with the largest voting count is selected as the standard for switching to the corresponding backbone module, left lung module, and right lung module for further recognition.

[0078] After a specific model is selected, for example, when switching to the left lung module, multiple first continuous image frames within a first predetermined time are continuously cached according to the set sampling frequency, for example, 20 frames of images within the past 5 seconds, and sent to the corresponding module in parallel for reasoning to obtain the left and right bronchial prediction labels of the current frame. The disclosed solution dynamically tracks changes in recognition results and switches between different modules in a timely manner, thereby achieving accurate, real-time, and continuous bronchial location positioning.

[0079] By utilizing the above-mentioned hierarchical recognition algorithm designed based on the characteristics of anatomical structure, the present disclosure can overcome the recognition difficulties caused by the high similarity and numerous parts of bronchoscopic examination images, and take into account the recognition efficiency while ensuring accuracy. The preliminary positioning of the trunk module avoids the use of complex models for intensive prediction of all frame images, while the attention mechanism and sequence learning strategy of the left and right lung modules effectively integrate local and contextual information, further improving the level of fine positioning of segmental bronchi. The dynamic recognition results provide doctors with real-time feedback on the location of the lens, ensuring that all parts can be accurately identified during the bronchial examination, which is conducive to the realization of comprehensive and systematic bronchoscopic examinations and reducing the risk of missed diagnosis. The intelligent positioning technology of bronchoscopy provided by the present disclosure is one of the core components of the bronchoscopic auxiliary diagnosis and treatment system, and provides important location prior knowledge for other functional modules such as intelligent identification of mucosal abnormalities and bronchoscopic operation quality assessment, thereby promoting the overall level of clinical application of bronchoscopy. FIG5 schematically shows a flow chart of a method 500 for obtaining the real-time bronchial endoscopic status based on a status request for obtaining the endoscopic status of the bronchoscope according to an embodiment of the present disclosure. Through machine learning, the system integrates an abnormality detection mechanism to determine whether the image detected in real time by the bronchoscope is a possible lesion site or abnormal site, so that when the model output result is determined to be abnormal or not as expected, the specific location of the lesion site or abnormal site can be determined and a prompt for the doctor to respond can be given to remind the doctor to observe the site or the area near the site more carefully, and to collect images and save videos in a timely manner, or to take other corresponding measures. It should be understood that the method 500 can be implemented by hardware, firmware, software, or a combination thereof. Figure 5-1 As shown, the method 500 includes the following steps:

[0080] Step S502: Perform real-time detection on the bronchoscopic examination image based on the target detection model trained to determine abnormal images in the bronchoscopic examination image, and obtain the position coordinates corresponding to the abnormal image, which can ensure that the abnormal area output by the target detection can be correctly mapped to the abnormal area in the original image. Specifically, a target detection model trained with a large number of clinical data including texture abnormalities, color abnormalities, thickness abnormalities, shape abnormalities and normal part images is used to analyze and identify the preprocessed bronchoscopic examination image, and abnormal data with a confidence level not lower than a preset threshold is detected, and the coordinates corresponding to the abnormal data output by these target detection models are converted into the position coordinates of the original image to achieve recognition of a wide range of airway diseases and abnormalities, and a status request for identifying potential lesions or abnormal parts during bronchoscopy is issued based on these abnormal data.

[0081] Step S504: In response to the abnormal area associated with the position coordinates, a status request for feedback of the abnormal condition of the abnormal area is sent. After confirming the abnormal data, the abnormal data can be preprocessed to reduce irrelevant information in the image and highlight the abnormal features of the abnormal image. The preprocessed abnormal data needs to be associated with the image position data to determine the specific location and range of the abnormal area, thereby providing a clearer image basis and more accurate location information for abnormal detection. According to the position coordinates of the identified abnormal area, the specific information of the image, etc., a status request that can only be used for bronchial examination status is issued, including a display request for the specific branch name of the abnormal lesion site, the specific category of the abnormality, etc.

[0082] In some embodiments, the abnormal mucosa at the abnormal site that can be identified includes abnormal color, abnormal shape, abnormal surface texture structure, abnormal mucosal thickness, etc.

[0083] In some embodiments, the request for feedback on the status of the abnormal area may also include, based on the identification of the abnormal area, performing cropping, image enhancement, and other processing on the identified abnormal area, sending a screenshot request to automatically save the abnormal image of the abnormal area locally or in the cloud, and classifying / time-sharing storage based on the initial judgment of the abnormal type.

[0084] like Figure 5-2 As shown, in the above step S502, it further includes:

[0085] Step S5021, the bronchoendoscopic examination image is pre-processed by scaling and normalizing pixel values, and input into the target detection network to obtain the detection frame of the abnormal image corresponding to the abnormal data. In one embodiment, the target detection model can be constructed using the yolov8 algorithm. In order to train the model, bronchoendoscopic examination images are first collected, including positive pictures of abnormal mucosa containing different abnormal types and normal negative pictures that do not contain abnormalities. The coordinate position and abnormal category of the abnormal mucosa in the positive sample are annotated using a preset annotation frame. The category can include abnormal mucosal texture, abnormal mucosal color, abnormal mucosal thickness, abnormal shape of the corresponding area of ​​the mucosa, and other abnormal functions. After the annotated data is prepared, when dividing the data set, the data distribution in the set should be guaranteed to be the same (the number ratio between the positive samples and the negative samples in each set can be roughly the same), and the data of the same case video should not be divided into different data sets to prevent the same part frames of the same case from having a high degree of similarity and causing the model to deviate from the evaluation indicators of the validation set and the test set, greatly enhancing the reliability of the model. For example, the ratio of the number of training sets, validation sets, and test sets used in the model training process can be 8:1:1.

[0086] In order to increase the generalization ability of the model and improve the detection accuracy of the model, the training samples are expanded online using image enhancement. Data enhancement methods include horizontal flipping, upside down flipping, affine transformation, contrast enhancement, random superposition noise, etc. When performing image enhancement that will change the abnormal position, the position of the annotation of the image should be transformed at the same time to ensure that the position of the abnormal mucosa / abnormal part after the transformation is consistent with the coordinate position after the transformation. During model training and before inputting the trained target detection model, the image can be uniformly scaled to 512×512 (when scaling the image, the labeled coordinates still need to be transformed in the same way to ensure that the coordinate position after the transformation is consistent), and then the pixel value is divided by 255 and normalized to between 0 and 1. The target detection model identifies the abnormal area in the input image and obtains the detection box of the abnormal data. At the same time, the model assigns a confidence score to each detected abnormal area. The confidence indicates the degree of confidence of the model in the abnormal prediction.

[0087] Next, in step S5022, redundancy removal is performed on the detection frame of the abnormal image where the abnormal data is located to obtain a target detection frame. The abnormality may take many forms, from simple point abnormalities to complex collective / regional abnormalities, and redundant data deletion is performed according to the abnormal form. Specifically, the redundancy removal process can directly take the detection frame whose confidence is less than the preset detection threshold t in the detection frame of the abnormal data, and filter out the detection frame with high confidence and mark it as the target detection frame. Specifically, the final abnormal detection result is also optimized by merging or suppressing overlapping detection frames to obtain a target detection frame including the best result.

[0088] Specifically, after the training of the target detection model yolov8 is completed, the test set is used to evaluate the model prediction effect, the F1 value of the model under different detection thresholds is calculated, and finally the threshold corresponding to the highest F1 value is selected as the preset detection threshold t of the model.

[0089] Specifically, the detection box of the abnormal data can be consistent with the form of the annotation box in the training stage, and the abnormal category of the abnormal mucosa in the abnormal area can also be obtained through the output detection box. The presentation form of the abnormal area mentioned in the present invention can be abnormal points, or multiple collective / regional abnormalities can be identified, etc.

[0090] In some implementations, the de-redundancy processing of the target detection frame may be performed by executing the NMS (non-maximum suppression algorithm), or by merging the detection frames using post-processing techniques such as Soft-NMS and Weighted-NMS, so as to obtain the de-redundant target detection frame and retain it as the best detection result. Specifically, for highly overlapping detection frames, the merging may be performed by taking the union or intersection of the bounding boxes, or by weighted averaging according to the confidence level.

[0091] When the target detection model is running, the quality of bronchoendoscopic images may be affected by many factors, such as shooting angle, lighting conditions, equipment resolution, etc. For some images with obvious noise and redundancy, post-processing techniques such as non-maximum suppression (NMS) can obviously remove redundancy. At the same time, for bronchial images, there are dense bifurcations. When detecting abnormal parts, according to multi-scale target detection, each target may have multiple detection frames, and the target part may be blocked, the target similarity is very high, or the boundary between the potential lesion part and the normal part of the bronchus may not be clear. The use of post-processing techniques can fully consider the confidence, position and size of the detection frame, as well as the relationship between them. Images of different scales can be considered when merging redundancy, ensuring that redundancy can be effectively removed at all scales, making the image clearer and the boundary clearer, reducing the probability of abnormal misjudgment, and greatly improving the reliability of clinical applications.

[0092] Finally, in step S5023, the target detection frame is subjected to an inverse coordinate transformation to obtain the position coordinates of the abnormal area of ​​the bronchoendoscopic examination image. Specifically, the output target detection frame can be understood as the position coordinates of the abnormal mucosa on the input image 512x512. It is necessary to perform an inverse coordinate transformation on this target detection frame according to the scaling method of the previous model training to obtain the original image position coordinates of the abnormal mucosa on the original bronchoendoscopic examination image, so as to ensure that the abnormal area output by the target detection can be correctly mapped to the abnormal area in the original image.

[0093] In some embodiments, the target detection box input box is a scaled image obtained in a training manner. If the original image 1000*1000 is scaled to 500*500 after scaling, the target detection model outputs the model output area (200, 200), (400, 400), then the corresponding abnormal mucosa position coordinates are (400, 400), (800, 800). When the image length and width are scaled in any equal or unequal ratio, the inverse transformation method is the same, which is not limited here. In some embodiments, the original image position coordinates of the abnormal mucosa on the original bronchoendoscopic examination image are improved through an intuitive user interface, such as the location and classification of abnormal areas, to improve the accuracy of diagnosis.

[0094] Furthermore, when the abnormal area is synchronously displayed on the host interface, it is necessary to perform a translation transformation based on the coordinate inverse transformation and adapt it to the display interface according to the coordinate system of the image display screen to ensure that the abnormal area output by the target detection can be correctly mapped to the abnormal area in the original image, so as to facilitate the real-time observation of the auxiliary doctor and improve the accuracy of the diagnosis.

[0095] In some implementations, pixel coordinates of the abnormal area are extracted based on image metadata, including information such as pixel spacing and image direction matrix, and the pixel coordinates of the abnormal image are converted into actual physical coordinates as their position coordinates.

[0096] Using the above process, this embodiment realizes the real-time identification and positioning of abnormal mucosal areas or abnormal parts in bronchoscopic images based on the target detection model, and can roughly identify the abnormal category. By training the yolov8 model on a large amount of labeled data, and using data enhancement, threshold optimization and other strategies to improve the robustness and accuracy of the model, this method can automatically detect suspicious lesions during endoscopic examination. During bronchoscopy, the bronchial parts or bronchial areas observed in real time are detected and the position coordinates of the abnormal areas are confirmed, so as to provide real-time feedback on the abnormalities occurring during the detection process according to the actual needs of the doctor's operation; reduce the doctor's observation burden and reduce the risk of missed diagnosis. At the same time, accurate positioning of abnormal areas helps doctors make accurate biopsy sampling and treatment plans. Combining this method with the aforementioned bronchial anatomical structure positioning technology is expected to comprehensively improve the standardization and effectiveness of bronchoscopy.

[0097] Figure 6 The flowchart of a method 600 for obtaining the real-time bronchial endoscopic status in response to a recognition result and calling a status request for obtaining the endoscopic status of a bronchoscope according to another embodiment of the present disclosure is schematically shown. It should be understood that the method 600 can be implemented by hardware, firmware, software, or a combination thereof. Figure 6 As shown, the method 600 includes:

[0098] In step S602, preprocessing including scaling and pixel value normalization is performed on the bronchoendoscopy image.

[0099] In step S604, the preprocessed bronchoendoscopic examination image is classified for validity using a second pre-trained classifier, so that the corresponding category of the highest confidence of the output node is a valid image in the validity classification result, and the second pre-trained classifier is used to at least determine whether the bronchoendoscopic examination image is a valid image based on the input image. The second pre-trained classifier can be implemented using a lightweight convolutional neural network such as RepVGG, MobileNet, ShuffleNet, etc. For example, the output nodes of the classifier can be set to 2, corresponding to the two categories of valid and invalid, respectively.

[0100] The training process of the second pre-trained classifier is as follows: First, a large number of bronchoscopic images are collected, including valid images and invalid images, and the category information is manually annotated. All samples are divided into training set, validation set and test set in a ratio of, for example, 8:1:1, and the distribution of clear image and blurred image samples in each data set is ensured to be consistent. Then, the above-mentioned lightweight convolutional neural network classification model is constructed, the weights are randomly initialized, and the Adam optimizer and cross entropy loss function are used for training. During training, the input image can be uniformly scaled to 256×256, and data enhancement such as horizontal flipping, vertical flipping, contrast enhancement, noise addition, etc. can be performed to improve the generalization of the model. After data enhancement, the pixel value of the image is divided by 255 and normalized to the range of 0 to 1 as the input of the classifier. The training termination time is determined by monitoring the changes in the loss of the training set and the validation set, for example, when both losses no longer decrease, the training is stopped and the model file is saved. Then, the accuracy of the classifier model is evaluated on the test set. If it is not ideal, it is necessary to analyze the characteristics of the error samples and expand the training set in a targeted manner for iterative optimization of the model.

[0101] For example, the effective image may be an in vivo image acquired by the bronchus, and the in vitro image may be an image acquired by the bronchus outside the body cavity or other images that are not of interest in bronchoscopy.

[0102] Specifically, images collected when the bronchoscope is outside the body, such as when the endoscopic examination is over, the examination is suspended due to improper operation causing patient discomfort, or the examination is terminated due to an operational error during the examination, can be considered invalid images.

[0103] In some embodiments, the corresponding category of the highest confidence in the output node of the second pre-trained classifier can be selected as the classification result of the valid image or invalid image. When the process is actually running, the trained in vivo and in vitro recognition classification model (that is, the second pre-trained classifier mentioned above) can be loaded first, and the real-time bronchobronchial endoscopic examination images collected are preprocessed in the same way as the training stage, that is, scaled to 256×256 size, and the pixel value is divided by 255 to normalize to between 0 and 1, and then input into the classification model. The category with the highest confidence in the output node is used as the valid or invalid classification result of the current image.

[0104] At step S606, in response to identifying that the current bronchoscopic examination image is a valid image, a timer for obtaining the operation time is started for cumulative timing. An additional timer is required here to accumulate the valid time. Specifically, when the previous frame is an invalid image and the current frame is a valid image, the timer is turned on and starts counting; when the previous frame is a valid image and the current frame is an invalid image or the case is ended, the timer is paused.

[0105] If the current bronchoscopic examination image is determined to be an invalid image or the examination has been stopped, step S606 is executed, that is, the timing is paused and the accumulation of effective operation time is stopped.

[0106] Finally, when the entire case (i.e., the bronchoscopic examination of the user or patient) is completed, the accumulated time of the timer is recorded as the effective time of this bronchoscopic operation. This effective time can be used as one of the important indicators to measure the standardization and integrity of the endoscopic operation. At the same time, when the bronchoscopic examination is not completed and is in the inspection state for a long time, but the effective operation time is not accumulated, it will also indirectly remind the doctor to adjust the operation in time.

[0107] In summary, this embodiment Figure 6 The method flow shown essentially provides a method for automatically identifying the position of the endoscope based on the image content, and on this basis, the statistics of the effective operation time of the endoscope are realized. This method can record the actual working time of the endoscope in the body in real time, exclude irrelevant operation time, objectively evaluate the rationality of the time investment in endoscopic examination, and provide a quantitative basis for standardizing the operation process. When this method is combined with the previously discussed functions such as bronchial anatomical structure positioning and mucosal abnormal area identification, intelligent microscopic examination of the entire process of bronchoscopy can be realized, which can help improve the diagnosis and treatment effect.

[0108] Figure 7 The flowchart of a method 700 for obtaining the real-time bronchial endoscopic status in response to a recognition result and calling a status request for obtaining the endoscopic status of a bronchoscope according to another embodiment of the present disclosure is schematically shown. It should be understood that the method 700 can be implemented by hardware, firmware, software, or a combination thereof. Figure 7 As shown, the method 700 includes the following steps:

[0109] In step S702, the third pre-trained classifier is used to classify the bronchoscopic examination image by clarity to obtain a blurred image in the clarity classification result. The third pre-trained classifier is used to at least determine whether the bronchoscopic examination image is a blurred image based on the input image. The image is input into the third pre-trained clear and blurred classifier, which is also implemented using a lightweight convolutional neural network such as RepVGG, MobileNet, ShuffleNet, etc. The number of output nodes can be 2, corresponding to the two categories of clear image and blurred image respectively.

[0110] The training process of the third pre-trained clear and fuzzy classifier is as follows: First, a large number of bronchoendoscopic images are collected and divided into clear and fuzzy categories by manual annotation. Of course, it can also be classified into discrete levels of clarity based on resolution, signal-to-noise ratio, color fidelity, etc. All samples are divided into training set, validation set and test set in a ratio of 8:1:1, and the distribution of clear and fuzzy samples in each data set is ensured to be consistent. Then, the above lightweight classification model is constructed, the weights are randomly initialized, and the training is performed using the Adam optimizer and the cross entropy loss function. During training, the image is scaled to 256×256, and data enhancement such as horizontal flipping and vertical flipping are performed. After data enhancement, the image pixel value is normalized to the range of 0 to 1 by dividing it by 255 and then input into the network. By monitoring the changes in the loss of the training set and the validation set, it is decided when to terminate the training and save the model file. Then, the model accuracy is evaluated on the test set, and the characteristics of the error samples are analyzed when necessary, and the data set is expanded in a targeted manner to iteratively optimize the model.

[0111] In some embodiments, the images selected are those that are blurred with a resolution lower than a preset threshold, those that contain certain obstructions or foreign matter, those that capture lens contamination, insufficient light sources or damaged cameras, those that capture overlapping tissues, or those that are in vitro, those that are completed at the end of microscopy, those that suspend the examination due to patient discomfort caused by improper operation, and those that terminate the examination due to operational errors during the examination.

[0112] In some embodiments, a third pre-trained classifier is used to judge blurred images of bronchoscopic examination images and the effective operation time is not updated, so that the operational quality assessment of bronchoscopy is diversified and the reliability is improved. The operational quality assessment of the entire system retains a certain redundancy, thereby improving the robustness of the overall system.

[0113] In some embodiments, when judging a blurred image, a third pre-trained classifier can be directly used to judge whether the bronchoscopic examination image is a clear image or a blurred image; when system resources are sufficient, the second pre-trained classifier can be first used to judge whether the bronchoscopic examination image is a valid image, and then the third pre-trained classifier can be used to judge whether the bronchoscopic examination image is a clear image or a blurred image after it is judged as a valid image.

[0114] Next, in step S704, the proportion of blurred frames in the second predetermined number of consecutive image frames in the blurred image is obtained. The second predetermined number (e.g., 25 frames) of the second consecutive image frames of the bronchoendoscopic examination images are classified for clarity, preprocessed in the same manner as in the training stage (i.e., scaled to 256×256, pixel values ​​normalized to between 0 and 1 by dividing by 255), and input into the third pre-trained classifier. The category with the highest confidence of the output node is selected as the clarity and blur judgment result of the current frame.

[0115] In some embodiments, obtaining a ratio value of blurred frames in a predetermined number of second consecutive image frames in a blurred image includes: acquiring multiple second consecutive image frames in a current blurred image; performing clarity classification on the multiple second consecutive image frames through a third pre-trained classifier, and obtaining all blurred frames in the clarity classification results to obtain a ratio value.

[0116] In some embodiments, when the process of the method 700 is actually run, the second predetermined number (e.g., 25 frames) of second consecutive image frames of bronchoscopic examination images that are identified as valid images by the second pre-trained in vitro and in vivo classifier and are selected as valid images can be classified for clarity, pre-processed in the same manner as in the training stage (i.e., scaled to 256×256, pixel values ​​normalized to between 0 and 1 by dividing by 255), and input into the third pre-trained classifier. The category with the highest confidence of the output node is selected as the clarity and blur judgment result of the current frame.

[0117] Finally, in step S706, in response to the blurred image with a ratio value not less than the preset ratio being a blurred image, an adjustment request for adjusting the bronchoscope is issued. Specifically, in response to the image frames of a predetermined ratio (such as more than 13 frames, i.e., more than 50%) in the second predetermined number (such as 25 frames) being classified as blurred, it is determined that the current image is in a blurred state, and a prompt message for adjusting the bronchoscope is issued to remind the doctor that the focus, angle or position of the endoscope needs to be adjusted to obtain a clearer and more complete field of view, thereby improving the effectiveness of subsequent diagnosis and treatment.

[0118] In some embodiments, the clarity classification model may also use a second pre-trained classifier including a multi-level classification mode or a quantitative scoring method to perform clarity classification or clarity evaluation on the input image. The clarity / blur judgment result may be classified as "very clear", "clear", "slightly blurred", "blurred" and "very blurred". When it is judged to be "blurred" and / or "very blurred", an adjustment operation or a request for a clear shot is issued. Alternatively, the clarity / blur judgment result may be assigned a quantitative score, such as a value between 0 and 10, where 10 represents the highest clarity and 0 represents the worst clarity. The doctor may perform an adjustment operation based on the result of the quantitative score, or issue an adjustment request when the score is less than 5 points. Such clarity / blur judgment and status request may be more flexibly adapted to the needs of different scenarios, support accurate diagnosis and treatment decisions, and realize intelligent and reliable auxiliary diagnosis.

[0119] In summary, this embodiment combines Figure 7Essentially, it provides a method for automatically evaluating the clarity of the bronchoscopic field of view based on image content. The third pre-trained classifier is used to evaluate the quality of the field of view, and the proportion of clear and blurred images within a certain time window is used as the judgment standard to prompt the doctor whether it is necessary to adjust the endoscope to optimize the field of view during the examination. This method can reflect the imaging quality of the endoscopic field of view in real time, reduce blind operations during endoscopic examinations, provide objective operating guidance for doctors, and improve the diagnosis and treatment effects. Combining it with functions such as bronchial anatomical structure positioning, mucosal abnormal area identification, and microscopic examination time statistics, it can realize intelligent microscopic examination of the entire process of bronchoscopy, and significantly improve the level of bronchoscopy application.

[0120] In some embodiments, obtaining the real-time microscopic status of the bronchi also includes the following steps: determining a first display sub-style for indicating the real-time position of the bronchoscope and lighting up the inspected area based on the recognition results in the bronchoscopic examination image (such as airway structure / tracheal branches, etc.); and determining a corresponding second display sub-style such as the effective operation time of bronchoscopic examination, mucosal abnormality prompts, or field of view clarity (blurred image prompts) based on the request category of the status request to provide interactive feedback. Then, taking into account the first display sub-style and the second display sub-style, the final display style of the feedback graphic is determined and displayed in the interface area associated with each operation category. This method can be combined with the above. Figure 3-Figure 7 It combines the processing processes of bronchoscopic site recognition, mucosal abnormality recognition, endoscopic operation time statistics and visual field quality assessment to achieve multi-dimensional visual feedback during bronchoscopy, providing doctors with more intuitive and comprehensive auxiliary information to optimize operations.

[0121] In some embodiments, based on the characteristic elements in the recognition results (such as the anatomical structure features of the bronchial tree), at least the observation area of ​​the characteristic elements after emphasis processing is indicated, and the observation area includes the route that has been located or the area that has been located, so that the bronchoscopy route is emphasized and displayed. At the same time, the abnormal area associated with the abnormal data can be emphasized, so that the abnormal area is highlighted or flashing according to the abnormal lesion category, or appears in a specific color mark or the like.

[0122] The disclosed embodiment also provides a system 800 for assisting bronchoscopy. Figure 8As shown. The system can implement the bronchoscopic assisted microscopy scheme involved in various embodiments above. Specifically, the system 800 may include an image acquisition unit 802, an operation classification unit 804, an image recognition unit 806, a request unit 808 and a status feedback unit 810. Among them, the image acquisition unit 802 is responsible for real-time acquisition of image sequences during bronchoscopic examination to provide raw data for subsequent analysis; the operation classification unit 804 classifies the acquired bronchoscopic examination images according to the trachea, left and right main bronchi and smaller bronchial branches, such as upper and lower lobe bronchi and their segments and subsegmental bronchi. The image recognition unit 806 identifies and processes the specific parts of the bronchus to obtain the semantic understanding results of the image content, such as the specific part of the bronchus where the lens is located, the abnormal mucosal area, the image clarity recognition result, etc. The request unit 808 receives feedback from the recognition result and issues requests including emphasizing the processing of certain feature areas, starting statistical timing, sending abnormal images, requesting to issue adjustment prompts, etc. to obtain the bronchoscopic examination status.

[0123] Finally, the status feedback unit 810 obtains the status request from the doctor, and evaluates the reliability, integrity, and standardization of the current bronchial examination process according to the type of status request and the result given by the recognition unit. The real-time microscopic examination status of the bronchoscope, such as possible abnormal mucosa, is presented to the doctor in real time in an intuitive and friendly visual graphical manner, which plays a role in assisting optimization.

[0124] Through the collaboration of the above functional units, the bronchoscopic assisted microscopy system of this embodiment can track the entire endoscopic examination process, objectively record the operation behavior, quantitatively analyze the key microscopy indicators, and clearly present the feedback information to the doctor. The system provides intelligent monitoring and guidance throughout the bronchoscopic examination process, avoids the blindness of the operation, improves the doctor's standardized operation awareness and level, reduces omissions and misdiagnosis, and ultimately improves the overall effect of bronchoscopic diagnosis and treatment, and better serves patients.

[0125] In order to realize the above disclosure in the hardware and software level, Figure 1-7 The endoscopic assisted examination method described herein, the disclosed embodiment also provides an electronic device. Fig. 9 As shown, the electronic device 900 includes a processor 910 and a memory 920. The memory 920 stores executable program instructions, and when these instructions are executed by the processor 910, the electronic device can execute the method steps in the various embodiments described in conjunction with the drawings.

[0126] Specifically, the processor 910, as the core control and computing unit of the electronic device 900, is responsible for scheduling and executing the programs in the memory 920, coordinating the work of each hardware module, and performing calculations and analysis on the acquired bronchobronchial endoscopic examination image data. The processor 910 can be in the form of a general central processing unit (CPU), an application processor (AP), an intelligent processing unit (IPU) dedicated to artificial intelligence, etc., or it can be composed of a heterogeneous computing architecture composed of multiple processing units. The memory 920 is used to store the program code and data required for bronchoendoscopic assisted microscopy, including operating systems, applications, algorithm models, bronchoendoscopic examination images and recognition results. The memory 920 can use a variety of volatile or non-volatile storage media, such as DRAM, ROM, EEPROM, MRAM, SSD, etc. In addition, the electronic device 900 may also include other necessary functional components such as an interface circuit connected to the bronchoendoscopic device, a human-computer interaction module, and a network communication unit.

[0127] When the system is running, the image data collected by the endoscope device is transmitted to the electronic device 900 in real time through the interface circuit. After the processor 910 decodes and preprocesses the image, it is sent to the recognition model in the memory 920 to achieve semantic understanding of the image content, such as anatomical location, abnormal area detection, etc. The processor 910 evaluates the quality and integrity of the bronchial examination process based on the doctor's operating instructions and the recognition results to obtain quantitative status indicators. These status indicators are displayed to the doctor in real time after visualization, which plays a role in assisting in optimizing the endoscopic operation. During the whole process, the processor 910 obtains the program code in the memory 920, coordinates each module to perform tasks such as image acquisition, recognition, analysis, and display, and completes the execution of the bronchoscopic assisted microscopic examination method in the hardware system.

[0128] In summary, this embodiment uses a general computing device platform to realize the functions of the bronchoendoscopic assisted microscopy system through the collaboration of software and hardware. The electronic device integrates multiple technologies such as image processing, machine learning, and human-computer interaction, and can be used as a supporting device for bronchoendoscopic equipment, significantly improving the level of intelligence in the endoscopic examination process. Since it can be implemented through software, the disclosed embodiment also has good flexibility and scalability, and can continuously improve the performance of assisted microscopy by upgrading algorithm models and applications to meet changes in clinical needs.

[0129] In another embodiment, the bronchoendoscopic assisted microscopy method disclosed herein can also be provided in the form of a computer-readable storage medium. The storage medium stores program code, and when it is executed by the processor of an electronic device, the electronic device performs all the steps of the multiple embodiments described in conjunction with the accompanying drawings. These program codes can be written in any suitable high-level or low-level programming language, and can be carried and distributed through non-volatile storage media such as read-only memory (ROM), flash memory, disk, etc., for processors on various terminal devices to read and execute. This embodiment provides a more flexible and universal carrier form for the software implementation of the bronchoendoscopic assisted microscopy method and system.

[0130] Although multiple embodiments of the present disclosure have been shown and described herein, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Those skilled in the art may think of many changes, modifications, and alternatives without departing from the thought and spirit of the present disclosure. The appended claims are intended to define the scope of protection of the present disclosure, and therefore cover equivalents or alternatives within the scope of these claims.

Claims

1. A method for assisting bronchoscopy, characterized in that: include: Real-time acquisition of bronchoscopic images; operating a first pre-trained classifier to perform location classification on the bronchoscopic examination image, the bronchoscopic examination image including landmark identification information for identifying at least two lumens; In response to the bronchoscopic examination image after positioning and classification, obtaining a recognition result of the bronchoscopic examination image; In response to the recognition result, calling a status request for obtaining a microscopic examination status of the bronchoscope; In response to the status request, the real-time microscopic examination status of the bronchoscope is fed back.

2. The method according to claim 1, characterized in that In response to the classified bronchoscopic examination image, obtaining a recognition result of the bronchoscopic examination image includes: Obtain bronchial landmark identification information; utilizing the landmark identification information to obtain at least one lumen for determining the bronchoscopic image; In response to identifying the at least one lumen, triggering execution of a branching structure to the at least one lumen; Using a pre-trained feature fusion classification network, image features and sequence features in a plurality of first continuous image frames collected within a first predetermined time are fused to obtain fused image features; The fused image features are input into a preset multi-dimensional fully connected layer to obtain a classification result of the last frame of the plurality of first continuous image frames as a recognition result of the specific part.

3. The method according to claim 2, characterized in that The pre-trained feature fusion classification network is used to perform feature fusion on the image features and sequence features collected within the first predetermined time. When obtaining the fused image features, the method includes: Performing feature extraction on the plurality of first continuous image frames to obtain image features; Compressing the input image features based on global averaging to obtain a multi-dimensional feature vector; Based on the activation function, the multidimensional feature vector is generated as an attention weight coefficient, and the attention weight coefficient is fused after element-by-element multiplication operation with each of the image features to obtain a fused image feature.

4. The method according to claim 1, characterized in that: In response to the recognition result, calling a status request for obtaining the microscopic examination status of the bronchoscope includes: Performing real-time detection on the bronchoscopic examination image based on a target detection model trained to determine abnormal images in the bronchoscopic examination image, and acquiring position coordinates corresponding to the abnormal images; In response to the abnormal area associated with the position coordinates, a status request for feeding back an abnormal condition of the abnormal area is sent.

5. The method according to claim 4, characterized in that Performing real-time detection on the bronchoscopic examination image based on a target detection model trained to determine an abnormal image in the bronchoscopic examination image, and obtaining position coordinates of the abnormal image includes: Performing preprocessing of scaling and pixel value normalization on the bronchoendoscopic examination image, and inputting the preprocessing into the target detection network to obtain a detection frame of the abnormal image; performing redundancy removal processing on the detection frame of the abnormal image to obtain a target detection frame; and The target detection frame is subjected to an inverse coordinate transformation to obtain the position coordinates of the abnormal image in the bronchial endoscopy image.

6. The method according to claim 1, characterized in that In response to the recognition result, calling a status request for obtaining the microscopic examination status of the bronchoscope includes: performing preprocessing including scaling and pixel value normalization on the bronchoendoscopy image; Using a second pre-trained classifier to perform validity classification on the pre-processed bronchoscopic examination image, so as to obtain a corresponding category of the highest confidence of the output node as a valid image in the validity classification result, wherein the second pre-trained classifier is used to at least determine whether the bronchoscopic examination image is a valid image based on the input image; In response to identifying the valid image, a timer for obtaining the operation time is started to accumulate time.

7. The method according to claim 1, characterized in that In response to the recognition result, calling a status request for obtaining the microscopic examination status of the bronchoscope includes: Using a third pre-trained classifier to perform clarity classification on the bronchoscopic examination image to obtain a blurred image in the clarity classification result, wherein the third pre-trained classifier is used to at least determine whether the bronchoscopic examination image is a blurred image based on an input image; Obtaining a ratio value of blurred frames in a predetermined number of second continuous image frames in the blurred image; In response to the blurred image having the ratio value not less than the preset ratio being a blurred image, an adjustment request for adjusting the bronchoscope is issued.

8. The method according to any one of claims 1 to 7, characterized in that: In response to the status request, feeding back the real-time microscopic examination status of the bronchoscope further includes: determining, according to the recognition result of the bronchoendoscopic examination image, a first display substyle of at least one feedback graphic, and displaying, according to a request category of a status request, a second display substyle of the at least one interactive feedback graphic; The display style of the at least one feedback graphic is determined according to the first display sub-style and the second display sub-style, and the at least one feedback graphic is displayed in an associated area of ​​the operation instruction based on the display style.

9. The method according to claim 8, characterized in that Determining, according to the recognition result of the bronchoendoscopic examination image, a first display sub-pattern of at least one feedback graphic comprises: Based on the feature element in the recognition result, at least an observation area of ​​the feature element after being processed by emphasis is indicated.

10. A system for assisting bronchoendoscopy, implementing the method according to any one of claims 1 to 9, the system comprising: An image acquisition unit, used for acquiring bronchoendoscopic examination images in real time; An operating classification unit is used to operate a first pre-trained classifier to perform positioning classification on the bronchoscopy image; an image recognition unit, configured to obtain a recognition result of the bronchoscopic examination image in response to performing positioning classification, wherein the bronchoscopic examination image includes at least two cavities; a request unit, in response to the recognition result, calling a status request for obtaining a microscopic examination status of the bronchoscope; The status feedback unit is configured to feedback the real-time microscopic examination status of the bronchoscope in response to the status request.

11. An electronic device, characterized in that: include: processor; as well as A memory storing executable program instructions, which, when executed by the processor, enables the device to implement the method according to any one of claims 1 to 9.

12. A computer-readable storage medium having executable program instructions stored thereon, wherein when the program instructions are executed by one or more processors, the method according to any one of claims 1 to 9 is implemented.

Images