Intelligent processing system and method for bronchoscopic images

Through the intelligent bronchoscopic image processing system, the foreground and background features of bronchoscopic images are extracted and processed using Yolo model and hollow convolutional neural network, which solves the problem of image noise interference and structural complexity, and achieves the improvement of image quality and lesion diagnosis. Auxiliary effect.

CN118781100BActive Publication Date: 2024-11-12JILIN UNIVERSITY
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202411245086.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-06
Publication Date
2024-11-12
Estimated Expiration
2044-09-06

AI Technical Summary

Technical Problem

Bronchoscopic images are susceptible to noise during the acquisition process, resulting in a decrease in image quality, which increases the difficulty of doctors to identify lesion areas, especially due to factors such as equipment jitter, tissue reflection and secretion occlusion, as well as the complexity of bronchial structure and branch morphology variability.

Method used

The bronchoscopic image intelligent processing system is used to perform foreground detection through Yolo model, extract the foreground part of the bronchoscopic image, and combine the hollow convolutional neural network to extract the foreground and background features, and use the significant features of attention-enhanced attention to combine with the perception network for feature interaction and Weighted processing generates enhanced bronchoscopic images, removes noise interference, and highlights the main and branching characteristics of the bronchial duct.

Benefits of technology

It significantly improves image quality, enhances visibility of main branches and branches of the bronchial tract, reduces noise interference, and provides doctors with more accurate image information and auxiliary diagnostic basis.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118781100B_ABST
    Figure CN118781100B_ABST
Patent Text Reader

Abstract

The present application provides a bronchoscopic image intelligent processing system and method, which relates to the field of image intelligent processing. The system acquires bronchoscopic images and processes the bronchoscopic images through an image processing and analysis algorithm based on artificial intelligence and machine learning, thereby capturing foreground and background feature information in the bronchoscopic images, and performing joint perception of the foreground features and background features in the bronchoscopic images, thereby highlighting and enhancing the main branches and branch features of the bronchus in the image, and removing non-bronchial cluttered noise interference in the bronchoscopic images, providing auxiliary basis for doctors to interpret the images and diagnose bronchial lesions.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of intelligent image processing, and specifically to a bronchoscopic image intelligent processing system and method. Background Art

[0002] Bronchoscopy is a common clinical diagnostic method used to observe the internal structure of the respiratory tract, especially the condition of the bronchi, in order to facilitate early detection and diagnosis of various respiratory diseases, such as inflammation and tumors. However, bronchoscopic images are often affected by a variety of factors during the acquisition process, which can cause noise, such as equipment jitter, tissue reflection, and secretion occlusion, which can seriously affect the doctor's interpretation of the image and identification of the lesion area. In addition, the bronchial structure is complex and changeable, and the bronchial branches at different levels have different shapes. These bronchial branches will blend with the background information, which also increases the difficulty of analysis and identification for doctors.

[0003] Therefore, an intelligent bronchoscopic image processing system is desired. Summary of the invention

[0004] This application is made in consideration of the above problems. One object of this application is to provide a bronchoscopic image intelligent processing system and method.

[0005] The present application provides a bronchoscopic image intelligent processing system, which includes:

[0006] A bronchoscope image acquisition module, used for acquiring bronchoscope images to be processed;

[0007] A bronchoscope image foreground part extraction module, used for inputting the bronchoscope image to be processed into a foreground detector based on a Yolo model to obtain a bronchoscope image foreground part;

[0008] A bronchoscope image background portion determination module, configured to determine a bronchoscope image background portion in the bronchoscope image to be processed based on the bronchoscope image foreground portion;

[0009] An image foreground and background feature extraction module, used for extracting features from the foreground portion of the bronchoscope image and the background portion of the bronchoscope image to obtain a bronchial foreground feature map and a bronchial background feature map respectively;

[0010] A bronchial foreground-background feature joint perception enhancement module, used for inputting the bronchial foreground feature map and the bronchial background feature map into an attention-enhanced salient feature joint perception network to obtain a bronchial foreground-background joint perception enhancement feature map;

[0011] A bronchoscope image enhancement module, used for generating an enhanced bronchoscope image based on the bronchial foreground-background joint perception enhancement feature map;

[0012] The bronchial foreground-background feature joint perception enhancement module includes:

[0013] a first shape reshaping unit, configured to perform feature shape reshaping on the bronchial foreground feature map and the bronchial background feature map to obtain a bronchial foreground feature shape reshaping matrix and a bronchial background feature shape reshaping matrix;

[0014] An interactive unit, used for inputting the bronchial foreground feature shape reshaping matrix and the bronchial background feature shape reshaping matrix into a feature channel-by-channel interactive perception module to obtain a bronchial foreground-background feature dependency matrix and a bronchial background-foreground feature dependency matrix;

[0015] A random dropout unit, used for inputting the bronchial foreground-background feature dependency matrix and the bronchial background-foreground feature dependency matrix into a random dropout module to obtain a pruned optimized bronchial foreground-background feature dependency matrix and a pruned optimized bronchial background-foreground feature dependency matrix;

[0016] A first matrix multiplication unit is used to multiply the pruned optimized bronchial foreground-background feature dependency matrix with the bronchial background feature shape reshaping matrix to obtain a dependency optimized bronchial background-foreground feature dependency matrix;

[0017] A second matrix multiplication unit is used to multiply the pruned optimized bronchial background-foreground feature dependency matrix with the bronchial foreground feature shape reshaping matrix to obtain a dependency optimized bronchial foreground-background feature dependency matrix;

[0018] A second shape reshaping unit is used to perform feature shape reshaping on the dependency-optimized bronchial background-foreground feature dependency matrix and the dependency-optimized bronchial foreground-background feature dependency matrix to obtain an optimized bronchial foreground feature map and an optimized bronchial background feature map;

[0019] A weighting unit is used to calculate the weighted sum of the optimized bronchial foreground feature map and the optimized bronchial background feature map to obtain the bronchial foreground-background joint perception enhancement feature map.

[0020] For example, according to the bronchoscopic image intelligent processing system of the present application, the image foreground and background feature extraction module is used to:

[0021] Passing the foreground part of the bronchoscope image through a foreground feature extractor based on a first deep neural network model to obtain the bronchial foreground feature map;

[0022] The background portion of the bronchoscope image is passed through a background feature extractor based on a second deep neural network model to obtain the bronchial background feature map.

[0023] For example, according to the bronchoscopic image intelligent processing system of the present application, the foreground feature extractor based on the first deep neural network model is a foreground feature extractor based on the first void convolutional neural network model, and the background feature extractor based on the second deep neural network model is a background feature extractor based on the second void convolutional neural network model.

[0024] For example, according to the bronchoscopic image intelligent processing system of the present application, the interaction unit includes:

[0025] A first matrix multiplication subunit is used for performing matrix multiplication on the transposed matrix of the bronchial foreground feature shape reshaping matrix and the bronchial background feature shape reshaping matrix, and then dividing the matrix by the scale of the bronchial foreground-background feature dependency matrix by position to obtain a bronchial foreground-background feature interaction matrix;

[0026] a second matrix multiplication subunit, configured to multiply the transposed matrix of the bronchial foreground feature shape reshaping matrix by the bronchial background feature shape reshaping matrix and then divide the result by the scale of the bronchial background feature shape reshaping matrix by position to obtain a bronchial background-foreground feature interaction matrix;

[0027] The normalization processing subunit is used to normalize the bronchial foreground-background feature interaction matrix and the bronchial background-foreground feature interaction matrix respectively to obtain the bronchial foreground-background feature dependency matrix and the bronchial background-foreground feature dependency matrix.

[0028] For example, according to the bronchoscopic image intelligent processing system of the present application, the normalization processing subunit is used to: use the Softmax function to perform soft maximum normalization processing on the bronchial foreground-background feature interaction matrix and the bronchial background-foreground feature interaction matrix to obtain the bronchial foreground-background feature dependency matrix and the bronchial background-foreground feature dependency matrix.

[0029] For example, according to the bronchoscopic image intelligent processing system of the present application, the bronchoscopic image enhancement module is used to: input the bronchial foreground-background joint perception enhancement feature map into the diffusion model-based image optimizer to obtain the enhanced bronchoscopic image.

[0030] The present application also provides a bronchoscopic image intelligent processing method, which includes:

[0031] Obtaining bronchoscopic images to be processed;

[0032] Inputting the to-be-processed bronchoscopic image into a foreground detector based on a Yolo model to obtain a foreground portion of the bronchoscopic image;

[0033] Based on the foreground portion of the bronchoscope image, determining a background portion of the bronchoscope image in the bronchoscope image to be processed;

[0034] Performing feature extraction on the foreground part of the bronchoscope image and the background part of the bronchoscope image respectively to obtain a bronchial foreground feature map and a bronchial background feature map;

[0035] Inputting the bronchial foreground feature map and the bronchial background feature map into an attention-enhanced salient feature joint perception network to obtain a bronchial foreground-background joint perception-enhanced feature map;

[0036] generating an enhanced bronchoscopic image based on the bronchial foreground-background joint perception enhancement feature map;

[0037] The bronchial foreground feature map and the bronchial background feature map are input into the attention-enhanced salient feature joint perception network to obtain a bronchial foreground-background joint perception enhancement feature map, including:

[0038] Performing feature shape reshaping on the bronchial foreground feature map and the bronchial background feature map to obtain a bronchial foreground feature shape reshaping matrix and a bronchial background feature shape reshaping matrix;

[0039] Inputting the bronchial foreground feature shape reshaping matrix and the bronchial background feature shape reshaping matrix into a feature channel-by-channel interactive perception module to obtain a bronchial foreground-background feature dependency matrix and a bronchial background-foreground feature dependency matrix;

[0040] Inputting the bronchial foreground-background feature dependency matrix and the bronchial background-foreground feature dependency matrix into a random dropout module to obtain a pruned optimized bronchial foreground-background feature dependency matrix and a pruned optimized bronchial background-foreground feature dependency matrix;

[0041] multiplying the pruned optimized bronchial foreground-background feature dependency matrix with the bronchial background feature shape reshaping matrix to obtain a dependency optimized bronchial background-foreground feature dependency matrix;

[0042] multiplying the pruned optimized bronchial background-foreground feature dependency matrix with the bronchial foreground feature shape reshaping matrix to obtain a dependency optimized bronchial foreground-background feature dependency matrix;

[0043] Performing feature shape reshaping on the dependency-optimized bronchial background-foreground feature dependency matrix and the dependency-optimized bronchial foreground-background feature dependency matrix to obtain an optimized bronchial foreground feature map and an optimized bronchial background feature map;

[0044] The weighted sum of the optimized bronchial foreground feature map and the optimized bronchial background feature map is calculated to obtain the bronchial foreground-background joint perception enhancement feature map.

[0045] For example, according to the bronchoscopic image intelligent processing method of the present application, wherein the feature extraction is performed on the foreground part of the bronchoscopic image and the background part of the bronchoscopic image to obtain a bronchial foreground feature map and a bronchial background feature map, respectively, including:

[0046] Passing the foreground part of the bronchoscope image through a foreground feature extractor based on a first deep neural network model to obtain the bronchial foreground feature map;

[0047] The background portion of the bronchoscope image is passed through a background feature extractor based on a second deep neural network model to obtain the bronchial background feature map.

[0048] For example, according to the intelligent processing method of bronchoscopic images of the present application, the foreground feature extractor based on the first deep neural network model is a foreground feature extractor based on the first hole convolutional neural network model, and the background feature extractor based on the second deep neural network model is a background feature extractor based on the second hole convolutional neural network model.

[0049] According to the bronchoscopic image intelligent processing system and method of the present application, bronchoscopic images are collected and processed through image processing and analysis algorithms based on artificial intelligence and machine learning, so as to capture the foreground and background feature information in the bronchoscopic images, and jointly perceive the foreground features and background features in the bronchoscopic images, thereby highlighting and enhancing the main bronchial branches and branch features in the image, and removing non-bronchial cluttered noise interference in the bronchoscopic images, providing auxiliary basis for doctors to interpret images and diagnose bronchial lesions. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings of the embodiments of the present application are briefly introduced below. Obviously, the drawings described below only relate to some embodiments of the present application, and are not intended to limit the present application.

[0051] Figure 1 A schematic diagram of the application architecture of the bronchoscopic image intelligent processing system in an embodiment of the present application is shown;

[0052] Figure 2 A schematic diagram of the structure of the intelligent bronchoscopic image processing system in an embodiment of the present application is shown;

[0053] Figure 3 A schematic diagram of the structure of a bronchial foreground-background feature joint perception enhancement module of a bronchoscope image intelligent processing system in an embodiment of the present application is shown;

[0054] Figure 4 A flow chart of the intelligent processing method for bronchoscopic images in an embodiment of the present application is shown;

[0055] Figure 5 The application scenario diagram of the bronchoscopic image intelligent processing system in the embodiment of the present application is shown. DETAILED DESCRIPTION

[0056] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without creative work also fall within the scope of protection of the present application.

[0057] The terms used in this specification are those common terms currently widely used in the art in consideration of the functions of the present application, but these terms may vary according to the intentions of those skilled in the art, precedents, or new technologies in the art. In addition, specific terms may be selected, and in this case, their detailed meanings will be described in the detailed description of the present application. Therefore, the terms used in the specification should not be understood as simple names, but rather as a general description based on the meaning of the terms and the present application.

[0058] Although the present application makes various references to certain modules in the system according to the embodiments of the present application, any number of different modules can be used and run on the user terminal and / or server. The modules are only illustrative, and different aspects of the system and method can use different modules.

[0059] Flowcharts are used in the present application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed accurately in order. On the contrary, various steps may be processed in reverse order or simultaneously as required. At the same time, other operations may also be added to these processes, or a certain step or several steps of operations may be removed from these processes.

[0060] Figure 1 A schematic diagram of the application architecture of the bronchoscopic image intelligent processing system in an embodiment of the present application is shown, including a server 100 and a terminal device 200.

[0061] The terminal device 200 and the server 100 can be connected via the Internet to achieve mutual communication. Optionally, the above-mentioned Internet uses standard communication technology and / or protocol. The Internet is usually the Internet, but it can also be any network, including but not limited to any combination of a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a private network or a virtual private network. In some embodiments, the data exchanged through the network can be represented by technologies and / or formats including Hyper Text Markup Language (HTML), Extensible Markup Language (XML), etc. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPsec), etc. can be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can also be used to replace or supplement the above-mentioned data communication technologies.

[0062] The server 100 can provide various network services for the terminal device 200, wherein the server 100 can be a single server, a server cluster consisting of several servers, or a cloud computing center. Specifically, the server 100 may include a processor 110 (Center Processing Unit, CPU), a memory 120, an input device 130, and an output device 140, etc. The input device 130 may include a keyboard, a mouse, a touch screen, etc., and the output device 140 may include a display device, such as a liquid crystal display (Liquid Crystal Display, LCD), a cathode ray tube (Cathode Ray Tube, CRT), etc.

[0063] The memory 120 may include a read-only memory (ROM) and a random access memory (RAM), and provides the processor 110 with program instructions and data stored in the memory 120. In the embodiment of the present application, the memory 120 may be used to store the corresponding method of the bronchoscopic image intelligent processing system in the embodiment of the present application.

[0064] The processor 110 calls the program instructions stored in the memory 120, and the processor 110 is used to execute the steps of any one of the bronchoscopic image intelligent processing methods in the embodiments of the present application according to the obtained program instructions.

[0065] In addition, the application architecture diagram in the embodiment of the present application is intended to more clearly illustrate the technical solution in the embodiment of the present application, and does not constitute a limitation on the technical solution provided in the embodiment of the present application. Of course, for other application architectures and business applications, the technical solution provided in the embodiment of the present application is also applicable to similar problems.

[0066] The following is a non-restrictive description of the bronchoscopic image intelligent processing system provided according to at least one embodiment of the present application through several examples or embodiments. As described below, different features in these specific examples or embodiments can be combined with each other without conflicting with each other to obtain new examples or embodiments, and these new examples or embodiments also fall within the scope of protection of the present application.

[0067] In response to the above technical problems, the technical concept of the present application is to collect bronchoscopic images and process the bronchoscopic images through image processing and analysis algorithms based on artificial intelligence and machine learning, so as to capture the foreground and background feature information in the bronchoscopic images, and jointly perceive the foreground features and background features in the bronchoscopic images, so as to highlight and enhance the main bronchial branches and branch features in the image, and remove non-bronchial cluttered noise interference in the bronchoscopic images, thereby providing auxiliary basis for doctors to interpret the images and diagnose bronchial lesions.

[0068] Figure 2The structure diagram of the bronchoscopic image intelligent processing system 800 in the embodiment of the present application is shown. The bronchoscopic image intelligent processing system 800 includes: a bronchoscopic image acquisition module 810, used to obtain a bronchoscopic image to be processed; a bronchoscopic image foreground part extraction module 820, used to input the bronchoscopic image to be processed into a foreground detector based on the Yolo model to obtain a bronchoscopic image foreground part; a bronchoscopic image background part determination module 830, used to determine the bronchoscopic image background part in the bronchoscopic image to be processed based on the bronchoscopic image foreground part; an image foreground and background feature extraction module 840, used to extract the bronchoscopic image background part in the bronchoscopic image to be processed; The module 850 is used to extract features from the foreground part of the bronchoscope image and the background part of the bronchoscope image respectively to obtain a bronchial foreground feature map and a bronchial background feature map; the module 850 is used to input the bronchial foreground feature map and the bronchial background feature map into an attention-enhanced salient feature joint perception network to obtain a bronchial foreground-background joint perception enhancement feature map; the module 860 is used to generate an enhanced bronchoscope image based on the bronchial foreground-background joint perception enhancement feature map.

[0069] Specifically, in the technical solution of the present application, first, the bronchoscopic image to be processed is obtained. Next, it is considered that the bronchoscopic image to be processed often contains many elements that are not related to the diagnosis, such as instruments, secretions, bubbles, etc. These elements not only increase the complexity of image processing, but may also interfere with the doctor's judgment. Therefore, in the technical solution of the present application, the bronchoscopic image to be processed is input into a foreground detector based on the Yolo model to obtain the foreground part of the bronchoscope image. It should be understood that the Yolo model is an efficient object target detection model that can quickly identify and locate multiple objects in an image. For bronchoscopic images, the foreground usually refers to the bronchial structure itself, including main branches and branches. By using the Yolo model, these structures can be located quickly and accurately, and the foreground part of the bronchoscope image can be highlighted, which is crucial for subsequent feature extraction and enhancement.

[0070] Furthermore, based on the foreground portion of the bronchoscope image, the background portion of the bronchoscope image in the bronchoscope image to be processed is determined. It should be understood that the background portion of the bronchoscope image usually contains more noise information and redundant information that is irrelevant to the subsequent image enhancement task. After determining the foreground, by processing the background semantics in a targeted manner, these noises can be effectively removed to further improve the image quality. In addition, while enhancing the foreground, reasonable processing of the background can help retain useful details in the image, such as the main branches and branch features of the bronchi, and avoid the loss of important information.

[0071] Then, the foreground part of the bronchoscope image is subjected to feature mining through a foreground feature extractor based on the first atrous convolutional neural network model to extract implicit feature information of the foreground part of the bronchoscope image, including the bronchial structure, such as key detail features of the main branches and branches, thereby obtaining a bronchial foreground feature map. In addition, the background part of the bronchoscope image is subjected to a background feature extractor based on the second atrous convolutional neural network model to obtain a bronchial background feature map. It is worth mentioning that the extraction of background features helps to retain other important information that may exist in the image, such as vascular distribution and position information. Although this information is not a direct basis for bronchial diagnosis, it is helpful for perceiving and enhancing the foreground features of the bronchi, and can further reduce the impact of noise on the final image quality.

[0072] Accordingly, the image foreground and background feature extraction module 840 is used to: pass the foreground part of the bronchoscope image through a foreground feature extractor based on a first deep neural network model to obtain the bronchial foreground feature map; pass the background part of the bronchoscope image through a background feature extractor based on a second deep neural network model to obtain the bronchial background feature map. Wherein, the foreground feature extractor based on the first deep neural network model is a foreground feature extractor based on a first atrous convolutional neural network model, and the background feature extractor based on the second deep neural network model is a background feature extractor based on a second atrous convolutional neural network model.

[0073] It should be understood that the bronchial foreground feature map and the bronchial background feature map respectively contain the foreground semantic features and background feature information about the bronchoscope image. In order to be able to jointly perceive the foreground features and image background semantic features about the bronchus in the image, so as to highlight the main branches and branch features of the bronchus, and at the same time suppress or remove the interference of non-bronchial cluttered noise, and provide a more accurate auxiliary basis for doctors to interpret the image and diagnose bronchial lesions, in the technical solution of the present application, the bronchial foreground feature map and the bronchial background feature map are further input into the attention-enhanced salient feature joint perception network to obtain the bronchial foreground-background joint perception enhancement feature map. It should be understood that the attention-enhanced salient feature joint perception network is a feature fusion technology, which enhances the interaction between the bronchial foreground feature map and the bronchial background feature map through multiple steps, and finally obtains an associated semantic and interactive information that combines the advantages of both, that is, the bronchial foreground-background joint perception enhancement feature map.

[0074] Specifically, first, the feature matrix along the channel dimension in the input bronchial foreground feature map and the bronchial background feature map is expanded into a feature vector to facilitate the subsequent channel-by-channel interactive perception operation. Next, the dependency between the two sets of feature vectors, the bronchial foreground feature and the background feature, is calculated to obtain the bronchial foreground-background feature and the bronchial background-foreground feature dependency matrix. Then, the dependency matrix is ​​sparsely processed by pruning operation to enhance the generalization ability of the model. Subsequently, the pruned dependency matrix is ​​used to weight the original features to strengthen the feature expression related to the structure of the bronchus (such as the main branch and the branch) and the features of the most relevant parts of the image enhancement task, while suppressing unimportant features. Finally, the optimized feature matrix is ​​reshaped into a feature map form, and the weighted sum is calculated to obtain the bronchial foreground-background joint perception enhancement feature map. In this way, the bronchial structure semantics in the bronchial foreground feature map and the background semantic information in the background feature map can be fully utilized to form a more comprehensive and rich feature representation. In addition, the attention-enhanced salient feature joint perception network can strengthen the key information in the bronchial foreground feature map and the bronchial background feature map through the dependency matrix, while removing redundant parts to further improve the feature quality. Through such a processing flow, the visibility of the main branches and branches of the bronchus can be significantly enhanced, while effectively removing the interference of non-bronchial cluttered noise. For doctors, this means that they can obtain clearer and more accurate image information, providing a reliable auxiliary basis for the diagnosis of bronchial lesions.

[0075] Accordingly, if Figure 3As shown, the bronchial foreground-background feature joint perception enhancement module 850 includes: a first shape reshaping unit 851, used to perform feature shape reshaping on the bronchial foreground feature map and the bronchial background feature map to obtain a bronchial foreground feature shape reshaping matrix and a bronchial background feature shape reshaping matrix; an interaction unit 852, used to input the bronchial foreground feature shape reshaping matrix and the bronchial background feature shape reshaping matrix into a feature channel-by-channel interactive perception module to obtain a bronchial foreground-background feature dependency matrix and a bronchial background-foreground feature dependency matrix; a random inactivation unit 853, used to input the bronchial foreground-background feature dependency matrix and the bronchial background-foreground feature dependency matrix into a random inactivation module to obtain a pruned optimized bronchial foreground-background feature dependency matrix and a pruned optimized bronchial background-foreground feature dependency matrix; a first matrix multiplication unit 854 , used for multiplying the pruned optimized bronchial foreground-background feature dependency matrix with the bronchial background feature shape reshaping matrix to obtain a dependency optimized bronchial background-foreground feature dependency matrix; a second matrix multiplication unit 855, used for multiplying the pruned optimized bronchial background-foreground feature dependency matrix with the bronchial foreground feature shape reshaping matrix to obtain a dependency optimized bronchial foreground-background feature dependency matrix; a second shape reshaping unit 856, used for performing feature shape reshaping on the dependency optimized bronchial background-foreground feature dependency matrix and the dependency optimized bronchial foreground-background feature dependency matrix to obtain an optimized bronchial foreground feature map and an optimized bronchial background feature map; a weighting unit 857, used for calculating the weighted sum of the optimized bronchial foreground feature map and the optimized bronchial background feature map to obtain the bronchial foreground-background joint perception enhancement feature map.

[0076] Among them, the interaction unit 852 includes: a first matrix multiplication subunit, which is used to perform matrix multiplication on the transposed matrix of the bronchial foreground feature shape reshaping matrix and the transposed matrix of the bronchial background feature shape reshaping matrix, and then divide the matrix by the scale of the bronchial foreground-background feature dependency matrix by position to obtain a bronchial foreground-background feature interaction matrix; a second matrix multiplication subunit, which is used to perform matrix multiplication on the transposed matrix of the bronchial foreground feature shape reshaping matrix and the bronchial background feature shape reshaping matrix, and then divide the matrix by the scale of the bronchial background feature shape reshaping matrix by position to obtain a bronchial background-foreground feature interaction matrix; a normalization processing subunit, which is used to perform normalization processing on the bronchial foreground-background feature interaction matrix and the bronchial background-foreground feature interaction matrix, respectively, to obtain the bronchial foreground-background feature dependency matrix and the bronchial background-foreground feature dependency matrix.

[0077] Specifically, the normalization processing subunit is used to: use the Softmax function to perform soft maximum normalization processing on the bronchial foreground-background feature interaction matrix and the bronchial background-foreground feature interaction matrix to obtain the bronchial foreground-background feature dependency matrix and the bronchial background-foreground feature dependency matrix.

[0078] In a specific example, the bronchial foreground-background feature joint perception enhancement module 850 is used to: input the bronchial foreground feature map and the bronchial background feature map into the attention-enhanced salient feature joint perception network and process them with the following salient feature joint perception formula to obtain the bronchial foreground-background joint perception enhancement feature map; wherein the salient feature joint perception formula is:

[0079] ;

[0080] ;

[0081] ;

[0082] ;

[0083] ;

[0084] ;

[0085] ;

[0086] ;

[0087] ;

[0088] ;

[0089] ;

[0090] in, is the bronchial foreground feature map, is the bronchial background feature map, represents the feature shape reshaping process, represents the transpose of a vector, and are the bronchial foreground feature shape reshaping matrix and the bronchial background feature shape reshaping matrix, is the scale of the bronchial foreground feature shape reshaping matrix and the bronchial background feature shape reshaping matrix, for function, is matrix multiplication, and are the bronchial foreground-background feature dependency matrix and the bronchial background-foreground feature dependency matrix, represents pruning optimization processing, and They are respectively the pruned optimized bronchial foreground-background feature dependency matrix and the pruned optimized bronchial background-foreground feature dependency matrix. and The dependency relationship optimizes the bronchial foreground-background feature dependency matrix and the dependency relationship optimizes the bronchial background-foreground feature dependency matrix, respectively. and They are respectively optimized bronchial foreground feature map and optimized bronchial background feature map. and are all weighted hyperparameters, It is the bronchial foreground-background joint perception enhancement feature map.

[0091] Furthermore, the bronchial foreground-background joint perception enhancement feature map is input into the diffusion model-based image optimizer to obtain an enhanced bronchoscopic image. In other words, the image optimization process is performed using the joint perception enhancement features between the foreground semantics and background semantics of the bronchi in the bronchoscopic image, so as to generate an enhanced bronchoscopic image through the diffusion model-based image optimizer. In this way, the main branches and branch features of the bronchi in the enhanced image can be highlighted, and the non-bronchial cluttered noise interference in the bronchoscopic image can be removed, providing auxiliary basis for doctors to interpret the image and diagnose bronchial lesions.

[0092] Correspondingly, the bronchoscopic image enhancement module 860 is used to: input the bronchial foreground-background joint perceptual enhancement feature map into the diffusion model-based image optimizer to obtain the enhanced bronchoscopic image.

[0093] In a preferred example, the bronchoscopic image enhancement module 860 includes: an optimization unit, used to optimize the bronchial foreground-background joint perceptual enhancement feature map to obtain an optimized bronchial foreground-background joint perceptual enhancement feature map; a diffusion unit, used to input the optimized bronchial foreground-background joint perceptual enhancement feature map into an image optimizer based on a diffusion model to obtain the enhanced bronchoscope image.

[0094] Wherein, the optimization unit is used to: cluster all feature values ​​of the bronchial foreground-background joint perceptual enhancement feature map based on the distance between feature values, such as the difference absolute value distance, and determine the ratio of the number of features of the cluster features to the number of features of the bronchial foreground-background joint perceptual enhancement feature map; arrange the cluster features into a bronchial foreground-background joint perceptual enhancement cluster vector, and divide the second norm of the bronchial foreground-background joint perceptual enhancement cluster vector by the second norm of the bronchial foreground-background joint perceptual enhancement feature vector obtained after the bronchial foreground-background joint perceptual enhancement feature map is expanded to obtain a first bronchial foreground-background joint perceptual enhancement clustering related weight value; divide a first power value of the one-norm of the bronchial foreground-background joint perceptual enhancement cluster vector with the ratio value as an exponent by a second power value of the one-norm of the bronchial foreground-background joint perceptual enhancement feature vector with the ratio value as an exponent to obtain a second bronchial foreground-background joint perceptual enhancement Cluster-related weight value; for each eigenvalue of the bronchial foreground-background joint perception enhancement clustering vector, multiply it by the inverse of the difference between the first bronchial foreground-background joint perception enhancement clustering-related weight value and the second bronchial foreground-background joint perception enhancement clustering-related weight value to obtain the optimized eigenvalue of the bronchial foreground-background joint perception enhancement clustering vector; for each eigenvalue outside the cluster in the bronchial foreground-background joint perception enhancement feature map, multiply it by the inverse of the sum of the first bronchial foreground-background joint perception enhancement clustering-related weight value and the second bronchial foreground-background joint perception enhancement clustering-related weight value to obtain the optimized out-of-class eigenvalue of the bronchial foreground-background joint perception enhancement feature map; the optimized eigenvalue of the bronchial foreground-background joint perception enhancement clustering vector and the optimized out-of-class eigenvalue of the bronchial foreground-background joint perception enhancement feature map are combined to form the optimized bronchial foreground-background joint perception enhancement feature map.

[0095] In a specific example, the optimization unit is used to optimize the bronchial foreground-background joint perception enhancement feature map using the following optimization formula to obtain the optimized bronchial foreground-background joint perception enhancement feature map; wherein the optimization formula is:

[0096] ;

[0097] ;

[0098] in, is the bronchial foreground-background joint perception enhancement feature vector, is the number of features of the bronchial foreground-background joint perception enhancement feature vector, is the bronchial foreground-background joint perception enhanced clustering vector, is the number of features of the bronchial foreground-background joint perception enhanced clustering vector, is the ratio value, represents the clustering feature set corresponding to the bronchial foreground-background joint perception enhancement clustering vector, and Represents the binary norm and the uninorm of the vector respectively. Power, is each feature value of the bronchial foreground-background joint perception enhancement feature map, is each feature value of the optimized bronchial foreground-background joint perceptual enhancement feature map.

[0099] That is, considering that the bronchial foreground feature map and the bronchial background feature map represent the image semantic features of the foreground part and the background part of the bronchoscope image respectively, when further performing the significant joint perception based on the attention mechanism, the significant joint perception aggregation alignment conflict based on the image semantic space attention weight will also be caused due to the difference in the spatial distribution of the image semantic features caused by the difference in the semantic space distribution of the source images, thereby causing the loss of aggregation key information and affecting the expression effect of the bronchial foreground-background joint perception enhancement feature map.

[0100] Therefore, in order to avoid the loss of key suffix semantic information relative to the original feature set as a whole due to aggregation conflict based on the aggregated features of the bronchial foreground-background joint perceptual enhancement feature map, the clustering ratio of the number of features of the bronchial foreground-background joint perceptual enhancement clustering vector relative to the number of features of the bronchial foreground-background joint perceptual enhancement feature map is used as a decision function to perform an adversarial decision on the set absolute representation of the bronchial foreground-background joint perceptual enhancement clustering vector and the norm of the bronchial foreground-background joint perceptual enhancement feature vector, and the clustering ratio is used with the bronchial foreground-background joint perceptual enhancement clustering vector and the set absolute representation of the norm of the bronchial foreground-background joint perceptual enhancement feature vector respectively. The clustering intrinsic conflict representation of the two norms of the bronchial foreground-background joint perception enhancement feature vector is positively and negatively interacted to construct the optimized bronchial foreground-background joint perception enhancement feature map based on the aggregated features and the solid alignment guardrail of the original feature set as a whole, so as to achieve the mitigation of the intention of harmful information loss based on the aggregated risk transferability of the optimized bronchial foreground-background joint perception enhancement feature map, and improve the expression effect of the optimized bronchial foreground-background joint perception enhancement feature map, so as to improve the image quality of the enhanced bronchoscope image obtained by inputting the bronchial foreground-background joint perception enhancement feature map into the image optimizer based on the diffusion model. In this way, a higher quality enhanced bronchoscope image can be obtained to highlight the main branches and branch information of the bronchus, and remove the noise interference in the bronchoscope image, providing an auxiliary basis for doctors to interpret the image and diagnose bronchial lesions.

[0101] Based on the above embodiments, see Figure 4 FIG. 1 is a flowchart of a bronchoscopic image intelligent processing method in an embodiment of the present application. For example, the bronchoscopic image intelligent processing method can be executed by a server, which can be Figure 1 The server 100 shown in FIG. Figure 4 As shown, the intelligent processing method of bronchoscopic images according to the embodiment of the present application includes the following steps: S510, obtaining a bronchoscopic image to be processed; S520, inputting the bronchoscopic image to be processed into a foreground detector based on a Yolo model to obtain a foreground part of the bronchoscopic image; S530, determining a bronchoscopic image background part in the bronchoscopic image to be processed based on the foreground part of the bronchoscopic image; S540, performing feature extraction on the foreground part of the bronchoscopic image and the background part of the bronchoscopic image respectively to obtain a bronchial foreground feature map and a bronchial background feature map; S550, inputting the bronchial foreground feature map and the bronchial background feature map into an attention-enhanced salient feature joint perception network to obtain a bronchial foreground-background joint perception enhancement feature map; S560, generating an enhanced bronchoscope image based on the bronchial foreground-background joint perception enhancement feature map.

[0102] In one example, feature extraction is performed on the foreground portion of the bronchoscope image and the background portion of the bronchoscope image to obtain a bronchial foreground feature map and a bronchial background feature map, respectively, including: passing the foreground portion of the bronchoscope image through a foreground feature extractor based on a first deep neural network model to obtain the bronchial foreground feature map; passing the background portion of the bronchoscope image through a background feature extractor based on a second deep neural network model to obtain the bronchial background feature map.

[0103] In one example, the foreground feature extractor based on the first deep neural network model is a foreground feature extractor based on a first hole convolutional neural network model, and the background feature extractor based on the second deep neural network model is a background feature extractor based on a second hole convolutional neural network model.

[0104] Here, those skilled in the art can understand that the specific operations of each step in the above-mentioned bronchoscopic image intelligent processing method have been referred to above. Figure 2 to Figure 3 The bronchoscopic image intelligent processing system 800 is described in detail, and therefore, its repeated description will be omitted.

[0105] Figure 5 FIG. 1 is an application scenario diagram of the bronchoscopic image intelligent processing system according to an embodiment of the present application. Figure 5 As shown, in this application scenario, first, a bronchoscopic image to be processed is obtained (for example, Figure 5 D), then, the bronchoscopic image to be processed is input to a server (for example, Figure 5 In S) as shown in , the server is capable of using the bronchoscopic image intelligent processing algorithm to process the bronchoscopic image to be processed to obtain an enhanced bronchoscopic image.

[0106] Based on the above embodiments, another exemplary embodiment of an electronic device is also provided in the embodiments of the present application. In some possible implementations, the electronic device in the embodiments of the present application may include a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor can implement the steps of the bronchoscopic image intelligent processing method in the above embodiments when executing the program.

[0107] For example, in the case of electronic equipment Figure 1 Taking the server 100 in the example as an example, the processor in the electronic device is the processor 110 in the server 100, and the memory in the electronic device is the memory 120 in the server 100.

[0108] The embodiment of the present application also provides a computer-readable storage medium, on which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, the bronchoscopic image intelligent processing method according to the embodiment of the present application described with reference to the above figures can be executed. The computer-readable storage medium includes, but is not limited to, for example, volatile memory and / or non-volatile memory. The volatile memory may, for example, include a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may, for example, include a read-only memory (ROM), a hard disk, a flash memory, etc.

[0109] The embodiment of the present application also provides a computer program product or a computer program, which includes computer executable instructions, and the computer executable instructions are stored in a computer readable storage medium. The processor of the computer device reads the computer executable instructions from the computer readable storage medium, and the processor executes the computer executable instructions, so that the computer device executes the bronchoscopic image intelligent processing method according to the embodiment of the present application.

[0110] Those skilled in the art will appreciate that the contents disclosed in this application may be subject to various variations and improvements. For example, the various devices or components described above may be implemented by hardware, or by software, firmware, or a combination of some or all of the three.

[0111] In addition, although the present application makes various references to certain units in the system according to embodiments of the present application, any number of different units can be used and run on the client and / or server. The units are only illustrative, and different aspects of the system and method can use different units.

[0112] Those skilled in the art will appreciate that all or part of the steps in the above method can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, such as a read-only memory, a disk or an optical disk. Optionally, all or part of the steps in the above embodiment can also be implemented using one or more integrated circuits. Accordingly, each module / unit in the above embodiment can be implemented in the form of hardware or in the form of software function modules. The present application is not limited to any particular form of combination of hardware and software.

[0113] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art to which this application belongs. It should also be understood that terms such as those defined in common dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an idealized or extremely formal sense, unless explicitly defined as such herein.

[0114] The above is an explanation of the present application and should not be considered as limiting thereof. Although several exemplary embodiments of the present application have been described, those skilled in the art will readily appreciate that many modifications may be made to the exemplary embodiments without departing from the novel teachings and advantages of the present application.

Claims

1. A bronchoscopic image intelligent processing system, It is characterized in that include: A bronchoscope image acquisition module, used for acquiring bronchoscope images to be processed; A bronchoscope image foreground part extraction module, used for inputting the bronchoscope image to be processed into a foreground detector based on a Yolo model to obtain a bronchoscope image foreground part; A bronchoscope image background portion determination module, configured to determine a bronchoscope image background portion in the bronchoscope image to be processed based on the bronchoscope image foreground portion; An image foreground and background feature extraction module, used for extracting features from the foreground portion of the bronchoscope image and the background portion of the bronchoscope image to obtain a bronchial foreground feature map and a bronchial background feature map respectively; A bronchial foreground-background feature joint perception enhancement module, used for inputting the bronchial foreground feature map and the bronchial background feature map into an attention-enhanced salient feature joint perception network to obtain a bronchial foreground-background joint perception enhancement feature map; A bronchoscope image enhancement module, used for generating an enhanced bronchoscope image based on the bronchial foreground-background joint perception enhancement feature map; The bronchial foreground-background feature joint perception enhancement module includes: a first shape reshaping unit, configured to perform feature shape reshaping on the bronchial foreground feature map and the bronchial background feature map to obtain a bronchial foreground feature shape reshaping matrix and a bronchial background feature shape reshaping matrix; An interactive unit, used for inputting the bronchial foreground feature shape reshaping matrix and the bronchial background feature shape reshaping matrix into a feature channel-by-channel interactive perception module to obtain a bronchial foreground-background feature dependency matrix and a bronchial background-foreground feature dependency matrix; A random dropout unit, used for inputting the bronchial foreground-background feature dependency matrix and the bronchial background-foreground feature dependency matrix into a random dropout module to obtain a pruned optimized bronchial foreground-background feature dependency matrix and a pruned optimized bronchial background-foreground feature dependency matrix; A first matrix multiplication unit is used to multiply the pruned optimized bronchial foreground-background feature dependency matrix with the bronchial background feature shape reshaping matrix to obtain a dependency optimized bronchial background-foreground feature dependency matrix; A second matrix multiplication unit is used to multiply the pruned optimized bronchial background-foreground feature dependency matrix with the bronchial foreground feature shape reshaping matrix to obtain a dependency optimized bronchial foreground-background feature dependency matrix; A second shape reshaping unit is used to perform feature shape reshaping on the dependency-optimized bronchial background-foreground feature dependency matrix and the dependency-optimized bronchial foreground-background feature dependency matrix to obtain an optimized bronchial foreground feature map and an optimized bronchial background feature map; A weighting unit is used to calculate the weighted sum of the optimized bronchial foreground feature map and the optimized bronchial background feature map to obtain the bronchial foreground-background joint perception enhancement feature map.

2. The bronchoscopic image intelligent processing system according to claim 1, It is characterized in that The image foreground and background feature extraction module is used to: Passing the foreground part of the bronchoscope image through a foreground feature extractor based on a first deep neural network model to obtain the bronchial foreground feature map; The background portion of the bronchoscope image is passed through a background feature extractor based on a second deep neural network model to obtain the bronchial background feature map.

3. The bronchoscopic image intelligent processing system according to claim 2, It is characterized in that The foreground feature extractor based on the first deep neural network model is a foreground feature extractor based on the first hole convolutional neural network model, and the background feature extractor based on the second deep neural network model is a background feature extractor based on the second hole convolutional neural network model.

4. The bronchoscopic image intelligent processing system according to claim 3, It is characterized in that The interaction unit comprises: A first matrix multiplication subunit is used for performing matrix multiplication on the transposed matrix of the bronchial foreground feature shape reshaping matrix and the bronchial background feature shape reshaping matrix, and then dividing the matrix by the scale of the bronchial foreground-background feature dependency matrix by position to obtain a bronchial foreground-background feature interaction matrix; a second matrix multiplication subunit, configured to multiply the transposed matrix of the bronchial foreground feature shape reshaping matrix by the bronchial background feature shape reshaping matrix and then divide the result by the scale of the bronchial background feature shape reshaping matrix by position to obtain a bronchial background-foreground feature interaction matrix; The normalization processing subunit is used to normalize the bronchial foreground-background feature interaction matrix and the bronchial background-foreground feature interaction matrix respectively to obtain the bronchial foreground-background feature dependency matrix and the bronchial background-foreground feature dependency matrix.

5. The bronchoscopic image intelligent processing system according to claim 4, It is characterized in that The normalization processing subunit is used to: use the Softmax function to perform soft maximum normalization processing on the bronchial foreground-background feature interaction matrix and the bronchial background-foreground feature interaction matrix to obtain the bronchial foreground-background feature dependency matrix and the bronchial background-foreground feature dependency matrix.

6. The bronchoscopic image intelligent processing system according to claim 5, It is characterized in that The bronchoscopic image enhancement module is used to: input the bronchial foreground-background joint perception enhancement feature map into an image optimizer based on a diffusion model to obtain the enhanced bronchoscopic image.

7. A bronchoscopic image intelligent processing method, It is characterized in that include: Obtaining bronchoscopic images to be processed; Inputting the to-be-processed bronchoscopic image into a foreground detector based on a Yolo model to obtain a foreground portion of the bronchoscopic image; Based on the foreground portion of the bronchoscope image, determining a background portion of the bronchoscope image in the bronchoscope image to be processed; Performing feature extraction on the foreground part of the bronchoscope image and the background part of the bronchoscope image respectively to obtain a bronchial foreground feature map and a bronchial background feature map; Inputting the bronchial foreground feature map and the bronchial background feature map into an attention-enhanced salient feature joint perception network to obtain a bronchial foreground-background joint perception-enhanced feature map; generating an enhanced bronchoscopic image based on the bronchial foreground-background joint perception enhancement feature map; The bronchial foreground feature map and the bronchial background feature map are input into the attention-enhanced salient feature joint perception network to obtain a bronchial foreground-background joint perception enhancement feature map, including: Performing feature shape reshaping on the bronchial foreground feature map and the bronchial background feature map to obtain a bronchial foreground feature shape reshaping matrix and a bronchial background feature shape reshaping matrix; Inputting the bronchial foreground feature shape reshaping matrix and the bronchial background feature shape reshaping matrix into a feature channel-by-channel interactive perception module to obtain a bronchial foreground-background feature dependency matrix and a bronchial background-foreground feature dependency matrix; Inputting the bronchial foreground-background feature dependency matrix and the bronchial background-foreground feature dependency matrix into a random dropout module to obtain a pruned optimized bronchial foreground-background feature dependency matrix and a pruned optimized bronchial background-foreground feature dependency matrix; multiplying the pruned optimized bronchial foreground-background feature dependency matrix with the bronchial background feature shape reshaping matrix to obtain a dependency optimized bronchial background-foreground feature dependency matrix; multiplying the pruned optimized bronchial background-foreground feature dependency matrix with the bronchial foreground feature shape reshaping matrix to obtain a dependency optimized bronchial foreground-background feature dependency matrix; Performing feature shape reshaping on the dependency-optimized bronchial background-foreground feature dependency matrix and the dependency-optimized bronchial foreground-background feature dependency matrix to obtain an optimized bronchial foreground feature map and an optimized bronchial background feature map; The weighted sum of the optimized bronchial foreground feature map and the optimized bronchial background feature map is calculated to obtain the bronchial foreground-background joint perception enhancement feature map.

8. The intelligent processing method for bronchoscopic images according to claim 7, It is characterized in that The foreground part of the bronchoscope image and the background part of the bronchoscope image are respectively subjected to feature extraction to obtain a bronchial foreground feature map and a bronchial background feature map, including: Passing the foreground part of the bronchoscope image through a foreground feature extractor based on a first deep neural network model to obtain the bronchial foreground feature map; The background portion of the bronchoscope image is passed through a background feature extractor based on a second deep neural network model to obtain the bronchial background feature map.

9. The intelligent processing method for bronchoscopic images according to claim 8, It is characterized in that The foreground feature extractor based on the first deep neural network model is a foreground feature extractor based on the first hole convolutional neural network model, and the background feature extractor based on the second deep neural network model is a background feature extractor based on the second hole convolutional neural network model.

Citation Information

Patent Citations

  • Image difference detection method and system and electronic equipment

    CN115810110A

  • Lesion segmentation method and system based on dual-decoder UNet and Transform

    CN117197156A

  • Bronchoscope position determination method and apparatus, system, device, and medium

    WO2023097944A1