Method and apparatus, system for identifying fibrous structures in organ images

Abstract

The application discloses a method and device and system for identifying fiber structure in an organ image. The method comprises: acquiring an image of a target region in a target organ, wherein the image is collected by a confocal endomicroscope; performing feature extraction on the image to obtain a feature map of fiber structure corresponding to the image; processing the feature map of the fiber structure by using an attention network to obtain an attention feature map corresponding to the feature map of the fiber structure, wherein the attention feature map is used for identifying a target fiber structure in the fiber structure; performing superposition processing on the attention feature map and the image to obtain an attention distribution heat map corresponding to the image; and identifying the type of the fiber structure in the image according to the attention distribution heat map. The application solves the technical problem that the current method for detecting lesions in the body is relatively complex, a large amount of time is consumed, and real-time observation of the pathological structure cannot be performed.

Description

Technical Field

[0001] This application relates to the field of image processing, and more specifically, to a method, apparatus, and system for identifying fibrous structures in organ images. Background Technology

[0002] In related technologies, one method for detecting lesions in the body is to use an endoscope to observe the lesions in the corresponding tissues or lumens. Simultaneously, biopsies and other methods can be used for pathological examination to clarify the diagnosis and determine the histological type. Currently, methods for detecting lesions in the body are complex, time-consuming, and cannot provide real-time observation of the lesion structure.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This application provides a method, apparatus, and system for identifying fibrous structures in organ structure diagrams, to at least solve the technical problems that current methods for detecting lesions in vivo are complex, require a lot of time, and cannot observe lesion structures in real time.

[0005] According to one aspect of the embodiments of this application, a method for identifying fibrous structures in an organ image is provided, comprising: acquiring an image of a target region in a target organ, wherein the image is acquired by a confocal endoscope; performing feature extraction on the image to obtain a feature map of the fibrous structure corresponding to the image; processing the feature map of the fibrous structure using an attention network to obtain an attention feature map corresponding to the feature map of the fibrous structure, wherein the attention feature map is used to identify the target fibrous structure in the fibrous structure; superimposing the attention feature map and the image to obtain an attention distribution heatmap corresponding to the image; and identifying the type of fibrous structure in the image based on the attention distribution heatmap.

[0006] Optionally, an attention network is used to process the feature map of the fiber structure to obtain the attention feature map corresponding to the feature map of the fiber structure. This includes: inputting the feature map of the fiber structure into a first type of convolutional layer and a second type of convolutional layer in the attention network to obtain a first type of output result and a second type of output result, wherein both the first type of output result and the second type of output result are matrices; performing matrix multiplication on the first type of output result and the second type of output result to obtain the attention feature map corresponding to the feature map of the fiber structure.

[0007] Optionally, matrix multiplication is performed on the first type of output results and the second type of output results to obtain the attention feature map corresponding to the feature map of the fiber structure, including: normalizing the first type of output results to obtain attention weights, wherein the attention weights are the importance of each pixel in the feature map of the fiber structure; and performing matrix multiplication on the attention weights and the second type of output results to obtain the attention feature map corresponding to the feature map of the fiber structure.

[0008] Optionally, the attention network is used to process the feature map of the fiber structure to obtain the attention feature map corresponding to the feature map of the fiber structure. This further includes: inputting the feature map of the fiber structure into three convolutional layers; performing matrix multiplication between the first output of the first convolutional layer and the second output of the second convolutional layer to obtain a first calculation result; normalizing the first calculation result to obtain attention weights, where the attention weights represent the importance of each pixel in the feature map of the fiber structure; and performing matrix multiplication between the third output of the third convolutional layer and the attention weights to obtain the attention feature map corresponding to the feature map of the fiber structure, where the first, second, and third outputs are all matrices.

[0009] Optionally, the attention network is used to process the feature map of the fiber structure to obtain the attention feature map corresponding to the feature map of the fiber structure. This further includes: inputting the feature map of the fiber structure into at least four convolutional layers; performing matrix multiplication between the first output of the first convolutional layer and the second output of the second convolutional layer to obtain the (n-2)th calculation result, where n is an integer greater than or equal to 3 and less than N, and N is the number of convolutional layers included in the at least four convolutional layers; performing matrix multiplication between the nth output of the nth convolutional layer and the (n-2)th calculation result to obtain the (n-1)th calculation result; normalizing the (n-1)th calculation result to obtain attention weights, where the attention weights represent the importance of each pixel in the feature map of the fiber structure; and performing matrix multiplication between the Nth output of the Nth convolutional layer and the attention weights to obtain the attention feature map corresponding to the feature map of the fiber structure, where the first output, second output, nth output, and Nth output are all matrices.

[0010] Optionally, the attention feature map and the image are overlaid to obtain the attention distribution heatmap corresponding to the image, including: upsampling the attention feature map to obtain the upsampled target attention feature map, wherein the size of the target attention feature map is the same as the size of the image of the target region; and overlaying the target attention feature map and the image to obtain the attention distribution heatmap.

[0011] Optionally, identifying the type of fiber structure in the image based on the attention distribution heatmap includes: comparing the pixel value of each pixel in the attention distribution heatmap with a preset threshold; determining the region formed by pixels with pixel values ​​greater than the preset threshold as a first region; determining the identification information of the first region, wherein the identification information includes: the area of ​​the first region, the number of first regions, and the distribution of the first regions in the attention distribution heatmap; and identifying the type of fiber structure in the image based on the identification information of the first region.

[0012] Optionally, identifying the type of fiber structure in the image based on the identification information of the first region includes: if the identification information of the first region meets preset conditions, determining that the fiber structure in the image is complete, wherein the preset conditions include at least one of the following: the area of ​​the first region is greater than a preset area, the number of the first regions is greater than a preset number, and the distribution of the first regions in the attention distribution heatmap is uniform.

[0013] According to another aspect of the embodiments of this application, an organ image processing method is also provided, comprising: acquiring an image of a target region in a target organ, wherein the image is acquired by a confocal endoscope; performing feature extraction on the image to obtain a feature map of the fibrous structure corresponding to the image; processing the feature map of the fibrous structure using an attention network to obtain an attention feature map corresponding to the feature map of the fibrous structure, wherein the attention feature map is used to identify the target fibrous structure in the fibrous structure; superimposing the attention feature map and the image to obtain an attention distribution heatmap corresponding to the image; and displaying the attention distribution heatmap.

[0014] According to another aspect of the embodiments of this application, an apparatus for identifying fibrous structures in an organ image is also provided, comprising: an acquisition module for acquiring an image of a target region in a target organ, wherein the image is acquired by a confocal endoscope; an extraction module for extracting features from the image to obtain a feature map of the fibrous structure corresponding to the image; a first processing module for processing the feature map of the fibrous structure using an attention network to obtain an attention feature map corresponding to the feature map of the fibrous structure, wherein the attention feature map is used to identify the target fibrous structure in the fibrous structure; a second processing module for superimposing the attention feature map and the image to obtain an attention distribution heatmap corresponding to the image; and an identification module for identifying the type of fibrous structure in the image based on the attention distribution heatmap.

[0015] According to another aspect of the embodiments of this application, a system for identifying fibrous structures in organ images is also provided, comprising: a confocal endoscope and a processor, wherein the confocal endoscope is used to acquire images of a target region in a target organ; and the processor is connected to the confocal endoscope and is used to execute the above-described method for identifying fibrous structures in organ images and the above-described method for processing organ images.

[0016] According to another aspect of the embodiments of this application, a non-volatile storage medium is also provided. The non-volatile storage medium includes a stored program, wherein, when the program is running, the device where the non-volatile storage medium is located executes the above-described method for identifying fibrous structures in organ images and the method for processing organ images.

[0017] According to another aspect of the embodiments of this application, an electronic device is also provided, including: a memory and a processor, the processor being configured to run a program stored in the memory, wherein the program, when running, executes the above-described method for identifying fibrous structures in organ images and the method for processing organ images.

[0018] In this embodiment, an image of a target region in a target organ is acquired using a confocal microscopy endoscope. Feature extraction is performed on the image to obtain a feature map of the corresponding fibrous structure. An attention network is used to process the feature map of the fibrous structure to obtain an attention feature map corresponding to the feature map of the fibrous structure, where the attention feature map is used to identify the target fibrous structure within the fibrous structure. The attention feature map and the image are superimposed to obtain an attention distribution heatmap corresponding to the image. By using confocal microscopy endoscope technology to acquire images of the target organ and employing a deep learning network model to identify the acquired images, the morphology of glands or cells in the target organ can be identified using confocal microscopy endoscope. This simplifies the detection method of lesions in vivo and allows for real-time observation of the diseased structure of the organ, thus solving the problems of current methods for detecting lesions in vivo being complex, time-consuming, and unable to provide real-time observation of diseased structures. Attached Figure Description

[0019] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0020] Figure 1a A hardware block diagram of a computer terminal (or mobile device) for implementing a method for recognizing fibrous structures in organ images is shown.

[0021] Figure 1b This is a flowchart of a method for identifying fibrous structures in an organ image according to an embodiment of this application;

[0022] Figure 2a This is a schematic diagram of a confocal endoscopic microscopy for identifying lung cancer structures according to an embodiment of this application;

[0023] Figure 2b This is a schematic diagram illustrating the processing of a feature map of an organ fibrous structure according to an embodiment of this application.

[0024] Figure 3 This is a flowchart of another method for processing organ images according to an embodiment of this application;

[0025] Figure 4 This is a structural block diagram of a device for identifying fibrous structures in organ images according to an embodiment of this application;

[0026] Figure 5 This is a structural block diagram of a system for identifying fibrous structures in organ images according to an embodiment of this application;

[0027] Figure 6 This is a schematic diagram illustrating an application of identifying fibrous structures in lung images according to an embodiment of this application. Detailed Implementation

[0028] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0029] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0030] First, some nouns or terms that appear in the description of the embodiments of this application shall be interpreted as follows:

[0031] Fibrous structure: Disorders or disappearances of fibrous structures can be used to characterize the presence of corresponding lesions. For example, abnormalities such as disordered or absent fibrous structures in lung tissue can indicate the condition of lung tissue lesions, including the presence of nodules or cancerous changes in lung tissue.

[0032] Confocal laser endomicroscopy (CLE) is an endoscope that incorporates a confocal microscope. By using confocal technology, CLE increases the magnification of the endoscope, allowing observation of fine mucosal or fibrous structures that are inaccessible with conventional endoscopes. During examination of lesions within the body, doctors can observe the morphology of glands, cells, and other tissues in real-time without the need for forceps or subsequent biopsies. CLE can be used for gastroscopy, colonoscopy, and bronchoscopy.

[0033] According to an embodiment of this application, a method for identifying fibrous structures in an organ structure diagram is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0034] The methods and embodiments provided in this application can be executed on mobile terminals, computer terminals, or similar computing devices. Figure 1a A hardware block diagram of a computer terminal (or mobile device) for implementing a method to identify fibrous structures in organ images is shown. Figure 1a As shown, the computer terminal 10 (or mobile device 10) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) (processor 102 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that the structure shown in FIG1 is merely illustrative and does not limit the structure of the aforementioned electronic device. For example, the computer terminal 10 may also include... Figure 1a The more or fewer components shown, or having the same Figure 1a The different configurations shown.

[0035] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0036] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the method for identifying fibrous structures in organ structure images in this embodiment of the application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the above-mentioned method for identifying fibrous structures in organ images. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0037] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0038] The display can be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0039] Figure 1b This is a flowchart of a method for identifying fibrous structures in an organ image according to an embodiment of this application, such as... Figure 1b As shown, the method includes the following steps:

[0040] Step S102: Obtain an image of the target region in the target organ, wherein the image is acquired by confocal microscopy.

[0041] As an optional embodiment, the target organ can be the lung, stomach, intestines, or other organs. Specifically, when the target organ is the lung, a confocal microendoscopy can be applied to a bronchoscope, and images of the target area of ​​the lung can be acquired using the resulting confocal microendoscopy. When the target organ is the stomach, a confocal microendoscopy can be applied to a gastroscopy, and images of the target area of ​​the stomach can be acquired using the resulting confocal microendoscopy. When the target organ is the intestines, a confocal microendoscopy can be applied to a colonoscope, and images of the target area of ​​the intestines can be acquired using the resulting confocal microendoscopy. This specification does not impose any limitations on these methods.

[0042] For the sake of brevity, the following explanation will use the lung as the target organ and the application of confocal endoscopy to bronchoscopy as an example.

[0043] In the embodiments provided in this application, confocal laser microendoscopy technology is used for bronchoscopes. The bronchoscope is modified by adding a confocal laser microendoscopy to make it a bronchial confocal microendoscopy.

[0044] Step S104: Extract features from the image to obtain the feature map of the fiber structure corresponding to the image.

[0045] During endoscopic examination, images of the bronchial region captured by a confocal microscopy endoscope are input into a pre-trained deep learning network model. A feature extraction network (typically a ResNet network) extracts feature maps of the fibrous structures corresponding to the bronchial region. Abnormalities in the fibrous structure indicate an abnormality in the target organ, such as a lesion. These feature maps can include texture and shape features from the target region in the image.

[0046] ResNet, also known as a residual network, is a type of convolutional neural network.

[0047] Step S106: The feature map of the fiber structure is processed by an attention network to obtain the attention feature map corresponding to the feature map of the fiber structure. The attention feature map is used to identify the target fiber structure in the fiber structure.

[0048] In this step, an attention network is used to process the feature map of the fiber structure to obtain the corresponding attention feature map. The attention network can highlight the features of the fiber structure that need to be emphasized in the feature map, and suppress features that do not belong to the fiber structure or are not important. The target fiber structure in the attention feature map refers to the fiber structure that needs to be emphasized.

[0049] Attention networks introduce the attention mechanism into spatial domain-based neural networks. Unlike spectral domain-based convolutional neural networks, attention networks do not require complex calculations using matrices such as Laplacian; they update node features simply by using the representations of neighboring nodes.

[0050] Attention mechanisms, an informal term, enable neural networks to focus on a subset of their inputs, i.e., select specific inputs. When computational resources are limited, attention mechanisms are a primary means of addressing information overload problems, acting as a resource allocation scheme to allocate computational resources to more important tasks.

[0051] It should be noted that the aforementioned deep learning network model may include modules such as feature extraction networks and attention networks.

[0052] Step S108: Overlay the attention feature map and the image to obtain the attention distribution heatmap corresponding to the image.

[0053] Step S110: Identify the type of fiber structure in the image based on the attention distribution heatmap.

[0054] In this step, the region corresponding to the target fibrous structure in the image of the target area is determined by the attention distribution heatmap. Based on the determined region, the type of fibrous structure in the image of the target area can be determined, thereby judging whether there is an abnormality in the target organ.

[0055] Through the above steps, by applying confocal microscopy to endoscopy, images of the target organ are acquired, and a deep learning network model is used to identify the acquired images. This achieves the goal of identifying the morphology of glands or cells in the target organ using confocal microscopy, thereby simplifying the detection method of lesions in the body and enabling real-time observation of the diseased structure of the organ.

[0056] In some embodiments, when processing the feature map of the fiber structure using an attention network to obtain the attention feature map corresponding to the feature map of the fiber structure, the process includes the following steps: inputting the feature map of the fiber structure into a first type of convolutional layer and a second type of convolutional layer in the attention network, respectively, to obtain a first type of output result and a second type of output result, wherein both the first type of output result and the second type of output result are matrices; performing matrix multiplication on the first type of output result and the second type of output result to obtain the attention feature map corresponding to the feature map of the fiber structure. The first type of convolutional layer and the second type of convolutional layer each contain several different convolutional layers. Using different convolutional layers can extract different features from the feature map of the fiber structure, thereby obtaining different output results.

[0057] When performing matrix multiplication on the first and second type of outputs to obtain the attention feature map corresponding to the feature map of the fiber structure, the following method can be used: Normalize the first type of output to obtain attention weights, where the attention weights represent the importance of each pixel in the feature map of the fiber structure; perform matrix multiplication on the attention weights and the second type of output to obtain the attention feature map corresponding to the feature map of the fiber structure. Normalization refers to mapping the pixel values ​​of each pixel to the range of 0-1.

[0058] Taking the identification of fibrous structures in the lungs as an example, such as Figure 2a As shown, confocal microscopy can be used to acquire images of target areas of the human lung (such as...). Figure 2a As shown in 101), the image of the target region is then input into the feature extraction module 102. The feature extraction network contained in the feature extraction module 102 processes the input image of the target region to obtain the feature map of the fiber structure output by the feature extraction module 102. The extracted feature map of the fiber structure is then input into the fiber structure attention network. The feature map of the fiber structure output by the feature extraction module 102 is then input into three one-dimensional convolutional layers (i.e., ... Figure 2a As shown in 103 / 104 / 105, three results are output respectively. By setting three one-dimensional convolutional layers, different features of the fiber structure feature map can be learned. The feature extraction module 102 includes a feature extraction network. Matrix multiplication is performed on the output results of two of the one-dimensional convolutional layers (103 / 104), and 0-1 normalization learning is performed on the calculated result 106 to obtain attention weights. Matrix multiplication is then performed on the attention weights and the output result of another one-dimensional convolutional layer 105, thereby highlighting the features of the fiber structure of interest, removing useless noise information, and obtaining the activated fiber structure features (i.e., the attention feature map mentioned above).

[0059] In this step, Figure 2a As shown, 103 and 104 are first-type one-dimensional convolutional layers, and 105 is a second-type one-dimensional convolutional layer.

[0060] When determining the attention feature map corresponding to the feature map of the fiber structure, the following method can also be used: input the feature map of the fiber structure into at least three convolutional layers respectively; perform matrix multiplication between the first output result corresponding to the first convolutional layer and the second output result corresponding to the second convolutional layer to obtain the first calculation result; normalize the first calculation result to obtain the attention weight, where the attention weight is the importance of each pixel in the feature map of the fiber structure; perform matrix multiplication between the third output result corresponding to the third convolutional layer and the attention weight to obtain the attention feature map corresponding to the feature map of the fiber structure, where the first output result, the second output result, and the third output result are all matrices.

[0061] In this step, the first convolutional layer, the second convolutional layer, and the third convolutional layer are respectively as follows: Figure 2a As shown in Figures 103, 104, and 105, different features can be learned by setting three convolutional layers.

[0062] The following is a specific embodiment to illustrate the process of generating attention weights in the above scheme. Assume that the feature extraction module 102 outputs a feature map of w*h*c. By inputting the feature map into a one-dimensional convolutional layer 103, and assuming that the one-dimensional convolutional layer 103 includes n 1*1*c convolutional kernels, the one-dimensional convolutional layer 103 outputs a result of w*h*n①, where w, h, and c are the width, height, and number of feature channels of the feature map, respectively.

[0063] By inputting the feature map into a one-dimensional convolutional layer 104, assuming that the one-dimensional convolutional layer 104 also includes n 1*1*c convolutional kernels, the one-dimensional convolutional layer 104 outputs a result of w*h*n②.

[0064] Performing matrix multiplication on these two results means multiplying the w*h features of the first channel of result ① and the w*h features of the first channel of result ② by matrix multiplication, multiplying the w*h features of the second channel of result ① and the w*h features of the second channel of result ② by matrix multiplication, and so on. Finally, after 0-1 normalization learning, the corresponding attention weights are obtained.

[0065] As an optional embodiment, the attention network is used to process the feature map of the fiber structure to obtain the attention feature map corresponding to the feature map of the fiber structure. This can also be achieved through the following method: Input the feature map of the fiber structure into at least four convolutional layers; perform matrix multiplication between the first output of the first convolutional layer and the second output of the second convolutional layer to obtain the (n-2)th calculation result, where n is an integer greater than or equal to 3 and less than N, and N is the number of convolutional layers in the at least four convolutional layers; perform matrix multiplication between the nth output of the nth convolutional layer and the (n-2)th calculation result to obtain the (n-1)th calculation result; normalize the (n-1)th calculation result to obtain the attention weights, where the attention weights represent the importance of each pixel in the feature map of the fiber structure; perform matrix multiplication between the Nth output of the Nth convolutional layer and the attention weights to obtain the attention feature map corresponding to the feature map of the fiber structure, where the first output, second output, nth output, and Nth output are all matrices.

[0066] exist Figure 2a In this process, the feature map of the fiber structure output by the feature extraction module 102 is input into three one-dimensional convolutional layers for processing. In some optional embodiments, the feature map of the fiber structure output by the feature extraction module 102 can also be input into four or more one-dimensional convolutional layers for processing, such as... Figure 2b As shown, the feature maps of the fiber structure are input into four one-dimensional convolutional layers (i.e., ...). Figure 2b The results shown are 103 / 104 / 105 / 110, each outputting four results. The outputs of 103 and 104 are multiplied by a matrix to obtain the first result. The output of 105 is then multiplied by the first result to obtain the second result (i.e.,...). Figure 2b (106 in the middle). The second calculation result is subjected to 0-1 normalization learning to obtain the attention weights. The attention weights are then multiplied by the output of another one-dimensional convolutional layer 110 to remove useless noise information and obtain the activated fiber structure features (i.e., the attention feature map mentioned above).

[0067] It should be noted that, Figure 2b This explanation uses four one-dimensional convolutional layers as an example. If there are five one-dimensional convolutional layers, after obtaining the second calculation result, a matrix multiplication is performed using the second calculation result and the output of one of the one-dimensional convolutional layers to obtain the third calculation result. The third calculation result is then subjected to 0-1 normalization learning to obtain attention weights. A matrix multiplication is then performed between the attention weights and the output of the last one-dimensional convolutional layer to remove useless noise information and obtain the activated fiber structure features (i.e., the attention feature map).

[0068] If the number of one-dimensional convolutional layers is six, and so on, the specific steps will not be elaborated further. It's understandable that the more one-dimensional convolutional layers there are, the more features of the fiber structure's feature map can be learned, resulting in more accurate identification of the fiber structure's morphology. However, considering that a higher number of one-dimensional convolutional layers places higher demands on the processor, and taking into account factors such as processing speed, in practical implementation, a more efficient approach is generally adopted. Figure 2a The architecture shown involves setting up three one-dimensional convolutional layers to process the feature maps of the fiber structure.

[0069] In an optional embodiment of this application, step S106, which involves overlaying the attention feature map and the image of the target region to obtain an attention distribution heatmap corresponding to the image of the target region, includes the following steps: upsampling the attention feature map to obtain an upsampled target attention feature map, wherein the size of the target attention feature map is the same as the size of the image of the target region; and overlaying the target attention feature map and the image of the target region to obtain an attention distribution heatmap.

[0070] The attention feature map obtained above is linearly upsampled to ensure that its size matches that of the original image. Then, the feature map of the activated fiber structure obtained after upsampling is superimposed on the original image to visualize the attention distribution, resulting in a heatmap of the attention distribution corresponding to the original image (e.g., ...). Figure 2a shown in 108).

[0071] It is understandable that, such as Figure 2a Figures 103-107 show the processing flow for processing the feature maps of fiber structures using an attention network.

[0072] There are several methods for identifying the type of fibrous structure in an image of a target region based on an attention distribution heatmap. For example, it can be achieved by: comparing the pixel value of each pixel in the attention distribution heatmap with a preset threshold; defining the region formed by pixels with values ​​greater than the preset threshold as a first region; determining the identification information of the first region, which includes: the area of ​​the first region, the number of first regions, and the distribution of the first regions in the attention distribution heatmap; and identifying the type of fibrous structure in the target organ image based on the identification information of the first region.

[0073] Specifically, the pixel value of each pixel in the attention distribution heatmap 108 is compared one by one with a pre-set threshold. Pixels with pixel values ​​greater than the threshold are determined to belong to the first region (also called the activation region or the region connected component). Then, the first region on the attention distribution heatmap 108 is delineated. Attention region calculation refers to calculating the area of ​​each first region in the attention distribution heatmap 108. The first region contains the features of the fiber structure that need to be highlighted.

[0074] Among them, the following methods are used to identify the type of fibrous structure in the target organ image based on the identification information of the first region:

[0075] Method 1: If the area of ​​the first region is greater than the preset area, the fibrous structure in the target organ structure diagram is determined to be complete; if the area of ​​the first region is not greater than the preset area, the fibrous structure in the target organ structure is determined to be incomplete.

[0076] Method 2: If the number of the first region is greater than the preset number, the fibrous structure in the target organ structure diagram is determined to be complete; if the number of the first region is not greater than the preset number, the fibrous structure in the target organ structure diagram is determined to be incomplete.

[0077] Method 3: If the first region is uniformly distributed in the attention distribution heatmap, the fibrous structure in the target organ structure map is intact; if the first region is unevenly distributed in the attention distribution heatmap, the fibrous structure in the target organ structure map is incomplete.

[0078] Method 4: If the area of ​​each first region is greater than the preset area and the number of each first region is greater than the preset number, then the fibrous structure in the target organ structure diagram is determined to be intact.

[0079] Alternatively, if the area of ​​the first region is greater than the preset area and the distribution of each first region in the attention distribution heatmap is uniform, then the fibrous structure in the target organ structure map is determined to be intact.

[0080] Alternatively, if the number of identified first regions is greater than the preset number and the distribution of each first region in the attention distribution heatmap is uniform, the fibrous structure in the target organ structure map is determined to be intact.

[0081] Method 5: If the area of ​​the first region is greater than the preset area, the number of first regions is greater than the preset number, and the distribution of each first region in the attention distribution heatmap is uniform, then it can be determined that the fibrous structure in the target organ structure map is intact.

[0082] In the above technical solutions, the integrity of the fibrous structure in the bronchial region structure diagram 101 is determined based on the number, size, and distribution of activated regions. Figure 2a The attention distribution heatmap shown in Figure 10901 indicates that several first regions marked in the heatmap are larger than a preset area, the number of first regions is greater than a preset number, and the distribution of each first region in the attention distribution heatmap is uniform. Therefore, it can be determined that the fibrous structure is intact, and the target area corresponding to this attention distribution heatmap does not contain lung cancer. Figure 2a The attention distribution heatmap shown in Figure 10902 does not contain a clearly defined first region, thus indicating incomplete or absent fibrous structures. This corresponds to the presence of lung cancer in the target area of ​​the attention distribution heatmap. Figure 2a As shown in Figure 10903, the number of first regions in the attention distribution heatmap is less than a preset number, and the distribution of each first region in the attention distribution heatmap is uneven. If a large amount of fibrous structure disappears and disorder occurs, lung cancer is present in that region. The first regions in the attention distribution heatmap can be marked with different colors or shapes than other regions; this application does not impose any restrictions on this.

[0083] In summary, the more activated regions there are, the larger the area of ​​the activated regions, and the more uniform the distribution of the activated regions, the more likely it is that "the fibrous structure is intact and there is no lung cancer." Conversely, the fewer activated regions there are, the smaller the area of ​​the activated regions, and the more dispersed the distribution of the activated regions, the more likely it is that "the fibrous structure has disappeared or completely disappeared, and there is a possibility of lung cancer."

[0084] The specific number, area, or distribution of regions used as thresholds for judging the integrity of fibrous structures can generally be determined based on clinical data.

[0085] It should be noted that although the above example is based on the identification of lung cancer structures, the solution in this application embodiment is only used to identify specific types of lung structures and is not necessarily used to identify lung cancer, nor can it directly determine whether the target object has lung cancer. For example, it is necessary to combine other symptoms of the target object for judgment.

[0086] Figure 3 This is a flowchart of another organ image processing method according to an embodiment of this application, such as... Figure 3 As shown, the method includes the following steps:

[0087] Step S302: Acquire an image of the target region in the target organ, wherein the image is acquired by a confocal endoscope.

[0088] Step S304: Extract features from the image to obtain the feature map of the fiber structure corresponding to the image.

[0089] Step S306: The feature map of the fiber structure is processed by an attention network to obtain the attention feature map corresponding to the feature map of the fiber structure. The attention feature map is used to identify the target fiber structure in the fiber structure.

[0090] Step S308: Overlay the attention feature map and the image to obtain the attention distribution heatmap corresponding to the image.

[0091] Step S310: Display the heatmap of attention distribution.

[0092] It should be noted that, Figure 3 Preferred embodiments of the shown examples can be found in the description of the examples shown in Figure 1, and will not be repeated here.

[0093] Figure 4 This is a structural block diagram of a device for identifying fibrous structures in organ images according to an embodiment of this application, as shown below. Figure 4 As shown, the device includes:

[0094] The acquisition module 40 is used to acquire images of the target region in the target organ, wherein the images are acquired by a confocal endoscope.

[0095] As an optional embodiment, the target organ can be the lung, stomach, intestines, or other organs. Specifically, when the target organ is the lung, a confocal microendoscopy can be applied to a bronchoscope, and images of the target area of ​​the lung can be acquired using the resulting confocal microendoscopy. When the target organ is the stomach, a confocal microendoscopy can be applied to a gastroscopy, and images of the target area of ​​the stomach can be acquired using the resulting confocal microendoscopy. When the target organ is the intestines, a confocal microendoscopy can be applied to a colonoscope, and images of the target area of ​​the intestines can be acquired using the resulting confocal microendoscopy. This specification does not impose any limitations on these methods.

[0096] For the sake of brevity, the following explanation will use the lung as the target organ and the application of confocal endoscopy to bronchoscopy as an example.

[0097] In the embodiments provided in this application, confocal laser microendoscopy technology is used for bronchoscopes. The bronchoscope is modified by adding a confocal laser microendoscopy to make it a bronchial confocal microendoscopy.

[0098] The extraction module 42 is used to extract features from the image to obtain the feature map of the fiber structure corresponding to the image.

[0099] During endoscopic examination, images of the bronchial region captured by a confocal microscopy endoscope are input into a pre-trained deep learning network model. A feature extraction network (typically a ResNet network) extracts feature maps of the fibrous structures corresponding to the bronchial region. Abnormalities in the fibrous structure indicate an abnormality in the target organ, such as a lesion. These feature maps can include texture and shape features from the target region in the image.

[0100] ResNet, also known as a residual network, is a type of convolutional neural network.

[0101] It should be noted that the feature extraction network mentioned above is a module in a deep learning network model.

[0102] The first processing module 44 is used to process the feature map of the fiber structure using an attention network to obtain the attention feature map corresponding to the feature map of the fiber structure, wherein the attention feature map is used to identify the target fiber structure in the fiber structure.

[0103] An attention network is used to process the feature map of the fiber structure to obtain the corresponding attention feature map. The attention network can highlight the features of the fiber structure that need to be emphasized in the feature map, and suppress features that do not belong to the fiber structure or are not important. The target fiber structure in the attention feature map refers to the fiber structure that needs to be emphasized.

[0104] Attention networks introduce the attention mechanism into spatial domain-based neural networks. Unlike spectral domain-based convolutional neural networks, attention networks do not require complex calculations using matrices such as Laplacian; they update node features simply by using the representations of neighboring nodes.

[0105] Attention mechanisms, an informal term, enable neural networks to focus on a subset of their inputs, i.e., select specific inputs. When computational resources are limited, attention mechanisms are a primary means of addressing information overload problems, acting as a resource allocation scheme to allocate computational resources to more important tasks.

[0106] It should be noted that the aforementioned deep learning network model may include modules such as feature extraction networks and attention networks.

[0107] The second processing module 46 is used to overlay the attention features and the image to obtain the attention distribution heatmap corresponding to the image.

[0108] The recognition module 48 is used to identify the type of fiber structure in the image based on the attention distribution heatmap.

[0109] It should be noted that, Figure 4Preferred embodiments of the shown examples can be found in the description of the examples shown in Figure 1, and will not be repeated here.

[0110] According to an optional embodiment of this application, the first processing module 44 includes: a first processing unit, configured to input the feature map of the fiber structure into a first type of convolutional layer and a second type of convolutional layer in the attention network, respectively, to obtain a first type of output result and a second type of output result, wherein both the first type of output result and the second type of output result are matrices; and a second processing unit, configured to perform matrix multiplication on the first type of output result and the second type of output result to obtain an attention feature map corresponding to the feature map of the fiber structure. The first type of convolutional layer and the second type of convolutional layer each contain several different convolutional layers, and different convolutional layers can be used to extract different features from the feature map of the fiber structure, thereby obtaining different output results.

[0111] According to another optional embodiment of this application, the second processing unit includes: a normalization subunit, used to normalize the first type of output result to obtain attention weights, wherein the attention weights are the importance of each pixel in the feature map of the fiber structure; and a calculation subunit, used to perform matrix multiplication on the attention weights and the second type of output result to obtain the attention feature map corresponding to the feature map of the fiber structure. The normalization process refers to mapping the pixel values ​​of each pixel to a range of 0-1.

[0112] Taking the identification of fibrous structures in the lungs as an example, such as Figure 2a As shown, confocal microscopy can be used to acquire images of target areas of the human lung (such as...). Figure 2a As shown in 101), the image of the target region is then input into the feature extraction module 102. The feature extraction network contained in the feature extraction module 102 processes the input image of the target region to obtain the feature map of the fiber structure output by the feature extraction module 102. The extracted feature map of the fiber structure is then input into the fiber structure attention network. The feature map of the fiber structure output by the feature extraction module 102 is then input into three one-dimensional convolutional layers (i.e., ... Figure 2a As shown in 103 / 104 / 105, three results are output respectively. By setting three one-dimensional convolutional layers, different features of the fiber structure feature map can be learned. Matrix multiplication is performed on the outputs of two of the one-dimensional convolutional layers (103 / 104), and 0-1 normalization is performed on the calculated result 106 to obtain attention weights. Matrix multiplication is then performed on the attention weights and the output of the other one-dimensional convolutional layer 105, thereby highlighting the key fiber structure features, removing useless noise information, and obtaining the activated fiber structure features (i.e., the attention feature map mentioned above).

[0113] In this step, Figure 2aAs shown, 103 and 104 are first-type one-dimensional convolutional layers, and 105 is a second-type one-dimensional convolutional layer.

[0114] As an optional embodiment of this application, the first processing module 44 further includes: an input unit for inputting the feature map of the fiber structure into at least three convolutional layers respectively; a third processing unit for performing matrix multiplication on the first output result corresponding to the first convolutional layer and the second output result corresponding to the second convolutional layer among the at least three convolutional layers to obtain a first calculation result; a fourth processing unit for normalizing the first calculation result to obtain attention weights, wherein the attention weights are the importance of each pixel in the feature map of the fiber structure; and a fifth processing unit for performing matrix multiplication on the third output result corresponding to the third convolutional layer among the at least three convolutional layers and the attention weights to obtain an attention feature map corresponding to the feature map of the fiber structure, wherein the first output result, the second output result, and the third output result are all matrices.

[0115] In this step, the first one-dimensional convolutional layer, the second one-dimensional convolutional layer, and the third one-dimensional convolutional layer are respectively as follows: Figure 2a As shown in Figures 103, 104, and 105, different features can be learned by setting three one-dimensional convolutional layers.

[0116] The following is a specific embodiment to illustrate the process of generating attention weights in the above scheme. Assume that the feature extraction module 102 outputs a w*h*c feature map. By inputting the feature map into a one-dimensional convolutional layer 103, and assuming that the one-dimensional convolutional layer 103 includes n 1*1*c convolutional kernels, the one-dimensional convolutional layer 103 outputs a w*h*n result ①.

[0117] By inputting the feature map into a one-dimensional convolutional layer 104, assuming that the one-dimensional convolutional layer 104 also includes n 1*1*c convolutional kernels, the one-dimensional convolutional layer 104 outputs a result of w*h*n②.

[0118] Performing matrix multiplication on these two results means multiplying the w*h features of the first channel of result ① and the w*h features of the first channel of result ② by matrix multiplication, multiplying the w*h features of the second channel of result ① and the w*h features of the second channel of result ② by matrix multiplication, and so on. Finally, after 0-1 normalization learning, the corresponding attention weights are obtained.

[0119] As an optional embodiment, the first processing module 44 is further configured to input the feature map of the fiber structure into at least four convolutional layers respectively; perform matrix multiplication on the first output result corresponding to the first convolutional layer and the second output result corresponding to the second convolutional layer among the at least four convolutional layers to obtain the (n-2)th calculation result, where n is an integer greater than or equal to 3 and less than N, and N is the number of convolutional layers included in the at least four convolutional layers; perform matrix multiplication on the nth output result corresponding to the nth convolutional layer among the at least four convolutional layers and the (n-2)th calculation result to obtain the (n-1)th calculation result; normalize the (n-1)th calculation result to obtain attention weights, where the attention weights are the importance of each pixel in the feature map of the fiber structure; and perform matrix multiplication on the Nth output result corresponding to the Nth convolutional layer among the at least four convolutional layers and the attention weights to obtain the attention feature map corresponding to the feature map of the fiber structure, where the first output result, the second output result, the nth output result, and the Nth output result are all matrices.

[0120] exist Figure 2a In this process, the feature map of the fiber structure output by the feature extraction module 102 is input into three one-dimensional convolutional layers for processing. In some optional embodiments, the feature map of the fiber structure output by the feature extraction module 102 can also be input into four or more one-dimensional convolutional layers for processing, such as... Figure 2b As shown, the feature maps of the fiber structure are input into four one-dimensional convolutional layers (i.e., ...). Figure 2b The results shown are 103 / 104 / 105 / 110, each outputting four results. The outputs of 103 and 104 are multiplied by a matrix to obtain the first result. The output of 105 is then multiplied by the first result to obtain the second result (i.e.,...). Figure 2b (106 in the middle). The second calculation result is subjected to 0-1 normalization learning to obtain the attention weights. The attention weights are then multiplied by the output of another one-dimensional convolutional layer 110 to remove useless noise information and obtain the activated fiber structure features (i.e., the attention feature map mentioned above).

[0121] It should be noted that, Figure 2b This explanation uses four one-dimensional convolutional layers as an example. If there are five one-dimensional convolutional layers, after obtaining the second calculation result, a matrix multiplication is performed using the second calculation result and the output of one of the one-dimensional convolutional layers to obtain the third calculation result. The third calculation result is then subjected to 0-1 normalization learning to obtain attention weights. A matrix multiplication is then performed between the attention weights and the output of the last one-dimensional convolutional layer to remove useless noise information and obtain the activated fiber structure features (i.e., the attention feature map).

[0122] If the number of one-dimensional convolutional layers is six, and so on, the specific steps will not be elaborated further. It's understandable that the more one-dimensional convolutional layers there are, the more features of the fiber structure's feature map can be learned, resulting in more accurate identification of the fiber structure's morphology. However, considering that a higher number of one-dimensional convolutional layers places higher demands on the processor, and taking into account factors such as processing speed, in practical implementation, a more efficient approach is generally adopted. Figure 2a The architecture shown involves setting up three one-dimensional convolutional layers to process the feature maps of the fiber structure.

[0123] In some optional embodiments of this application, the second processing module 46 includes: a sampling unit, used to upsample the attention feature map to obtain an upsampled target attention feature map, wherein the size of the target attention feature map is the same as the size of the image of the target region; and a superposition unit, used to superimpose the target attention feature map and the image of the target region to obtain an attention distribution heatmap.

[0124] In some other optional embodiments of this application, the identification module 48 includes: a comparison unit, configured to compare the pixel value of each pixel in the attention distribution heatmap with a preset threshold; a first determination unit, configured to determine the region formed by pixels with pixel values ​​greater than the preset threshold as a first region; a second determination unit, configured to determine the identification information of the first region, wherein the identification information includes: the area of ​​the first region, the number of first regions, and the distribution of the first region in the attention distribution heatmap; and an identification unit, configured to identify the type of fibrous structure in the target organ structure diagram based on the identification information of the first region.

[0125] As an optional embodiment of this application, the above-mentioned identification unit is used to perform at least one of the following: when the identification information of the first region meets the preset conditions, it determines that the fiber structure in the image is complete, wherein the preset conditions include at least one of the following: the area of ​​the first region is greater than the preset area, the number of the first regions is greater than the preset number, and the distribution of the first regions in the attention distribution heatmap is uniform.

[0126] Figure 5 This is a structural block diagram of a system for recognizing fibrous structures in organ images according to an embodiment of this application, such as... Figure 5 As shown, the system includes: a confocal endoscope 50 and a processor 52, wherein,

[0127] The confocal endoscope 50 is used to acquire images of target regions within target organs.

[0128] According to an optional embodiment of this application, the target organ can be the lung, stomach, intestines, or other organs. Specifically, when the target organ is the lung, a confocal microendoscopy can be applied to a bronchoscope, and the resulting confocal microendoscopy can be used to acquire images of the target region of the lung; when the target organ is the stomach, a confocal microendoscopy can be applied to a gastroscopy, and the resulting confocal microendoscopy can be used to acquire images of the target region of the stomach; when the target organ is the intestines, a confocal microendoscopy can be applied to a colonoscope, and the resulting confocal microendoscopy can be used to acquire images of the target region of the intestines. This specification does not limit the scope of the application.

[0129] The processor 52, connected to the confocal endoscope 50, is used to perform the above-described method for identifying fibrous structures in organ images and the above-described method for processing organ images.

[0130] As an optional embodiment of this application, the processor 52 may be a locally deployed server or a cloud server.

[0131] The processor 52 and the confocal endoscope 50 can be connected wirelessly.

[0132] It should be noted that, Figure 5 Preferred embodiments of the shown examples can be found in the description of the examples shown in Figure 1, and will not be repeated here.

[0133] Figure 6 This is a schematic diagram illustrating an application of identifying fibrous structures in lung images according to an embodiment of this application, such as... Figure 6 As shown, a bronchoscopy with added confocal microscopy is used to acquire structural images of the bronchial region of the lungs. These images are then sent to a bronchoscopy navigation system, where the processor executes the following steps:

[0134] Step S602: Extract features from the image of the target region in the target organ to obtain the feature map of the fiber structure corresponding to the image of the target region.

[0135] Step S604: Use an attention network to process the feature map of the fiber structure to obtain the attention feature map corresponding to the feature map of the fiber structure.

[0136] Step S606: Overlay the attention feature map and the image of the target region to obtain the attention distribution heatmap corresponding to the image of the target region.

[0137] Step S608: Identify the type of fiber structure in the image of the target region based on the attention distribution heatmap.

[0138] It should be noted that, Figure 6Preferred embodiments of the shown examples can be found in the description of the examples shown in Figure 1, and will not be repeated here.

[0139] This application provides a non-volatile storage medium, which includes a stored program. When the program runs, it controls the device where the non-volatile storage medium is located to execute the above-mentioned method for identifying fibrous structures in organ images and the method for processing organ images.

[0140] The aforementioned non-volatile storage medium is used to store a program that performs the following functions: acquiring an image of a target region in a target organ, wherein the image is acquired by a confocal endoscope; extracting features from the image to obtain a feature map of the corresponding fibrous structure; processing the feature map of the fibrous structure using an attention network to obtain an attention feature map corresponding to the feature map of the fibrous structure, wherein the attention feature map is used to identify the target fibrous structure in the fibrous structure; overlaying the attention feature map and the image to obtain an attention distribution heatmap corresponding to the image; and identifying the type of fibrous structure in the image based on the attention distribution heatmap.

[0141] The process involves acquiring images of the target region within the target organ, obtained through confocal microscopy; extracting features from the images to obtain feature maps of the corresponding fibrous structures; processing the feature maps of the fibrous structures using an attention network to obtain attention feature maps corresponding to the feature maps of the fibrous structures, where the attention feature maps are used to identify the target fibrous structures within the fibrous structures; overlaying the attention feature maps and the images to obtain an attention distribution heatmap corresponding to the images; and displaying the attention distribution heatmap.

[0142] This application also provides an electronic device, including: a memory and a processor, wherein the processor is used to run a program stored in the memory, wherein the program executes the above-described method for identifying fibrous structures in organ images and the method for processing organ images.

[0143] The processor described above is used to run a program that performs the following functions: acquiring an image of a target region in a target organ, wherein the image is acquired by a confocal endoscope; extracting features from the image to obtain a feature map of the corresponding fibrous structure; processing the feature map of the fibrous structure using an attention network to obtain an attention feature map corresponding to the feature map of the fibrous structure, wherein the attention feature map is used to identify the target fibrous structure in the fibrous structure; overlaying the attention feature map and the image to obtain an attention distribution heatmap corresponding to the image; and identifying the type of fibrous structure in the image based on the attention distribution heatmap.

[0144] The process involves acquiring images of the target region within the target organ, obtained through confocal microscopy; extracting features from the images to obtain feature maps of the corresponding fibrous structures; processing the feature maps of the fibrous structures using an attention network to obtain attention feature maps corresponding to the feature maps of the fibrous structures, where the attention feature maps are used to identify the target fibrous structures within the fibrous structures; overlaying the attention feature maps and the images to obtain an attention distribution heatmap corresponding to the images; and displaying the attention distribution heatmap.

[0145] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0146] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0147] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0148] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0149] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0150] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to related technologies, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0151] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for identifying fibrous structures in organ images, characterized in that, include: Acquire an image of a target region in a target organ, wherein the image is acquired using a confocal endoscope. Feature extraction is performed on the image to obtain the feature map of the fiber structure corresponding to the image; An attention network is used to process the feature map of the fiber structure to obtain an attention feature map corresponding to the feature map of the fiber structure, wherein the attention feature map is used to identify the target fiber structure in the fiber structure; The attention feature map and the image are overlaid to obtain the attention distribution heatmap corresponding to the image; Identifying the type of fiber structure in the image based on the attention distribution heatmap includes: comparing the pixel value of each pixel in the attention distribution heatmap with a preset threshold; determining the region formed by pixels with pixel values ​​greater than the preset threshold as a first region; determining the identification information of the first region, wherein the identification information includes: the area of ​​the first region, the number of the first regions, and the distribution of the first regions in the attention distribution heatmap; identifying the type of fiber structure in the image based on the identification information of the first region, wherein if the identification information of the first region meets a preset condition, the fiber structure in the image is determined to be complete, wherein the preset condition includes at least one of the following: the area of ​​the first region is greater than a preset area, the number of the first regions is greater than a preset number, and the distribution of the first regions in the attention distribution heatmap is uniform.

2. The method according to claim 1, characterized in that, An attention network is used to process the feature map of the fiber structure to obtain an attention feature map corresponding to the feature map of the fiber structure, including: The feature map of the fiber structure is input into the first type of convolutional layer and the second type of convolutional layer in the attention network to obtain the first type of output result and the second type of output result, wherein the first type of output result and the second type of output result are both matrices; Perform matrix multiplication on the first type of output results and the second type of output results to obtain the attention feature map corresponding to the feature map of the fiber structure.

3. The method according to claim 2, characterized in that, Perform matrix multiplication on the first type of output and the second type of output to obtain the attention feature map corresponding to the feature map of the fiber structure, including: The first type of output result is normalized to obtain attention weights, wherein the attention weights are the importance of each pixel in the feature map of the fiber structure; Perform matrix multiplication on the attention weights and the second type of output to obtain the attention feature map corresponding to the feature map of the fiber structure.

4. The method according to claim 1, characterized in that, The feature map of the fiber structure is processed using an attention network to obtain an attention feature map corresponding to the feature map of the fiber structure, and the process further includes: The feature map of the fiber structure is input into three convolutional layers respectively; Perform matrix multiplication between the first output result of the first convolutional layer and the second output result of the second convolutional layer to obtain the first calculation result; The first calculation result is normalized to obtain the attention weight, wherein the attention weight is the importance of each pixel in the feature map of the fiber structure; The third output of the third convolutional layer is multiplied by the attention weight to obtain the attention feature map corresponding to the feature map of the fiber structure, wherein the first output, the second output, and the third output are all matrices.

5. The method according to claim 1, characterized in that, The feature map of the fiber structure is processed using an attention network to obtain an attention feature map corresponding to the feature map of the fiber structure, and the process further includes: The feature maps of the fiber structure are input into at least four convolutional layers respectively; Perform matrix multiplication between the first output result corresponding to the first convolutional layer and the second output result corresponding to the second convolutional layer in the at least four convolutional layers to obtain the (n-2)th calculation result, where n is an integer greater than or equal to 3 and less than N, and N is the number of convolutional layers included in the at least four convolutional layers. Perform matrix multiplication between the nth output result of the nth convolutional layer among the at least four convolutional layers and the (n-2)th calculation result to obtain the (n-1)th calculation result. The (n-1)th calculation result is normalized to obtain the attention weight, wherein the attention weight is the importance of each pixel in the feature map of the fiber structure; The Nth output of the Nth convolutional layer among the at least four convolutional layers is multiplied by the attention weights to obtain the attention feature map corresponding to the feature map of the fiber structure, wherein the first output, the second output, the nth output, and the Nth output are all matrices.

6. The method according to claim 1, characterized in that, The attention feature map and the image are overlaid to obtain an attention distribution heatmap corresponding to the image, including: The attention feature map is upsampled to obtain an upsampled target attention feature map, wherein the size of the target attention feature map is the same as the size of the image of the target region; The target attention feature map and the image are superimposed to obtain the attention distribution heatmap.

7. A method for processing organ images, characterized in that, include: Acquire an image of a target region in a target organ, wherein the image is acquired using a confocal endoscope. Feature extraction is performed on the image to obtain the feature map of the fiber structure corresponding to the image; An attention network is used to process the feature map of the fiber structure to obtain an attention feature map corresponding to the feature map of the fiber structure, wherein the attention feature map is used to identify the target fiber structure in the fiber structure; The attention feature map and the image are overlaid to obtain the attention distribution heatmap corresponding to the image; The attention distribution heatmap is displayed, wherein the heatmap is used to identify the type of fiber structure in the image. Specifically, the pixel value of each pixel in the attention distribution heatmap is compared with a preset threshold; the region formed by pixels with pixel values ​​greater than the preset threshold is defined as a first region; identification information of the first region is determined, wherein the identification information includes: the area of ​​the first region, the number of first regions, and the distribution of the first regions in the attention distribution heatmap; the type of fiber structure in the image is identified based on the identification information of the first region, wherein if the identification information of the first region meets preset conditions, the fiber structure in the image is determined to be complete, wherein the preset conditions include at least one of the following: the area of ​​the first region is greater than a preset area, the number of first regions is greater than a preset number, and the distribution of the first regions in the attention distribution heatmap is uniform.

8. A device for identifying fibrous structures in organ images, characterized in that, include: An acquisition module is used to acquire images of a target region in a target organ, wherein the images are acquired using a confocal endoscope. The extraction module is used to extract features from the image to obtain a feature map of the fiber structure corresponding to the image; The first processing module is used to process the feature map of the fiber structure using an attention network to obtain an attention feature map corresponding to the feature map of the fiber structure, wherein the attention feature map is used to identify the target fiber structure in the fiber structure; The second processing module is used to overlay the attention feature map and the image to obtain the attention distribution heatmap corresponding to the image; The identification module is used to identify the type of fiber structure in the image based on the attention distribution heatmap, including: comparing the pixel value of each pixel in the attention distribution heatmap with a preset threshold; determining the region formed by pixels with pixel values ​​greater than the preset threshold as a first region; determining the identification information of the first region, wherein the identification information includes: the area of ​​the first region, the number of the first regions, and the distribution of the first regions in the attention distribution heatmap; identifying the type of fiber structure in the image based on the identification information of the first region, wherein if the identification information of the first region meets the preset conditions, the fiber structure in the image is determined to be complete, wherein the preset conditions include at least one of the following: the area of ​​the first region is greater than a preset area, the number of the first regions is greater than a preset number, and the distribution of the first regions in the attention distribution heatmap is uniform.

9. A system for identifying fibrous structures in organ images, characterized in that, include: Confocal endoscope and processor, among which, The confocal endoscope is used to acquire images of the target region in the target organ; The processor, connected to the confocal endoscope, is used to execute the method for identifying fibrous structures in organ images as described in any one of claims 1 to 6, and the method for processing organ images as described in claim 7.

10. A non-volatile storage medium, characterized in that, The non-volatile storage medium includes a stored computer program, wherein the device containing the non-volatile storage medium executes the method for identifying fibrous structures in an organ image as described in any one of claims 1 to 6 and the method for processing an organ image as described in claim 7 by running the computer program.

11. An electronic device, characterized in that, include: A memory and a processor, the processor being configured to run a program stored in the memory, wherein the program, when running, performs the method for identifying fibrous structures in an organ image as described in any one of claims 1 to 6 and the method for processing an organ image as described in claim 7.

Images