Tissue assisted segmentation system, method, and storage medium

Abstract

The application discloses a kind of tissue auxiliary segmentation system, method and storage medium, the system includes: surgical instrument, display device, processor, utilize the first module output the first segmentation result of anatomical region image, and the intermediate processing result of second module and third module includes first feature map and second feature map respectively;According to the first segmentation result and second feature map, determine the region of interest in first feature map by second module, and adjust at least part of features in the note region;Second module outputs second segmentation result according to the first feature map after adjustment;Based on second segmentation result, update enhancement effect.The application improves the real-time and accuracy of endoscopic image segmentation.

Description

Technical Field

[0001] This invention belongs to the field of medical devices, and in particular relates to a tissue-assisted segmentation system, method and storage medium. Background Technology

[0002] During surgery, after the endoscope acquires images of the surgical area, it is typically necessary to accurately segment the region of interest (ROI) within the endoscopic image to determine the target tissue within the ROI to be extracted. Currently, the market mainly offers two types of image segmentation technologies: those based on the Unet neural network model and those based on the Transformer neural network model. These technologies automatically detect tissue regions and segment the detected tissue regions.

[0003] The aforementioned solutions suffer from the following problems: Transformer-based segmentation methods achieve higher segmentation accuracy, but their image segmentation efficiency cannot meet the demands of real-time segmentation; while UNet-based methods are more efficient, but their accuracy is slightly lacking. Especially when dealing with tissue region detection and segmentation in complex in vivo environments, both Transformer and UNet experience a decrease in segmentation accuracy, and real-time performance is difficult to achieve simultaneously. In other words, existing image segmentation solutions for endoscopy struggle to improve image perception capabilities under different shooting scenarios and achieve accurate real-time image segmentation results. Summary of the Invention

[0004] To address the technical problem that existing image segmentation schemes used in endoscopes struggle to improve image perception capabilities under different shooting scenarios and achieve accurate image segmentation results in real time, this invention proposes the following technical solution:

[0005] A first aspect of the present invention provides an image segmentation method, comprising: acquiring an endoscope image and inputting the endoscope image into a convolutional neural network, wherein the convolutional layer of the convolutional neural network is provided with a plurality of convolutional kernels; performing convolution processing on a single-channel input of the endoscope image using the plurality of convolutional kernels to obtain a plurality of feature components in each channel; determining the importance of each convolutional kernel based on each feature component in each channel, and fusing the feature components in each channel based on the importance to obtain a feature map of the single-channel output of the convolutional layer; and performing image segmentation on the feature map to obtain a segmentation result.

[0006] Preferably, determining the importance of each convolutional kernel based on the feature components within each channel includes: performing global pooling on the feature components within each channel, and obtaining the importance of each convolutional kernel based on the result of the global pooling, wherein the global pooling includes global average pooling and global max pooling.

[0007] Preferably, the step of fusing the feature components within each channel based on the importance to obtain the feature map of the single-channel output of the convolutional layer includes: converting the result of global pooling into weights, performing weight enhancement on the feature components within each channel respectively to obtain enhanced feature components within each channel, wherein the weights are used to measure the image perception intensity of each convolutional kernel in the current endoscopic imaging scenario; and adding the enhanced feature components within each channel to obtain the feature map of the single-channel output of the convolutional layer.

[0008] Preferably, the convolutional neural network includes multiple convolutional layers, each outputting a feature map at a different resolution. The step of performing image segmentation on the feature map to obtain a segmentation result includes: performing image segmentation on the lower-resolution feature map to obtain a lower-resolution segmentation mask; passing the lower-resolution segmentation mask and passing back the higher-resolution feature map, and fusing the passed segmentation mask, the passed-back feature map, and the current-resolution feature map to obtain a fused feature map; performing image segmentation on the fused feature map to obtain a segmentation mask at the current resolution, wherein when the current resolution meets a preset resolution, the current-resolution segmentation mask is used as the segmentation result.

[0009] Preferably, the step of using multiple convolution kernels to perform convolution processing on the single-channel input of the endoscope image to obtain multiple channel feature components includes: detecting the shooting distance of the endoscope image; selecting multiple convolution kernels corresponding to the shooting distance, and performing convolution processing on the single-channel input of the endoscope image to obtain corresponding channel feature components; wherein, the sizes of the multiple convolution kernels in different groups are different.

[0010] Preferably, the step of selecting multiple convolution kernels corresponding to the shooting distance and performing convolution processing on the single-channel input of the endoscope image to obtain the corresponding in-channel feature components includes: determining the distance range to which the shooting distance belongs, and selecting multiple convolution kernels with a size interval corresponding to the distance range, wherein the larger the distance range, the larger the size interval, and at least one of the multiple convolution kernels includes a convolution kernel with a size of 1*1; and performing convolution processing on the single-channel input of the endoscope image based on the multiple convolution kernels with the size interval to obtain the corresponding in-channel feature components.

[0011] A second aspect of the present invention provides an image segmentation model, the image segmentation model comprising at least an encoder and a decoder; the encoder includes at least one convolutional layer, the convolutional layer including a convolutional unit with multiple convolutional kernels, a pooling unit, and a fusion unit; the convolutional unit is used to: acquire an endoscope image; perform convolution processing on a single-channel input of the endoscope image using the multiple convolutional kernels to obtain multiple channel-specific feature components; the pooling unit is used to: determine the importance of each convolutional kernel based on the feature components in each channel; the fusion unit is used to: fuse the feature components in each channel based on the importance to obtain a feature map of the single-channel output of the convolutional layer; the decoder is used to: perform image segmentation on the feature map to obtain a segmentation result.

[0012] Preferably, the encoder includes n sequentially connected convolutional layers, each convolutional layer outputting feature maps of different resolutions, where n is a positive integer; the decoder includes n-1 sequentially connected network modules for generating and outputting segmentation masks of different resolutions based on the feature maps output by each convolutional layer; wherein each network module includes interconnected upsampling layers, fusion layers, residual blocks, and fully connected layers; the i-th convolutional layer is also connected to the upsampling layer of the i-th network module and the fusion layer of the (i-1)-th network module, for: when i≠1, sending the feature map to the upsampling layer of the i-th network module and back to the fusion layer of the (i-1)-th network module; when i=1, sending the feature map to the upsampling layer of the i-th network module; wherein the i-th convolutional layer is counted according to the reverse connection order, and the i-th network module is counted according to the forward connection order, i∈n; the upsampling layer of the i-th network module is used to: perform upsampling processing on the received feature map. The upsampled feature map is obtained and sent to the fusion layer of the i-th network module. The fully connected layer of the (i-1)-th network module is also connected to the fusion layer of the i-th network module, and is used to output the segmentation mask and pass it to the fusion layer of the i-th network module when i ≠ 1. The fusion layer of the i-th network module is used to: fuse the received feature map, the upsampled feature map and the segmentation mask when i ≠ 1 to obtain a first fused feature map; fuse the received feature map and the upsampled feature map when i = 1 to obtain a second fused feature map, and send the first fused feature map or the first fused feature map to the residual block of the i-th network module. The residual block of the i-th network module is used to: perform image segmentation on the received first fused feature map or the second fused feature map, and generate a segmentation mask through the fully connected layer of the i-th network module. The segmentation mask generated by the fully connected layer of the (n-1)-th network module is output as the final segmentation result.

[0013] A third aspect of the present invention provides a tissue-assisted segmentation system, the system comprising: a surgical instrument, wherein an image acquisition device is deployed at the distal end of the surgical instrument for acquiring an image of an anatomical region; a display device for displaying an enhancement effect on the anatomical region image; and a processor having a deep learning model deployed thereon, the deep learning model comprising a first module, a second module, and a third module connected in sequence, the processor being configured to: output a first segmentation result of the anatomical region image using the first module, wherein intermediate processing results of the second module and the third module respectively include a first feature map and a second feature map; determine a region of interest in the first feature map using the second module based on the first segmentation result and the second feature map, and adjust at least some features in the region of interest; output a second segmentation result based on the adjusted first feature map; and update the enhancement effect based on the second segmentation result.

[0014] Preferably, the second module includes a first decoder, configured to: determine a first segmentation mask based on the first segmentation result, and determine a second segmentation mask based on the second feature map, wherein the area covered by the first segmentation mask is larger than the area covered by the second segmentation mask; and map the first segmentation mask and the second segmentation mask onto the first feature map to obtain the region of interest in the first feature map.

[0015] Preferably, the region of interest includes a first region of interest mapped only by the first segmentation mask, a second region of interest mapped only by the second segmentation mask, and a third region of interest mapped by both the first segmentation mask and the second mask, and at least satisfies the following condition: both the first region of interest and the third region of interest are larger than the second region of interest.

[0016] Preferably, the first decoder is further configured to: increase the feature value in the first region of interest by a preset first amplitude, increase the feature value in the second region of interest by a preset second amplitude, and / or increase the feature value in the third region of interest by a preset third amplitude, wherein the preset third amplitude is greater than the preset second amplitude and the preset first amplitude.

[0017] Preferably, the first decoder is further configured to: map the first segmentation mask onto the first feature map to obtain a continuous feature point set, and map the second segmentation mask onto the first feature map to obtain a discrete feature point set; and determine the region of interest in the first feature map based on the continuous feature point set and the discrete feature point set, wherein the continuous feature point set includes at least a portion of the feature points in the discrete feature point set.

[0018] Preferably, the first decoder is further configured to: determine the number of consecutive feature points at each discrete position in the discrete feature point set, and select feature points at discrete positions where the number of consecutive feature points exceeds a preset first threshold; determine target feature points among the selected feature points at discrete positions that belong to the continuous feature point set, and increase the feature value of the target feature points.

[0019] Preferably, the first decoder is further configured to: determine the number of feature points that the second segmentation mask maps to the same discrete feature point in the discrete feature point set; determine discrete feature points whose number of mapped feature points exceeds a preset second threshold, and increase the feature value of the determined discrete feature points.

[0020] Preferably, the first module includes a first convolutional layer and a second decoder, the second module further includes a second convolutional layer, and the third module includes a third convolutional layer; wherein the third convolutional layer, the second convolutional layer, and the first convolutional layer are sequentially connected, and when the anatomical region image sequentially passes through the third convolutional layer, the second convolutional layer, and the first convolutional layer, a third feature map, a first feature map, and a second feature map are output respectively; the second decoder is connected to the first convolutional layer and the second convolutional layer respectively, and the second decoder outputs the first segmentation result based at least on the third feature map output by the first convolutional layer and the first feature map output by the second convolutional layer; the first decoder is connected to the second decoder, the second convolutional layer, and the third convolutional layer respectively.

[0021] Preferably, the third feature map is at a preset first resolution, the first segmentation result and the first feature map are at a preset second resolution, and the second feature map is at a preset third resolution, wherein the preset first resolution is greater than the preset second resolution, and the preset second resolution is greater than the preset third resolution; the second decoder further includes a first upsampling layer, which is connected to the first convolutional layer to upsample the third feature map from the preset first resolution to the preset second resolution; the first decoder further includes a second upsampling layer, which is connected to the second convolutional layer to upsample the first feature map from the preset second resolution to the preset third resolution; a third upsampling layer is also connected between the second decoder and the first decoder to upsample the first segmentation result from the preset second resolution to the preset third resolution.

[0022] Preferably, the first, second, and third convolutional layers each have multiple convolutional kernels of different sizes and multiple global pooling units connected to each kernel. Each convolutional kernel in the first convolutional layer performs convolution processing on the anatomical region image to obtain multiple corresponding convolutional results. Each global pooling unit in the first convolutional layer performs global pooling processing on the corresponding convolutional result to obtain multiple corresponding pooling results. The multiple pooling results are weighted to obtain a third feature map output by the first convolutional layer. The second convolutional layer performs convolution processing, global pooling processing, and weighting processing based on the third feature map to obtain the first feature map. The third convolutional layer performs convolution processing, global pooling processing, and weighting processing based on the first feature map to obtain the second feature map.

[0023] Preferably, based on the shooting distance of the anatomical region image, multiple convolutional kernels of a target size are selected from multiple convolutional kernels of different sizes set in the first convolutional layer, the second convolutional layer, and the third convolutional layer. The larger the shooting distance, the larger the size interval of the multiple convolutional kernels of the target size, and the multiple convolutional kernels of the target size include at least one convolutional kernel with a size of 1*1.

[0024] A fourth aspect of the present invention provides a tissue-assisted segmentation method, the method comprising: outputting a first segmentation result of an anatomical region image using a first module, wherein intermediate processing results of a second module and a third module respectively include a first feature map and a second feature map; determining a region of interest in the first feature map by the second module based on the first segmentation result and the second feature map, and adjusting at least some features in the region of interest; outputting a second segmentation result based on the adjusted first feature map; and updating the enhancement effect displayed in the display device based on the second segmentation result.

[0025] A fifth aspect of the present invention provides a computer device including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, it implements an image segmentation method as described in any embodiment of the present application.

[0026] A sixth aspect of the present invention provides a readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described image segmentation method or the steps of the above-described organization-assisted segmentation system.

[0027] The beneficial effects of this invention are as follows: The convolutional neural network input to the endoscopic image fuses the feature components within each channel by applying multi-kernel convolution and measuring the importance of each convolutional kernel. This results in finer-grained optimization of the representation features under the same channel dimension. At the same time, the importance of different convolutional kernels also represents the availability of different receptive fields under different shooting scenarios. The final fused feature map can better represent the semantic information of endoscopic images under different shooting scenarios. When used for image segmentation, it requires fewer network parameters to be calculated compared to models such as Transformer, i.e., it has higher real-time performance; and the features optimized by models such as Unet are more refined and comprehensive, i.e., it has higher accuracy.

[0028] Meanwhile, after the endoscopic image is input into the deep learning model, a sequentially connected multi-layer model is set up. The first module passes the lower-resolution first segmentation result to the second module (result passing, not feature passing) to help the second module determine the region of interest of the target tissue in the first feature map. At the same time, the higher-resolution second feature map is also passed to the second module, which also helps to determine the region of interest. With the help of three different resolutions, the determined region of interest is more accurate. The prediction of the second segmentation result based on the first feature map of the region of interest is also more accurate, while avoiding more computation. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the first embodiment of the image segmentation model in this invention;

[0030] Figure 2 This is a schematic diagram of the second embodiment of the image segmentation model in this invention;

[0031] Figure 3 This is a schematic diagram of the first embodiment of the image segmentation method in this invention;

[0032] Figure 4 This is a schematic diagram of an embodiment of the convolutional layer processing of endoscopic images in this invention;

[0033] Figure 5 This is a schematic diagram of the second embodiment of the image segmentation method in this invention;

[0034] Figure 6 This is a schematic diagram of the third embodiment of the image segmentation method in this invention;

[0035] Figure 7 This is a schematic diagram of the third embodiment of the image segmentation model in this invention;

[0036] Figure 8 This is the first flowchart of the tissue-assisted segmentation system in this invention;

[0037] Figure 9 This is the second flowchart of the tissue-assisted segmentation system in this invention;

[0038] Figure 10 This is a schematic diagram of the first embodiment of region-of-interest enhancement in this invention;

[0039] Figure 11 This is a schematic diagram of the second embodiment of the region of interest enhancement in this invention;

[0040] Figure 12 This is a schematic diagram of the third embodiment of the region of interest enhancement in this invention;

[0041] Figure 13 This is a schematic diagram of the fourth embodiment of the region of interest enhancement in this invention;

[0042] Figure 14 This is a schematic diagram of an embodiment of the tissue-assisted segmentation system of the present invention;

[0043] Figure 15 This is a schematic diagram of the fourth embodiment of the image segmentation method in this invention. Detailed Implementation

[0044] To facilitate understanding of the present invention, a more detailed description is provided below with reference to the accompanying drawings and specific embodiments. Preferred embodiments of the invention are shown in the drawings. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a thorough and complete understanding of the disclosure of the invention.

[0045] It should be noted that, unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. For example, the term "a plurality of" includes two or more.

[0046] To facilitate understanding, the image segmentation model used in the image segmentation method of this invention will be introduced below. Please refer to [link / reference]. Figure 1 The first embodiment of the image segmentation model in this invention will be described below:

[0047] like Figure 1 As shown, the image segmentation model includes at least an encoder 10 and a decoder 11; the encoder 10 includes at least one convolutional layer (such as...). Figure 1 The three convolutional layers 100 shown include multiple convolutional kernels (such as...). Figure 1 The three convolutional kernels 101 shown are convolutional unit 102 and pooling unit (e.g., ...). Figure 1The diagram shows three pooling units 103 and a fusion unit 104; the convolution unit 102 is used to: acquire an endoscope image; perform convolution processing on the single-channel input of the endoscope image using multiple convolution kernels 101 to obtain multiple channel feature components; the pooling unit 103 is used to: determine the importance of each convolution kernel based on the feature components in each channel; the fusion unit 104 is used to: fuse the feature components in each channel based on the importance to obtain a feature map of the single-channel output of the convolutional layer; the decoder 11 is used to: perform image segmentation on the feature map to obtain a segmentation result.

[0048] In this embodiment, a single-channel input to the endoscopic image refers to the encoded information of one input channel of the endoscopic image input to the convolutional layer. Then, the encoded information of each input channel is convolved by multiple convolutional kernels, which is the multi-kernel convolution (MKconv) process. For example, the grayscale image of the endoscopic image is a single-channel input, corresponding to one input channel of the convolutional layer; the red, green, and blue primary color images of the endoscopic image are three single-channel inputs, corresponding to three input channels of the convolutional layer.

[0049] In this embodiment, the feature map output by a single channel is the feature map output by one output channel of the convolutional layer, which fully represents the features of the endoscopic image in one semantic dimension. For example, feature maps representing contours, grayscale, or contrast are all single-channel output feature maps. Intra-channel feature components refer to multiple sets of constituent parameters that form the feature map, representing the features of a certain semantic dimension of the endoscopic image under different receptive fields. For example, multiple sets of grayscale parameters representing receptive fields from small to large, with multiple sets of grayscale parameters forming a feature map.

[0050] In this embodiment, the importance of each convolutional kernel can be obtained by performing global pooling on the feature components in each channel, such as global average pooling or global max pooling. Each feature component in each channel is assigned a weight value, and then the feature components in each channel are enhanced by the corresponding weight value to obtain the feature map of the output channel.

[0051] In addition, pooling in convolutional neural networks typically includes two applications: 1) pooling the feature maps of multiple output channels in a convolutional layer separately, and then weighting and expanding the multiple feature maps based on the pooling results to achieve the fusion of multiple channels; 2) performing pooling on the feature maps output from multiple channels in the last convolutional layer separately, and setting the pooling results as the weights of each node in the fully connected layer to replace the fully connected layer. Both applications add pooling layers outside the output channels of the convolutional layer; both applications are based on out-of-channel pooling, measuring the overall importance of the feature maps and comparing the semantic importance of multiple output channels. In this embodiment, the pooling unit is added within the output channels of the convolutional layer, providing a new application of pooling layers. On the one hand, the feature processing is more refined (feature optimization is more granular), and on the other hand, it can be used to measure the importance of the convolutional kernel and perform the fusion of feature components within each channel, so that the fused feature map can more accurately represent the semantic information of the endoscopic image in that output channel.

[0052] For example, a certain output channel represents the contour dimension of an endoscopic image. After convolution with three kernels of sizes 1*1, 3*3, and 5*5, three feature components within the channel are obtained, which respectively represent the contour dimension of the endoscopic image under three receptive fields. In this case, the role of intra-channel pooling is to measure the importance of the three receptive fields under the same contour dimension. However, if it is an existing usage, such as a convolutional layer having three output channels, representing the contour dimension, grayscale dimension, and contrast dimension of the endoscopic image respectively, then the role of extra-channel pooling is to measure the importance under three different dimensions.

[0053] Based on this, since endoscopic images often capture some complex and abnormal surgical scenarios (such as instrument obstruction, tissue deformation, bleeding, etc.), the channel pooling operation in this embodiment can fuse the channel feature components of multiple receptive fields, which is more conducive to the perception of endoscopic images in these surgical scenarios.

[0054] Please refer to Figure 2 The second embodiment of the image segmentation model in this invention will be described below:

[0055] like Figure 2As shown, the encoder 20 includes n sequentially connected convolutional layers 200, each convolutional layer 200 outputting feature maps of different resolutions, where n is a positive integer; the decoder 21 includes n-1 sequentially connected network modules, used to generate and output segmentation masks 211 of different resolutions based on the feature maps output by each convolutional layer 200; wherein each network module includes an upsampling layer 212, a fusion layer 213, a residual block 214, and a fully connected layer 215 interconnected; the i-th convolutional layer is also connected to the i-th convolutional layer... The upsampling layer connection of i network modules and the fusion layer connection with the (i-1)th network module are used for: when i ≠ 1, sending the feature map to the upsampling layer of the i-th network module and back to the fusion layer of the (i-1)th network module; when i = 1, sending the feature map to the upsampling layer of the i-th network module; wherein, the i-th convolutional layer is counted according to the reverse connection order, and the i-th network module is counted according to the forward connection order, i ∈ n; the upsampling layer of the i-th network module is used for: The received feature map is upsampled to obtain an upsampled feature map, and the upsampled feature map is sent to the fusion layer of the i-th network module. The fully connected layer of the (i-1)-th network module is also connected to the fusion layer of the i-th network module, and is used to output the segmentation mask and pass it to the fusion layer of the i-th network module when i≠1. The fusion layer of the i-th network module is used to: fuse the received feature map, the upsampled feature map, and the segmentation mask when i≠1 to obtain a first fused feature map; and fuse the received feature map and the upsampled feature map when i=1 to obtain a second fused feature map, and send the first fused feature map or the first fused feature map to the residual block of the i-th network module. The residual block of the i-th network module is used to: perform image segmentation on the received first fused feature map or the second fused feature map, and generate a segmentation mask through the fully connected layer of the i-th network module. The segmentation mask generated by the fully connected layer of the (n-1)-th network module is output as the final segmentation result.

[0056] In the encoder, the convolutional order of multiple convolutional layers is {n, n-1, ..., i+1, i, i-1, ..., 1}, which represents the forward connection order of the convolutional layers, and the feature maps are output sequentially from largest to smallest resolution. When the decoder performs image segmentation, it outputs feature maps sequentially according to the reverse connection order of the convolutional layers: {1, ..., i-1, i, i+1, ..., n-1, n}. This reverse connection order of the convolutional layers ({1, ..., i-1, i, i+1, ..., n-1, n}) is also the forward connection order of multiple network modules.

[0057] For example, if i=2, the fusion module will receive the feature map input from the third convolutional layer, the upsampled feature map input from the second upsampled layer, and the segmentation mask input from the first fully connected layer, and generate a fused feature map. At this time, the fused feature map can subsequently optimize the second upsampled feature map in two levels: the first level is to enhance the local features of the segmented region of the upsampled feature map at the current resolution based on the lower resolution segmentation mask; the second level is to enhance the single-point features of the segmented region of the upsampled feature map at the current resolution based on the higher resolution feature map.

[0058] In this embodiment, based on the aforementioned two-level optimization of the upsampled feature map, the first level is used to define the approximate segmentation region in the upsampled feature map, thereby narrowing the detection range of the upsampled feature map segmentation; the second level is used to highlight single-point features in the upsampled feature map that are more likely to belong to the segmentation region, thereby improving the feature differentiation between the segmented and non-segmented regions. This two-level optimization of the upsampled feature map improves the segmentation accuracy when used to generate a segmentation mask at the current resolution.

[0059] When i=1, since there is no i-1th network module, there is no connection between the i-th convolutional layer and the fusion layer of the i-1th network module, nor is there a connection between the fully connected layer of the i-1th network module and the fusion layer of the i-th network module.

[0060] Furthermore, the fusion algorithms performed by the fusion layer in the decoder and the fusion units in the convolutional layer are different. The fusion layer expands the number of channels, ensuring that the fused feature map's feature values ​​remain unchanged relative to the original feature map, upsampled feature map, and segmentation mask. The fused feature map is simply the sum of the channel numbers of these three. For example, if the feature map size is H*W*C1, the upsampled feature map size is H*W*C2, and the segmentation mask size is H*W*1, then the fusion layer, after fusing these three, obtains a fused feature map of size H*W*(C1+C2+1). The fusion unit, on the other hand, weights the feature values ​​of the feature components within each channel, causing the output feature map of that channel to change relative to the original feature values ​​of the feature components within each channel. Additionally, the size of each feature component within a channel is H*W*1, and after fusion by the fusion unit, the size of the feature map output by that single channel remains H*W*1.

[0061] In a preferred embodiment, the encoder of the image segmentation model includes 5 convolutional layers, and the decoder includes 4 network modules. Each convolutional layer may also be configured with a BN (Batch Normalization) layer, an activation layer (ReLU (linear rectification function)), and a pooling layer (with a convolutional kernel size of 2*2, edge expansion of 0, stride of 2, and providing an average pooling function).

[0062] Specifically, the single-channel input size of the endoscopic image is 1920*1024*1 or 1920*1024*3. The output feature size of the first convolutional layer is 960*512*32; the output feature size of the second convolutional layer is 480*256*64; the output feature size of the third convolutional layer is 240*128*128; the output feature size of the fourth convolutional layer is 120*64*256; and the output feature size of the fifth convolutional layer is 60*32*512. An input layer can be set before the convolutional layers to encode the endoscopic image, obtaining at least one single-channel input with a size of 1920*1024.

[0063] Specifically, each convolutional layer includes three convolutional kernels with sizes of 1*1, 3*3, and 5*5. The outputs of the three convolutional kernels are connected to three pooling units, and the three pooling units are connected to one fusion unit. The pooling units provide a global average pooling function, and the fusion unit provides a weighting function.

[0064] Specifically, in each network module, the upsampling layer provides an interpolation function to expand the scale of the input image to twice its original size; the fusion layer provides an expansion function "concat()" and performs feature fusion at twice the scale of the original input image; the segmentation mask output by the fully connected layer is input into the fusion layer after upsampling processing.

[0065] In the first network module, the upsampling layer is connected to the fifth convolutional layer, expanding the feature map resolution from 60*32 to 120*64. The fully connected layer outputs a segmentation mask with a resolution of 120*64. In the second network module, the upsampling layer is connected to the fourth convolutional layer, expanding the feature map resolution from 120*64 to 240*128. The segmentation mask resolution input to the fusion layer is upsampled from 120*64 to 240*128, and the fully connected layer outputs a segmentation mask with a resolution of 240*128. This process continues until the fourth network module's fully connected layer outputs a segmentation mask with a resolution of 960*512.

[0066] The image segmentation model in this invention has been described above. The image segmentation method in this invention will be described below. Please refer to [link / reference]. Figure 3 The first embodiment of the image segmentation method is provided below:

[0067] 31. Acquire an endoscope image and input the endoscope image into a convolutional neural network, wherein the convolutional layer of the convolutional neural network is provided with multiple convolutional kernels;

[0068] In this embodiment, the endoscopic image refers to a real-time image obtained by an endoscope, specifically an image of the target tissue taken after the endoscope is inserted into the body. The endoscopic image includes at least two parts: a foreground part of the target tissue area and a background part of the non-target tissue area. A convolutional neural network is used to identify the foreground part and segment it from the background part, forming a segmentation mask for the foreground part.

[0069] In this embodiment, the convolutional neural network, when segmenting the foreground and background, includes at least two image processing steps: one is extracting features from the endoscopic image, and the other is analyzing the features to distinguish between the foreground and background. The convolutional layers used to extract features from the endoscopic image are limited by the single receptive field of traditional single convolutional kernels, and the additional computational cost required to compensate for this limited receptive field, resulting in low feature extraction efficiency. Therefore, this embodiment provides multiple receptive fields using multiple convolutional kernels of different sizes, making feature extraction from the endoscopic image more efficient and precise.

[0070] For example, multiple convolutional kernels may include three convolutional kernels with sizes of 1*1, 3*3, and 5*5, or three convolutional kernels with sizes of 1*1, 5*5, and 9*9, or three convolutional kernels with sizes of 3*3, 5*5, and 7*7.

[0071] 32. The single-channel input of the endoscope image is convolved using multiple convolution kernels to obtain multiple feature components within each channel.

[0072] In this embodiment, multiple convolutional kernels perform convolution processing on the single-channel input of the endoscope image. This means that for each input channel of the convolutional layer, multiple convolutional kernels are used to perform feature extraction. For example, if the endoscope has a single-channel input of a grayscale image, then there are 3 convolutional kernels that will perform convolution processing on the grayscale image respectively, resulting in 3 feature components within the corresponding channels. If the endoscope has a single-channel input of three primary color images (red, green, and blue), then there are 3 sets of convolutional kernels (each set of convolutional kernels includes 3 convolutional kernels, for a total of 9 convolutional kernels) that will perform convolution processing on the 3 primary color images respectively, resulting in 3 feature components within the channels for each primary color image. The convolutional layer includes 9 feature components within the channels.

[0073] Specifically, convolutional kernels of different sizes extract the same single-channel input based on different receptive fields, resulting in intra-channel feature components with different fine-grained characteristics. The intra-channel feature components extracted by convolutional kernels of decreasing size are also characterized in descending order of feature fine-grainedness. Features of the foreground portion in a single-channel input are more roughly represented when extracted using a larger-sized convolutional kernel, such as the approximate range of the foreground portion. When extracted using a smaller-sized convolutional kernel, they are more finely represented, even down to a single pixel, such as the precise feature location of the foreground portion.

[0074] In this embodiment, for complex endoscopic imaging scenarios within the body, convolutional kernels of different sizes exhibit different image perception performance for different imaging scenarios, thereby improving the convolutional layer's ability to perceive endoscopic images under different imaging scenarios and more accurately representing the semantic features of endoscopic images. For example, when facing imaging scenarios such as instruments obscuring target tissue, target tissue deformation, or bleeding, convolutional processing performed with larger, smaller, and larger convolutional kernels respectively can achieve better image perception performance.

[0075] 33. Determine the importance of each convolutional kernel based on the feature components in each channel, and fuse the feature components in each channel based on the importance to obtain the feature map of the single-channel output of the convolutional layer;

[0076] In this embodiment, as previously described, the perception performance of convolutional kernels of different sizes varies for endoscopic images under different shooting scenarios. Here, the importance of each convolutional kernel is measured not only by the availability of different receptive fields, but also by the strength of the perception performance of different convolutional kernels for the endoscopic images captured in the current scene. For example, based on performing global pooling processing on the feature components within each channel, the importance of different convolutional kernels is measured from the global perspective of the feature components within each channel.

[0077] In this embodiment, the feature components within a channel are fused based on their importance. For the shooting scene, the feature components extracted by the convolution kernel with strong perception and a large receptive field are relatively enhanced, while the feature components extracted by the convolution kernel with weak perception and a small receptive field are relatively weakened. This makes the feature map of the final fused single-channel output more accurately represent the semantic information of the single-channel input and better distinguish between the foreground and background.

[0078] For example, such as Figure 4As shown, for a single-channel input 41 of an endoscopic image, convolutional kernels 421 (size 1*1), 422 (size 2*2), and 423 (size 3*3) are used to perform convolution processing on the single-channel input 41, respectively, to obtain intra-channel feature components 431, 432, and 433. Global pooling is then performed on the intra-channel feature components 431, 432, and 433 to obtain feature values ​​441, 442, and 443, respectively. Finally, the feature values ​​441, 442, and 443 are used for weighted fusion of the intra-channel feature components 431, 432, and 433 to obtain the feature map 45 of one channel output of the convolutional layer.

[0079] 34. Perform image segmentation on the feature map to obtain the segmentation result.

[0080] In this embodiment, the convolutional neural network inputting the endoscopic image fuses the feature components within each convolutional channel by applying multi-kernel convolution and measuring the importance of each convolutional kernel. This results in more fine-grained optimization of the same channel dimension. Furthermore, the importance of different convolutional kernels represents the availability of their receptive fields and image perception capabilities under different shooting scenarios. The resulting fused feature map better represents the semantic information of the endoscopic image under different shooting scenarios, requiring fewer network parameters for image segmentation and achieving higher real-time segmentation performance. The optimized semantic features are more refined and comprehensive, leading to higher accuracy. The final output image segmentation result balances higher efficiency and higher accuracy.

[0081] Please see Figure 5 A second embodiment of the image segmentation method is provided, as follows:

[0082] 51. Acquire an endoscope image and input the endoscope image into a convolutional neural network, wherein the convolutional layer of the convolutional neural network is provided with multiple convolutional kernels;

[0083] 52. The single-channel input of the endoscope image is convolved using multiple convolution kernels to obtain multiple feature components within each channel.

[0084] 53. Perform global pooling on the feature components in each channel respectively, and obtain the importance of each convolution kernel based on the result of global pooling, wherein the global pooling includes global average pooling and global max pooling;

[0085] In this embodiment, when calculating the importance of feature components within each channel, it is represented by their own global pooling results, choosing between global average pooling and global max pooling. Global average pooling focuses more on regional information, preserving more background information of the image, and exhibits better enhancement effects for small or large-scale changes in target structure; global max pooling, on the other hand, focuses more on single-point information, preserving more texture information of the image, and exhibits better enhancement effects for large or small-scale changes in target structure.

[0086] Specifically, when calculating global average pooling or global max pooling for the feature components within a channel, a feature value is calculated for each feature component within that channel. Multiple feature values ​​are then normalized to map to a numerical range of [0,1]. The importance of each convolutional kernel is determined by the mapped values. For example, after global average pooling, the feature components within three channels yield feature values ​​a, b, and c. Then, a / (a+b+c), b / (a+b+c), and c / (a+b+c) are normalized to obtain three mapped values ​​within the numerical range of [0,1].

[0087] 54. The result of global pooling is converted into weights, and the feature components in each channel are weighted to obtain the enhanced feature components in each channel. The weights are used to measure the image perception intensity of each convolution kernel in the current endoscope shooting scene.

[0088] 55. Add the feature components in each enhanced channel to obtain the feature map of the single-channel output of the convolutional layer;

[0089] In this embodiment, the normalized mapping values ​​after global pooling are used as weights. The magnitude of the weights determines the image perception intensity and receptive field availability of each convolutional kernel in the current endoscopic imaging scenario. Weights are applied to enhance the feature components within each channel, thus superimposing the image perception intensity and receptive field availability of the convolutional kernel onto the feature components within each channel. Therefore, more important feature components within each channel are relatively amplified, while less important feature components are relatively reduced. Finally, the enhanced feature components within each channel are summed, superimposing the influence of image perception intensity and receptive field onto the output channel, making the semantic information represented by the feature map of a single channel more accurate.

[0090] Specifically, when fusing feature components within each channel based on their importance, the aforementioned mapping value can be multiplied by each feature value of the corresponding feature component within that channel, and then the feature components within the channel can be summed to obtain the feature map of one output channel of the convolutional layer. A larger mapping value can relatively enhance the feature component within the corresponding channel, while a smaller mapping value can relatively weaken the feature component within the corresponding channel.

[0091] 56. Perform image segmentation on the feature map to obtain the segmentation result.

[0092] In this embodiment, the feature map output by the convolutional layer for each channel incorporates the image perception intensity and receptive field of each convolutional kernel in its output channel. This allows for better perception of the target tissue region in the foreground and the background in different endoscopic imaging scenarios, such as instrument obstruction, tissue deformation, and bleeding. When performing image segmentation on the feature map, the foreground and background are detected and segmented more accurately.

[0093] Please see Figure 6 A third embodiment of the image segmentation method is provided, as follows:

[0094] 61. Acquire an endoscope image and input the endoscope image into a convolutional neural network, wherein the convolutional layer of the convolutional neural network is provided with multiple convolutional kernels;

[0095] 62. The single-channel input of the endoscope image is convolved using multiple convolution kernels to obtain multiple feature components within each channel.

[0096] 63. Determine the importance of each convolutional kernel based on the feature components in each channel, and fuse the feature components in each channel based on the importance to obtain the feature map of the single-channel output of the convolutional layer;

[0097] 64. Perform image segmentation on the lower-resolution feature map to obtain a lower-resolution segmentation mask;

[0098] In this embodiment, the convolutional neural network includes multiple convolutional layers, each outputting a feature map at a different resolution. Further, at least three target convolutional layers are selected from the multiple convolutional layers of the convolutional neural network for final convolutional processing, and the output feature maps of these at least three target convolutional layers are obtained for image segmentation. The purpose is to utilize image information from at least three different resolutions to perform image segmentation, detecting foreground and background regions at different levels of detail, thereby improving the accuracy of image segmentation.

[0099] In this embodiment, when performing one round of image segmentation, since three feature maps of different resolutions are needed, the lower resolution, higher resolution, and current resolution are all concepts defined relative to these three different resolutions. The lower resolution feature map / segmentation mask refers to the feature map / segmentation mask with the lowest resolution among the three resolutions.

[0100] For example, when performing a round of image segmentation, feature maps with resolutions of 960*512, 480*256, and 240*128 are needed. The lower-resolution feature map is the 240*128 feature map. If the lower-resolution feature map is the lowest-resolution feature map among all feature maps, then the lower-resolution feature map is fused with the current-resolution feature map, and image segmentation is performed. That is, the 480*256 and 240*128 feature maps are fused, and image segmentation is performed to obtain a segmentation mask with a resolution of 240*128.

[0101] 65. The lower resolution segmentation mask is transmitted and the higher resolution feature map is transmitted back. The transmitted segmentation mask, the transmitted feature map, and the current resolution feature map are fused together to obtain a fused feature map.

[0102] As mentioned above, lower resolution, higher resolution, and current resolution are all concepts defined relative to these three different resolutions. In this embodiment, the higher resolution feature map refers to the highest resolution feature map among the three resolutions, and the current resolution is the resolution used for image segmentation in this round.

[0103] For example, when performing a certain round of image segmentation, feature maps with resolutions of 960*512, 480*256, and 240*128 are needed. The lower resolution is 240*128, the higher resolution is 960*512, and the current resolution is 480*256. Therefore, the segmentation mask with a resolution of 240*128 is passed, and the feature map with a resolution of 960*512 is passed back. Then, the feature map with a resolution of 480*256 is fused with the segmentation mask with a resolution of 240*128 and the feature map with a resolution of 960*512 to obtain a fused feature map with a resolution of 480*256.

[0104] Furthermore, the lower-resolution 240*128 feature map can be upsampled and image segmentation can be performed to obtain a segmentation mask with a resolution of 480*256. The 480*256 segmentation mask can then be upsampled again to obtain a lower-resolution 960*512 segmentation mask. Alternatively, the current 480*256 feature map can be upsampled to obtain a current 960*512 feature map. Finally, the lower-resolution 960*512 segmentation mask, the current 960*512 feature map, and the higher-resolution 960*512 feature map can be fused to obtain a fused feature map with a resolution of 960*512.

[0105] If the feature map at the current resolution is the single-channel output of the first of the aforementioned at least three target convolutional layers, then there is no lower-resolution feature map, and therefore no lower-resolution segmentation mask (i.e., no transmitted segmentation mask). In this case, it is only necessary to fuse the feature map at the current resolution and the transmitted feature map to obtain the fused feature map.

[0106] If the feature map at the current resolution belongs to the single-channel output of the last of the aforementioned at least three target convolutional layers, then there is no higher-resolution feature map (i.e., no backpropagated feature map). In this case, it is only necessary to fuse the current feature map and the passed segmentation mask to obtain the fused feature map; or no further feature fusion and image segmentation are required.

[0107] 66. Perform image segmentation on the fused feature map to obtain a segmentation mask at the current resolution, wherein when the current resolution meets the preset resolution, the segmentation mask at the current resolution is used as the segmentation result.

[0108] In a preferred embodiment, such as Figure 7As shown, the convolutional layer outputs feature maps 711, 712, 713, 714, and 715 with resolutions of 60*32, 120*64, 240*128, 480*256, and 960*512, respectively. If image segmentation is performed on the feature map with the current resolution of 120*64, the lower resolution is set to 60*32 and the higher resolution is set to 240*128. Then, in the fusion layer, the segmentation mask 721 with the lower resolution of 60*32 and the feature map 713 with the higher resolution of 240*128 are fused into the feature map 712 with the current resolution of 120*64 to obtain the fused feature map. Then, foreground detection is performed on the fused feature map in the residual block, and the foreground and background parts are segmented in the fully connected layer based on the detection results, and the segmentation mask 722 at the current resolution is output. Set 240*128 and 480*256 to the current resolution in sequence, and output the corresponding segmentation mask 723 and segmentation mask 724 in sequence.

[0109] Since feature map 711 has the lowest resolution among the five features, if the current resolution is set to 60*32, there is no lower resolution relative to 60*32. In this case, in the fusion layer, the feature map 712 with a higher resolution of 120*64 is fused to the feature map 711 with the current resolution of 60*32 to obtain the fused feature map. After foreground detection of residual blocks and foreground segmentation of fully connected layers, the segmentation mask 721 is output.

[0110] The preset resolution is set to the second highest resolution, such as feature map 714 being the second highest resolution among the five features. If the current resolution is set to 480*256, the output segmentation mask 724 is used as the segmentation result. Feature map 715, with a maximum resolution of 960*512, is only used in the fused feature map 714 to perform image segmentation at a resolution of 480*256, and does not need to perform image segmentation steps on its own.

[0111] Furthermore, an upsampling layer can be added before the fusion layer to make the fused feature map twice the resolution of the original feature map, and the output segmentation mask also twice the resolution of the original feature map. For example, feature map 711 has a resolution of 60*32, and after the upsampling layer, the output feature map has a resolution of 120*64; after the fusion layer, the output fused feature map has a resolution of 120*64; finally, the output segmentation mask 721 has a resolution of 120*64.

[0112] Please see Figure 8 The first process embodiment of the organization-assisted segmentation system is provided below:

[0113] 81. The first module outputs the first segmentation result of the anatomical region image, and the intermediate processing results of the second module and the third module respectively include the first feature map and the second feature map;

[0114] In this embodiment, the tissue-assisted segmentation system includes: a surgical instrument, wherein an image acquisition device is deployed at the end of the surgical instrument for acquiring images of the anatomical region; a display device for displaying enhancement effects on the images of the anatomical region; and a processor, wherein a deep learning model is deployed on the processor, and the deep learning model includes a first module, a second module, and a third module connected in sequence.

[0115] Specifically, surgical instruments are inserted into the patient's body through natural cavities or incisions to remove the target diseased tissue. Based on real-time images of the anatomical region acquired by the image acquisition device, a deep learning model deployed in the processor automatically identifies the cutting edge of the target tissue and renders the cutting edge onto the anatomical image displayed on the display device, resulting in an enhanced effect. The patient can more accurately see and remove the target tissue based on the displayed cutting edge.

[0116] In this embodiment, the deep learning model includes multiple modules, each generating feature maps of different resolutions. Further, three modules are selected from the deep learning model at each stage to perform tissue segmentation of the anatomical region image. The purpose is to utilize three feature maps of different resolutions to perform tissue segmentation, detecting foreground and background regions at varying degrees of detail, thereby improving the accuracy of tissue segmentation. The generated feature maps, from smallest to largest resolution, include a third feature map, a first feature map, and a second feature map. The three modules used for generating these feature maps are the first module, the second module, and the third module, respectively. Furthermore, the first module also outputs a first segmentation result based on the third feature map.

[0117] For example, when performing a certain round of tissue segmentation, the first module, the second module, and the third module output feature maps with resolutions of 960*512, 480*256, and 240*128 respectively. Then the resolution of the first feature map is 480*256, and the resolution of the second feature map is 960*512.

[0118] 82. Based on the first segmentation result and the second feature map, the second module determines the region of interest in the first feature map and adjusts at least some features in the region of interest;

[0119] In this embodiment, based on the first segmentation result, the approximate range of the target tissue can be determined in the first feature map, and the feature points within this approximate range of the first feature map can be adjusted. Based on the second feature map, the probability of each feature point belonging to the target tissue can be determined in the first feature map, and each feature point can be adjusted to different degrees based on the probability, further enhancing the differentiation between the foreground and background parts in the first feature map. The approximate range in the first feature map and each feature point with a higher probability of belonging to the target tissue constitute the region of interest.

[0120] In the second feature map, if the feature value is greater than or equal to a preset threshold, the feature point is classified as "a feature point that is likely to belong to the target organization" and belongs to the foreground part of the first feature map; if the feature value is less than the preset threshold, the feature point is classified as "a feature point that is likely to belong to the target organization" and belongs to the background part of the first feature map.

[0121] 83. The second module outputs a second segmentation result based on the adjusted first feature map, and updates the enhancement effect based on the second segmentation result.

[0122] In this embodiment, based on the adjusted first feature map, the foreground part can be perceived more accurately. When the second module predicts the target tissue based on the second feature map, the predicted second segmentation result is more accurate, that is, the cutting edge is more accurately divided in the anatomical region image.

[0123] In this embodiment, among the multiple modules included in the deep learning network, if the third module is the last module in the previous round of tissue segmentation, the cutting edge is directly rendered on the anatomical region image displayed on the display device based on the second segmentation result, and the effect is updated and enhanced.

[0124] In this embodiment, among the multiple modules included in the deep learning network, if the third module is not the last module in the current round of organizational segmentation, then the second module in the current round of organizational segmentation is used as the first module in the next round of organizational segmentation, and the third module in the current round of organizational segmentation is used as the second module in the next round of organizational segmentation. The second segmentation result generated in the current round of organizational segmentation is used as the first segmentation result in the next round of organizational segmentation. This process continues until the third module in a certain round of organizational segmentation belongs to the last module.

[0125] It should be noted that, in addition to rendering the cutting edges, the enhancement effect of anatomical region images can also be to increase the contrast and color of the target tissue and the background.

[0126] Please see Figure 9 The second embodiment of the organization-assisted segmentation system is provided below:

[0127] 91. Determine a first segmentation mask based on the first segmentation result, and determine a second segmentation mask based on the second feature map, wherein the area covered by the first segmentation mask is larger than the area covered by the second segmentation mask;

[0128] 92. Map the first segmentation mask and the second segmentation mask onto the first feature map to obtain the region of interest in the first feature map;

[0129] For example, such as Figure 10 As shown, the first segmentation mask 101 can determine the initial segmentation range 1011 of the foreground part (target tissue), and the second segmentation mask 102 can determine each feature point (such as feature point 1021) belonging to the foreground part at a finer granularity. Then, the positions of the initial segmentation range 1011 and feature point 1021 are mapped to the first feature map 103, and the region of interest (such as region 1031 and feature point 1032) of the foreground part is indicated in the first feature map 103, so that the feature points that are more likely to belong to the foreground part are enhanced.

[0130] In one embodiment, the region of interest includes a first region of interest mapped only by the first segmentation mask (e.g., feature point 1033 belongs to the first region of interest), a second region of interest mapped only by the second segmentation mask (e.g., feature point 1034 belongs to the second region of interest), and a third region of interest mapped by both the first segmentation mask and the second mask (e.g., feature point 1032 belongs to the third region of interest), and at least satisfies the following: both the first region of interest and the third region of interest are larger than the second region of interest, which means that the preset and extracted feature points of the foreground portion in the first segmentation result, the first feature map, and the second feature map are similar.

[0131] 93. Increase the feature value in the first region of interest by a preset first amplitude, increase the feature value in the second region of interest by a preset second amplitude, and / or increase the feature value in the third region of interest by a preset third amplitude.

[0132] The preset third amplitude is greater than the preset second amplitude and the preset first amplitude.

[0133] For example, such as Figure 11As shown, only the third region of interest in the first feature map 103 (such as feature point 1032 contained in region 1031) is enhanced, increasing its feature value to obtain feature points as shown in the first feature map 111 (corresponding to feature point 1111). For feature points in the first region of interest (such as feature point 1033) and feature points in the second region of interest (such as feature point 1035), the feature points shown in feature map 101 (corresponding to feature points 1112 and 1113) retain their original feature values.

[0134] Similarly, feature values ​​of at least one of the first, second, and third regions of interest can be added. When adding feature values ​​of at least one of the first and second regions of interest, as well as adding feature values ​​of the third region of interest, i.e. adding feature values ​​of at least two regions containing the third region of interest, the third region of interest is simultaneously mapped by the first and second segmentation masks. Therefore, the third magnitude of the increase in feature values ​​of the third region of interest is greater than the second and first magnitudes.

[0135] In one implementation, such as Figure 12 As shown, feature points in the first region of interest (e.g., feature point 1033) of feature map 103 are enhanced by a first amplitude to obtain feature points as shown in the enhanced feature map 121 (corresponding to feature point 1211); feature points in the second region of interest (e.g., feature point 1034) of feature map 103 are enhanced by a second amplitude to obtain feature points as shown in feature map 121 (corresponding to feature point 1212); and feature points in the third region of interest (e.g., feature point 1032) of feature map 103 are enhanced by a third amplitude to obtain feature points as shown in feature map 121 (corresponding to feature point 1213). Other feature points in feature map 93 (e.g., feature point 1035) do not require enhancement, and feature points as shown in feature map 121 (corresponding to feature point 1214) retain their original feature values.

[0136] Specifically, adding feature values ​​can be done by increasing the original feature values ​​according to preset feature values, or by weighting the original feature values ​​according to preset weights (such as a preset weight range of [1.1-1.5]). Here, the first amplitude, the second amplitude, and the third amplitude refer to the magnitude of the increase in feature values, or the percentage increase.

[0137] In another implementation, the first segmentation mask is mapped onto the first feature map to obtain a continuous set of feature points, and the second segmentation mask is mapped onto the first feature map to obtain a discrete set of feature points; based on the continuous set of feature points and the discrete set of feature points, a region of interest is determined in the first feature map, wherein the continuous set of feature points includes at least a portion of the feature points in the discrete set of feature points.

[0138] In this embodiment, as Figure 10 As shown, when the first segmentation mask 101 is mapped onto the first feature map 103, a complete tissue contour region is formed, and all feature points are continuous. Therefore, a continuous feature point set is obtained after mapping. When the second segmentation mask 102 is mapped onto the first feature map 103, there are multiple discrete feature points. Not all feature points are continuous. Therefore, a discrete feature point set for each discrete position is obtained after mapping.

[0139] In one implementation, the number of consecutive feature points at each discrete location in the discrete feature point set is determined, and feature points at discrete locations where the number of consecutive feature points exceeds a preset first threshold are selected; target feature points belonging to the continuous feature point set among the selected feature points at discrete locations are determined, and the feature value of the target feature points is increased.

[0140] Specifically, if the number of consecutive discrete points at the same discrete location exceeds a preset first threshold, it can be considered that the probability of that discrete location belonging to the target organization is relatively high. Therefore, the feature values ​​of all target feature points at that discrete location can be increased. The preset first threshold can be xy / a of the second feature map resolution x*y and rounded down (e.g., (480*256) / (60*32)=64), or it can be a fixed value, such as 100.

[0141] In one implementation, the number of feature points mapped by the second segmentation mask to the same discrete feature point in the discrete feature point set is determined; discrete feature points whose number of mapped feature points exceeds a preset second threshold are determined, and the feature value of the determined discrete feature points is increased.

[0142] For example, such as Figure 13 As shown, if the second threshold is preset to 2, then discrete feature points in feature map 103 that are mapped by feature points of two second segmentation masks (such as mapped feature 1036) are enhanced to obtain feature points as shown in feature map 131 (corresponding to feature point 1311). For discrete feature points that are mapped by feature points of only one second segmentation mask (such as mapped feature 1032), then feature points as shown in feature map 131 (corresponding to feature point 1312) retain their original feature values.

[0143] Furthermore, for discrete feature points mapped by a feature point of a second segmentation mask, but which do not belong to the set of continuous feature points (such as mapped feature 1034), and are also feature points as shown in feature diagram 131 (corresponding to feature point 1313), their original feature values ​​are preserved.

[0144] Please see Figure 14 An embodiment of a modular structure for an organization-assisted segmentation system is provided, as shown below:

[0145] like Figure 14 As shown, the first module 141 includes a first convolutional layer 1411 and a second decoder 1412, the second module 142 also includes a second convolutional layer 1421, and the third module 143 includes a third convolutional layer 1431.

[0146] The first convolutional layer 1411, the second convolutional layer 1421, and the third convolutional layer 1431 are sequentially connected. When the anatomical region image passes through the first convolutional layer 1411, the second convolutional layer 1421, and the third convolutional layer 1431 in sequence, a third feature map, the first feature map, and the second feature map are output respectively.

[0147] The second decoder 1412 is connected to the first convolutional layer 1411 and the second convolutional layer 1421 respectively. The second decoder 1412 outputs the first segmentation result based at least on the third feature map output by the first convolutional layer 1411 and the first feature map output by the second convolutional layer 1421.

[0148] The first decoder 1422 is connected to the second decoder 1412, the second convolutional layer 1421 and the third convolutional layer 1431 respectively.

[0149] In this embodiment, if the second module belongs to the first module (e.g., module 144) among the multiple modules included in the deep learning network, then the first segmentation result output by the first module does not exist. In this case, the first decoder only needs to determine the region of interest based on the first feature map and the second feature map, and output the second segmentation result.

[0150] In a preferred embodiment, such as Figure 14 As shown, the first, second, and third convolutional layers output third feature maps, first feature maps, and second feature maps with resolutions of 60*32, 120*64, and 240*128, respectively. At this time, in the first decoder, the first segmentation result with a resolution of 60*32 and the second feature map with a resolution of 240*128 are fused into the first feature map with a resolution of 120*64 to obtain a fused feature map, and the second segmentation result is output based on the fused feature map.

[0151] like Figure 14As shown, the third feature map is at a preset first resolution, the first segmentation result and the first feature map are at a preset second resolution, and the second feature map is at a preset third resolution, wherein the preset first resolution is greater than the preset second resolution, and the preset second resolution is greater than the preset third resolution; the second decoder further includes a first upsampling layer, which is connected to the first convolutional layer to upsample the third feature map from the preset first resolution to the preset second resolution; the first decoder further includes a second upsampling layer, which is connected to the second convolutional layer to upsample the first feature map from the preset second resolution to the preset third resolution; a third upsampling layer is also connected between the second decoder and the first decoder to upsample the first segmentation result from the preset second resolution to the preset third resolution.

[0152] For example, if the first resolution of the third feature map is 60*32, after passing through the first upsampling layer, a third feature map with a second resolution of 120*64 is output. After being processed by the second decoder, a first segmentation result with a second resolution of 120*64 is output. After passing through the third upsampling layer, the first segmentation result with a second resolution of 120*64 is output as a first segmentation result with a third resolution of 240*128.

[0153] For example, if the second resolution of the first feature map is 120*64, after passing through the second upsampling layer, the output first feature map has a third resolution of 240*128. The first decoder fuses the first feature map with a third resolution of 240*128, the first segmentation result, and the second feature map to obtain a fused feature map with a resolution of 240*128, and outputs a second segmentation result with a resolution of 240*128.

[0154] like Figure 14As shown, the first convolutional layer 1411, the second convolutional layer 1421, and the third convolutional layer 1423 are each provided with multiple convolutional kernels 145 of different sizes and multiple global pooling units 146 connected to each convolutional kernel. Each convolutional kernel 145 in the first convolutional layer 1411 performs convolution processing on the anatomical region image to obtain multiple corresponding convolutional results. Each global pooling unit 146 in the first convolutional layer 1411 performs global pooling processing on the corresponding convolutional processing result to obtain multiple corresponding pooling results. The multiple pooling results are weighted to obtain the third feature map output by the first convolutional layer 1411. The second convolutional layer 1421 performs convolution processing, global pooling processing, and weighting processing based on the third feature map to obtain the first feature map. The third convolutional layer 1431 performs convolution processing, global pooling processing, and weighting processing based on the first feature map to obtain the second feature map.

[0155] Please see Figure 15 A fourth embodiment of the image segmentation method is provided, as detailed below:

[0156] 151. Acquire an endoscope image and input the endoscope image into a convolutional neural network, wherein the convolutional layer of the convolutional neural network is provided with multiple convolutional kernels;

[0157] 152. Detect the imaging distance of the endoscopic image;

[0158] 153. Select multiple convolution kernels corresponding to the shooting distance, and perform convolution processing on the single-channel input of the endoscope image respectively to obtain the feature components in the channel. The sizes of the multiple convolution kernels in different groups are different.

[0159] In this embodiment, the imaging distance of the endoscopic image refers to the distance from the endoscope to the object being imaged, such as the distance from the endoscope to the target tissue. A larger imaging distance indicates that the target tissue occupies a smaller proportion of the entire endoscopic image, while a smaller imaging distance indicates that the target tissue occupies a larger proportion of the entire endoscopic image. Based on the proportion of the target tissue in the entire endoscopic image, multiple convolutional kernels of preset size groups are selected to improve the feature extraction accuracy of the endoscopic image by these multiple convolutional kernels.

[0160] Specifically, if the shooting distance is relatively small, the outline of the target tissue will be displayed more clearly in the endoscopic image, and the contour texture will be more uneven. In this case, using a larger convolutional kernel group can extract more detailed contour features with a smaller receptive field, resulting in more precise detection and segmentation of the foreground part of the target tissue. Conversely, using a larger convolutional kernel group may easily lose features of contour areas with greater unevenness. If the shooting distance is relatively large, the outline of the target tissue will be smoother. In this case, using a larger convolutional kernel group can quickly extract smooth feature contours with a larger receptive field, reducing computational complexity.

[0161] Furthermore, the number of convolutional kernels in different groups is the same, but the sizes of the convolutional kernels in different groups are different. This means that each convolutional kernel in different groups is ordered by size, and there is at least one pair of convolutional kernels with the same order but different sizes. For example, the size of the first group of convolutional kernels is [1*1, 3*3, 5*5], the size of the second group of convolutional kernels is [1*1, 5*5, 9*9], and the size of the third group of convolutional kernels is [3*3, 5*5, 7*7]. All of these belong to "multiple convolutional kernels in different groups with different sizes". Among them, in every two convolutional kernels with the same order, one group of convolutional kernels is always greater than or equal to the other group of convolutional kernels.

[0162] It should be noted that the aforementioned deep learning model may include some or all of the modules in the convolutional neural network described here, and execute the program steps of each module.

[0163] 154. Determine the importance of each convolutional kernel based on the feature components in each channel, and fuse the feature components in each channel based on the importance to obtain the feature map of the single-channel output of the convolutional layer;

[0164] 155. Perform image segmentation on the feature map to obtain the segmentation result.

[0165] In this embodiment, a set of convolutional kernels of appropriate size is selected based on the shooting distance, so that the receptive field of the set of convolutional kernels is suitable for the extraction of endoscopic images. Here, the importance of each receptive field is further measured from the suitable receptive fields, so as to fuse the feature components in the channel extracted by multiple receptive fields. This makes the feature map output by a single channel take into account both the applicability and importance of the receptive fields, improve the image perception ability of endoscopic images, more accurately represent the semantic information of endoscopic images, and make the segmentation mask detected and segmented closer to the real target tissue region.

[0166] In one embodiment, selecting multiple convolution kernels corresponding to the shooting distance and performing convolution processing on the single-channel input of the endoscope image to obtain the corresponding in-channel feature components includes: determining the distance range to which the shooting distance belongs, and selecting multiple convolution kernels with a size interval corresponding to the distance range, wherein the larger the distance range, the larger the size interval, and at least one of the multiple convolution kernels includes a convolution kernel with a size of 1*1; and performing convolution processing on the single-channel input of the endoscope image based on the multiple convolution kernels with the size interval to obtain the corresponding in-channel feature components.

[0167] In this embodiment, each group of convolutional kernels includes a kernel with a minimum size of 1*1. Multiple convolutional kernels are then set based on size intervals. For example, a size interval of 1 results in a group of convolutional kernels with sizes [1*1, 2*2, 3*3, ...], and a size interval of 2 results in a group of convolutional kernels with sizes [1*1, 3*3, 5*5, ...]. The working distance of the endoscope is divided into at least two distance ranges, and each distance range is mapped to at least one size interval to select the group of convolutional kernels.

[0168] Specifically, for example, if the working distance of an endoscope is 5mm-50mm, then the preset distance ranges [5mm, 20mm), [20mm, 35mm), and [35mm, 50mm] are mapped to size intervals of 2, 4, and 6, respectively, corresponding to convolution kernel groups with sizes of [1*1, 3*3, 5*5], [1*1, 5*5, 9*9], and [1*1, 7*7, 13*13].

[0169] It should be noted that other sorting schemes that can be easily conceived by those skilled in the art within the technical scope disclosed in this invention should also be within the protection scope of this invention, and will not be elaborated here.

[0170] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional modules is merely an example. In practical applications, the above functions can be assigned to different functional units or modules as needed, that is, the internal structure of the mobile terminal can be divided into different functional units or modules to complete all or part of the functions described above. The functional modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the modules in the mobile terminal can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0171] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps described in the above-described method embodiments.

[0172] This application provides a computer program product that, when run on a mobile terminal, enables the mobile terminal to implement the steps described in the above-described method embodiments.

[0173] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0174] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0175] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determination" or "if the described condition or event is detected" may be interpreted, depending on the context, as "once determination," "in response to determination," "once the described condition or event is detected," or "in response to the detection of the described condition or event."

[0176] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0177] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0178] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this invention. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0179] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0180] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0181] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0182] If an integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0183] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A tissue-assisted segmentation system, characterized in that, The system includes: Surgical instruments, wherein an image acquisition device is deployed at the end of the surgical instruments, the image acquisition device being used to acquire images of anatomical regions; A display device that displays an enhanced image of the anatomical region; A processor, wherein a deep learning model is deployed on the processor, the deep learning model comprising a first module, a second module, and a third module connected in sequence, the processor being configured to: The first module outputs a first segmentation result of the anatomical region image, and the intermediate processing results of the second module and the third module respectively include a first feature map and a second feature map; Based on the first segmentation result and the second feature map, the second module determines the region of interest in the first feature map and adjusts at least some features in the region of interest. The second module outputs a second segmentation result based on the adjusted first feature map; The enhancement effect is updated based on the second segmentation result.

2. The tissue-assisted segmentation system according to claim 1, characterized in that, The second module includes a first decoder, used for: A first segmentation mask is determined based on the first segmentation result, and a second segmentation mask is determined based on the second feature map, wherein the area covered by the first segmentation mask is larger than the area covered by the second segmentation mask; The first segmentation mask and the second segmentation mask are mapped onto the first feature map to obtain the region of interest in the first feature map.

3. The tissue-assisted segmentation system according to claim 2, characterized in that, The region of interest includes a first region of interest mapped only by the first segmentation mask, a second region of interest mapped only by the second segmentation mask, and a third region of interest mapped by both the first segmentation mask and the second mask, and at least satisfies the following condition: both the first region of interest and the third region of interest are larger than the second region of interest.

4. The tissue-assisted segmentation system according to claim 3, characterized in that, The first decoder is also used for: The feature values ​​in the first region of interest are increased by a preset first amplitude, the feature values ​​in the second region of interest are increased by a preset second amplitude, and / or the feature values ​​in the third region of interest are increased by a preset third amplitude, wherein the preset third amplitude is greater than the preset second amplitude and the preset first amplitude.

5. The tissue-assisted segmentation system according to claim 2, characterized in that, The first decoder is also used for: The first segmentation mask is mapped onto the first feature map to obtain a continuous set of feature points, and the second segmentation mask is mapped onto the first feature map to obtain a discrete set of feature points; Based on the continuous feature point set and the discrete feature point set, the region of interest in the first feature map is determined, wherein the continuous feature point set includes at least a portion of the feature points in the discrete feature point set.

6. The tissue-assisted segmentation system according to claim 5, characterized in that, The first decoder is also used for: Determine the number of consecutive feature points at each discrete location in the discrete feature point set, and select feature points at discrete locations where the number of consecutive feature points exceeds a preset first threshold. Identify the target feature points that belong to the continuous feature point set among the feature points at the selected discrete locations, and increase the feature value of the target feature points.

7. The tissue-assisted segmentation system according to claim 5, characterized in that, The first decoder is also used for: Determine the number of feature points in the discrete feature point set to which the second segmentation mask maps to the same discrete feature point; Identify discrete feature points whose number of mapped feature points exceeds a preset second threshold, and increase the feature value of the identified discrete feature points.

8. The tissue-assisted segmentation system according to any one of claims 2-7, characterized in that, The first module includes a first convolutional layer and a second decoder, the second module further includes a second convolutional layer, and the third module includes a third convolutional layer; The third convolutional layer, the second convolutional layer, and the first convolutional layer are sequentially connected. When the anatomical region image passes through the third convolutional layer, the second convolutional layer, and the first convolutional layer in sequence, a third feature map, a first feature map, and a second feature map are output respectively. The second decoder is connected to the first convolutional layer and the second convolutional layer respectively. The second decoder outputs the first segmentation result based on at least the third feature map output by the first convolutional layer and the first feature map output by the second convolutional layer. The first decoder is connected to the second decoder, the second convolutional layer, and the third convolutional layer, respectively.

9. The tissue-assisted segmentation system according to claim 8, characterized in that, The third feature map is a preset first resolution, the first segmentation result and the first feature map are preset second resolutions, and the second feature map is a preset third resolution, wherein the preset first resolution is greater than the preset second resolution, and the preset second resolution is greater than the preset third resolution; The second decoder further includes a first upsampling layer, which is connected to the first convolutional layer to upsample the third feature map from the preset first resolution to the preset second resolution; The first decoder further includes a second upsampling layer, which is connected to the second convolutional layer to upsample the first feature map from the preset second resolution to the preset third resolution; A third upsampling layer is also connected between the second decoder and the first decoder to upsample the first segmentation result from the preset second resolution to the preset third resolution.

10. The tissue-assisted segmentation system according to claim 8, characterized in that, The first convolutional layer, the second convolutional layer, and the third convolutional layer are each provided with multiple convolutional kernels of different sizes, and multiple global pooling units connected to each convolutional kernel; Each convolutional kernel in the first convolutional layer performs convolution processing on the anatomical region image to obtain multiple corresponding convolution results; Each global pooling unit in the first convolutional layer performs global pooling on the corresponding convolutional processing result to obtain multiple corresponding pooling results. The multiple pooling results are weighted to obtain the third feature map output by the first convolutional layer. The second convolutional layer performs convolution, global pooling, and weighting based on the third feature map to obtain the first feature map; The third convolutional layer performs convolution, global pooling, and weighting processing based on the first feature map to obtain the second feature map.

11. The tissue-assisted segmentation system according to claim 10, characterized in that, Based on the shooting distance of the anatomical region image, multiple convolutional kernels of a target size are selected from multiple convolutional kernels of different sizes set in the first convolutional layer, the second convolutional layer, and the third convolutional layer. The larger the shooting distance, the larger the size interval of the multiple convolutional kernels of the target size, and at least one of the multiple convolutional kernels of the target size includes a convolutional kernel of size 1*1.

12. A tissue-assisted segmentation method, applied to the tissue-assisted segmentation system as described in any one of claims 1-11, characterized in that, The method includes: The first module outputs a first segmentation result of the anatomical region image, and the intermediate processing results of the second module and the third module respectively include a first feature map and a second feature map; Based on the first segmentation result and the second feature map, the second module determines the region of interest in the first feature map and adjusts at least some features in the region of interest. The second module outputs a second segmentation result based on the adjusted first feature map; The enhancement effect displayed in the display device is updated based on the second segmentation result.

13. A readable storage medium, characterized in that, The readable storage medium stores a computer program that, when executed by a processor, implements the steps of the tissue-assisted segmentation method as described in claim 12.

Images