A method for segmenting mucosal lesion regions of a gastroscope multispectral image

Abstract

The application discloses a kind of mucosa pathological change area segmentation methods of gastroscope multispectral image, comprising: obtaining gastroscope multispectral image data and carrying out denoising processing and light uniformization correction;Spectral feature extraction network based on attention mechanism is constructed to extract each waveband channel feature and establish inter-waveband long-distance dependence relationship;Design adaptive feature fusion module to carry out cross-waveband information integration, strengthen lesion-related feature response;Using space-time convolution module to suppress image blur and geometric distortion caused by lens movement and digestive tract peristalsis;Based on light weight decoder network, the binary segmentation mask of mucosa pathological change area is output.The present application makes full use of the complementarity of multispectral information and inter-waveband dependence relationship, combined with space-time consistency modeling and feature gating mechanism, significantly improves the segmentation accuracy and anti-interference ability, and can realize end-to-end real-time processing.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method for segmenting mucosal lesion regions from gastroscopy multispectral images. Background Technology

[0002] With the rapid development of artificial intelligence technology, medical image analysis and intelligent diagnosis have become important research directions in modern medicine. Among these, automatic processing and feature extraction technologies for medical images are crucial for improving the efficiency and accuracy of clinical diagnosis. In the screening and early diagnosis of digestive system diseases, gastroscopy is one of the most direct and effective methods. Automated analysis of images acquired during gastroscopy can significantly reduce the workload of endoscopists and lower the rate of missed diagnoses. In recent years, multispectral imaging technology has been gradually introduced into gastroscopy due to its ability to acquire the unique reflectance characteristics of target tissues in different spectral bands. This technology, by acquiring image data in multiple bands such as visible light and near-infrared, can more comprehensively reflect the spectral response characteristics of tissues, providing rich information for the identification and diagnosis of mucosal lesions that is not available with traditional monospectral imaging.

[0003] Among these, mucosal lesion region segmentation in gastroscopic multispectral images is a core component for achieving automated diagnosis. Its purpose is to accurately locate and delineate potentially lesion-prone mucosal regions from complex gastroscopic multispectral images, providing fundamental support for subsequent lesion classification, severity assessment, and lesion boundary determination. The essence of this segmentation task is to achieve high-precision differentiation between lesion areas and normal tissues based on the spectral differences between different tissue types in multispectral images.

[0004] Existing technologies for medical image segmentation mainly include traditional threshold-based image processing methods, methods based on active contour models, and segmentation networks based on deep learning, which have developed rapidly in recent years. However, when these methods are applied to multispectral images from gastroscopy, they all face varying degrees of technical bottlenecks. First, gastroscopy multispectral images are characterized by high data dimensionality, high image noise, and uneven illumination. Traditional threshold segmentation methods struggle to effectively utilize the high-dimensional features of multispectral information, resulting in severely insufficient segmentation accuracy and coarse boundary processing. Second, multispectral images exhibit complex relationships of redundancy and complementarity between different bands. Existing segmentation networks based on single-modal images lack effective feature fusion mechanisms when processing multispectral data, failing to fully mine and integrate complementary information between spectral channels, leading to the omission of a large amount of valuable diagnostic information. Third, during gastroscopy, factors such as lens movement, patient respiration, and gastrointestinal peristalsis inevitably cause image blurring and geometric distortion. Existing methods are poorly robust to these interferences, easily leading to over-segmentation or under-segmentation. Furthermore, the boundary between mucosal lesions and normal tissue in the multispectral feature space is often blurred, especially in the early stages of lesions, where this difference is even more subtle and difficult to capture. Existing methods perform poorly in handling such low-contrast scenes, failing to meet the stringent segmentation accuracy requirements of clinical diagnosis. The aforementioned technical deficiencies severely restrict the promotion and deployment of gastroscopy multispectral image analysis technology in actual clinical applications. There is an urgent need for a new method that can effectively address the characteristics of multispectral images, possess strong anti-interference capabilities, and achieve high-precision segmentation. Summary of the Invention

[0005] The purpose of this invention is to provide a method for segmenting mucosal lesion areas in gastroscopy multispectral images, which can effectively solve the problems in the background art.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] A method for segmenting mucosal lesion regions from gastroscopic multispectral images includes the following specific steps: Step 1: Acquiring gastroscopic multispectral image data: Image data of multiple spectral bands in visible light and near-infrared light of the area to be detected are acquired using a gastroscopic multispectral imaging device to construct an original multispectral image dataset; Step 2: Multispectral image preprocessing: The original multispectral image data is denoised and illumination homogenized to eliminate the influence of equipment noise and uneven ambient illumination, resulting in a standardized multispectral image; Step 3: Spectral feature extraction and band association modeling: A spectral feature extraction network based on an attention mechanism is constructed to extract the channel features of each spectral band and establish a long-distance dependency model between bands to generate multispectral images. Step 4: Cross-band feature fusion and enhancement: Design an adaptive feature fusion module to integrate cross-band information of the multi-scale spectral feature representation, enhance the response of lesion-related features and suppress irrelevant interference features, and output a fused feature map; Step 5: Anti-interference feature enhancement: Use a spatiotemporal convolution module to perform temporal consistency modeling on the fused feature map, effectively suppressing image blurring and geometric distortion caused by camera movement, patient breathing and gastrointestinal peristalsis, and output robust feature representation; Step 6: Mucosal lesion region segmentation: Based on the robust feature representation, construct a lightweight decoder network, and sequentially perform feature upsampling, boundary detail restoration and probability map generation to output a binary segmentation mask of the mucosal lesion region.

[0008] Preferably, in step 1, the spectral acquisition range of the gastroscopy multispectral imaging device covers a preset spectral acquisition interval, includes multiple discrete spectral bands, the bandwidth of each band is a preset width, and the overlap rate between adjacent bands is controlled within a preset range.

[0009] Preferably, in step 2, the denoising process uses a nonlocal mean filtering algorithm based on three-dimensional block matching, where the search window size, the similar block size, and the attenuation coefficient control factor in the filtering parameters are preset values.

[0010] Preferably, in step 2, the illumination uniformity correction adopts a frequency domain processing method based on homomorphic filtering. After converting the image from the spatial domain to the frequency domain, Gaussian bandstop filtering is performed, wherein the center frequency of the Gaussian filter is set to a preset center frequency and the bandwidth is a preset bandwidth value.

[0011] Preferably, in step 3, the spectral feature extraction network uses a residual channel attention module, which includes a multi-layer residual structure. The number of output channels in each layer is a preset channel number sequence, the scaling factor of the residual connections is a preset value, and the compression ratio of the channel attention module is a preset ratio.

[0012] Preferably, in step 3, the long-distance inter-band dependency model is constructed based on a self-attention mechanism, the number of attention heads is set to a preset number, the dimension of the key vector is a preset dimension, the dimension of the query vector is a preset dimension, the dimension of the value vector is a preset dimension, and the temperature parameter of the attention weight is a preset parameter value.

[0013] Preferably, in step 4, the adaptive feature fusion module includes three parallel branches, which are responsible for local feature aggregation, global feature aggregation and band-specific feature extraction, respectively. The number of output channels of each branch is a preset number of channels. The fusion weight is dynamically determined by learnable parameters. The mathematical representation of the fusion operation is a weighted sum followed by batch normalization processing.

[0014] Preferably, in step 4, the enhanced lesion-related feature response adopts a feature gating mechanism. The gating signal is generated by the spectral difference significance detection module, the detection threshold is set to a preset detection threshold, and the feature response attenuation coefficient below the threshold is a preset attenuation coefficient.

[0015] Preferably, in step 5, the spatiotemporal convolution module includes two convolutional layers in the time dimension and three convolutional layers in the spatial dimension. The temporal convolution kernel size is a preset size, the spatial convolution adopts a depth-separable convolutional structure, and the number of convolution kernels in each layer is a preset sequence.

[0016] Preferably, in step 5, the weights of the loss function for time-series consistency modeling are set to preset weights and jointly optimized with the segmentation loss function. The number of iterations is a preset number of iterations, the initial learning rate is a preset learning rate, and a cosine annealing strategy is used for adjustment.

[0017] Preferably, in step 6, the lightweight decoder network includes four upsampling stages, each with an upsampling factor of a preset factor, the skip connection fusion method is channel-level concatenation followed by convolutional compression, and the activation function of the output layer is a preset activation function.

[0018] Preferably, in step 6, the boundary detail recovery adopts the boundary-aware loss function, the weight coefficient is set to a preset weight coefficient, and it is weighted and combined with the Dice loss function and the cross-entropy loss function, and the weight ratio of the three satisfies the preset ratio relationship.

[0019] Preferably, the present invention further includes a post-processing step: performing morphological closing operations on the output binary segmentation mask to eliminate small holes in the segmentation result, and setting the radius of the structuring element to a preset radius.

[0020] The beneficial effects of this invention are:

[0021] 1. High-precision segmentation capability: Full utilization of multispectral information: By constructing an attention-based spectral feature extraction network and a long-distance inter-band dependency model, complementary information between various spectral channels is effectively mined and integrated, making full use of the diagnostic value of multiple spectral bands within the preset spectral range. Compared with existing single-modal methods, the information utilization rate is significantly improved; Strong adaptability to low-contrast scenes: The spectral difference significance detection mechanism in the adaptive feature fusion module can enhance the response of lesion-related features and suppress interference features, significantly improving the segmentation accuracy of low-contrast scenes such as the early stage of mucosal lesions, with segmentation performance indicators reaching or exceeding the preset level.

[0022] 2. Strong anti-interference capability: Effective suppression of motion artifacts: The spatiotemporal convolution module effectively addresses image blurring and geometric distortion caused by lens movement, patient breathing, and digestive tract peristalsis during gastroscopy through temporal consistency modeling, maintaining the segmentation accuracy of motion-blurred images above the preset accuracy rate; Solving the problem of uneven illumination: Illumination homogenization correction and denoising based on homomorphic filtering significantly improve the impact of uneven illumination and equipment noise on segmentation results during gastroscopy, resulting in clearer and more accurate segmentation boundaries.

[0023] 3. High-efficiency segmentation performance: Lightweight network structure: The lightweight decoder network combined with depthwise separable convolution technology significantly reduces the number of model parameters and computational complexity while ensuring segmentation accuracy. The inference speed meets the needs of real-time clinical applications, and the processing time for a single frame image does not exceed the preset processing time; End-to-end processing flow: The complete process from the input of the original multispectral image to the output of the final segmentation mask realizes end-to-end processing without manual intervention and complex post-processing steps, which significantly improves the convenience of clinical applications. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the overall technical solution architecture of the gastroscopy multispectral image mucosal lesion region segmentation method according to an embodiment of this application;

[0025] Figure 2 This is a schematic diagram of the core principle framework of cross-band feature fusion and enhancement in the gastroscopy multispectral image mucosal lesion region segmentation method according to the embodiments of this application;

[0026] Figure 3 This is a flowchart of the mucosal lesion region segmentation stage of the gastroscopy multispectral image mucosal lesion region segmentation method according to an embodiment of this application. Detailed Implementation

[0027] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. The illustrative embodiments and descriptions herein are used to explain the present invention, but are not intended to limit the present invention.

[0028] like Figure 1-3 As shown: To make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to specific embodiments.

[0029] Example 1

[0030] This invention provides a method for segmenting mucosal lesion regions from gastroscopic multispectral images. This method comprises six core processing steps, from acquiring the original multispectral image to outputting the final binary segmentation mask, forming a complete end-to-end processing flow. The specific implementation of each step is described in detail below.

[0031] Step 1, acquire gastroscopy multispectral image data: acquire image data of multiple spectral bands of visible light and near infrared light of the area to be detected using a gastroscopy multispectral imaging device, and construct the original multispectral image dataset.

[0032] The spectral acquisition range of the gastroscopy multispectral imaging device covers a preset spectral acquisition interval, preferably 400 nm to 950 nm, with the visible light band covering 400 nm to 700 nm and the near-infrared band covering 700 nm to 950 nm. The spectral acquisition interval includes multiple discrete spectral bands, and the center wavelength of each discrete spectral band can be flexibly configured according to clinical diagnostic needs. Typical configuration schemes include the following two: In the first configuration scheme, eight discrete spectral bands are set, with center wavelengths of 420 nm, 480 nm, 540 nm, 590 nm, 640 nm, 700 nm, 800 nm, and 900 nm, respectively. The bandwidth of each band is set to 30 nm, and the overlap rate between adjacent bands is controlled between 15% and 2%. The first configuration option involves setting 16 discrete spectral bands, with center wavelengths of 430 nm, 460 nm, 490 nm, 520 nm, 550 nm, 580 nm, 610 nm, 640 nm, 670 nm, 700 nm, 730 nm, 760 nm, 800 nm, 850 nm, 900 nm, and 950 nm, respectively. The bandwidth of each band is set to 20 nm, and the overlap rate between adjacent bands is controlled between 10% and 15%. The gastroscopy multispectral imaging device adopts a prism-based or filter-based multispectral imaging architecture. The image sensor uses a complementary metal-oxide-semiconductor (CMOS) sensor with an effective pixel count of no less than 1920×1080, a dynamic range of no less than 12 bits, and a frame rate of no less than 25 frames per second. The raw multispectral image data is stored and transmitted in the form of a multi-channel tensor, with a data format of a three-dimensional tensor (H×W×C), where H represents the image height, W represents the image width, and C represents the number of spectral channels. The image data for each spectral channel is stored independently as an unsigned 16-bit integer. All channel data are packaged into a multidimensional array for caching and transmission. The construction process of the raw multispectral image dataset includes the following operations: First, continuous image acquisition of the area to be examined is performed using a gastroscopy multispectral imaging device, keeping the device position relatively fixed during acquisition, with an acquisition duration set to 30 to 60 seconds; second, the acquired raw image sequences are initially screened, removing single-frame images with low clarity or abnormal exposure; finally, the screened image sequences are organized chronologically to construct a multispectral image dataset containing N frames, where each frame contains image data from C spectral channels.

[0033] Step 2, Multispectral Image Preprocessing: Denoising and illumination homogenization correction are performed on the original multispectral image data to eliminate the effects of equipment noise and uneven ambient illumination, resulting in a standardized multispectral image.

[0034] The denoising process employs a nonlocal mean filtering algorithm based on 3D block matching. This algorithm fully utilizes the redundant information in the spectral dimension of the multispectral image, preserving edge and texture details as much as possible while removing noise. The specific implementation process of the 3D block matching nonlocal mean filtering algorithm includes the following operations: First, the input multispectral image is represented as a 3D data block structure, where the three dimensions correspond to the spatial height, spatial width, and number of spectral channels of the image, respectively. Second, when processing each pixel, a reference block is constructed centered on that pixel. The size of the reference block is a preset similarity block size, preferably 5×5×C, where C is the number of spectral channels. A search window is defined around the reference block, with a preset search window size, preferably 21×21×C. Then, all candidate blocks are traversed within the search window, and the similarity metric between the reference block and each candidate block is calculated. The denoising process uses Euclidean distance, specifically calculating the sum of squared differences between corresponding pixel values ​​of the reference block and candidate blocks. Based on the similarity metric, several candidate blocks similar to the reference block are selected, typically 5 to 8. Next, the selected similar blocks are weighted and aggregated. The aggregation weight is determined by the similarity metric, and the weight calculation formula includes an attenuation coefficient control factor. The value of the attenuation coefficient control factor directly affects the denoising intensity and smoothing effect; ideally, it should be between 10 and 30, but the specific value needs to be optimized based on the noise level and image characteristics. Finally, the weighted aggregation result is assigned to the center pixel of the reference block, and this process is repeated for all pixels to complete the denoising of a single frame. During the denoising process, the above denoising operation is performed independently for each spectral channel, with consistent denoising parameters across channels. After denoising is complete, the denoised multispectral image data is output.

[0035] Illumination homogenization correction employs a frequency domain processing method based on homomorphic filtering. This method can effectively eliminate the uneven illumination caused by the endoscope illumination system while maintaining image contrast and detail. The specific implementation process of illumination homogenization correction based on homomorphic filtering includes the following operations: First, a logarithmic transform is performed on the denoised multispectral image to convert the image from the spatial domain to the logarithmic domain. The mathematical expression for the logarithmic transform is that L equals the logarithmic value of the input image I, i.e., L equals log(I plus 1), where the addition of 1 is used to avoid taking the logarithm of zero. Second, a fast Fourier transform is performed on the logarithmic domain image to convert the image from the spatial domain to the frequency domain. The input data size of the fast Fourier transform is H×W×C, and the output is a complex frequency domain representation of the same size. Then, a Gaussian band-stop filter is designed and applied in the frequency domain. A Gaussian band-stop filter is a Gaussian filter with band-stop characteristics in the frequency domain. Its frequency response function exhibits high gain near the center frequency and low gain in the region far from the center frequency. The function of the Gaussian band-stop filter is to attenuate the low-frequency components (i.e., components with slowly changing illumination) in the image while retaining the high-frequency components (i.e., image details and edge information). The center frequency of the filter is set to a preset center frequency, preferably a value within the normalized frequency range of 0.02 to 0.05. The bandwidth is a preset bandwidth value, preferably a value within the normalized frequency range of 0.01 to 0.02. The transfer function of the Gaussian bandstop filter is expressed as H(u, v) equal to 1 minus exp(negative ((u squared plus v squared minus D0 squared) divided by (u squared plus v squared minus W squared)) to the power of n), where u and v are frequency coordinates, D0 is the cutoff radius corresponding to the center frequency, W is the bandwidth parameter, and n is the filter order, preferably 2. Next, an inverse fast Fourier transform is performed on the filtered frequency domain data to convert the image from the frequency domain back to the spatial domain. Finally, an exponential transform is performed on the result after the inverse transform to restore the grayscale range of the original image. The mathematical expression of the exponential transform is O equal to exp(L') multiplied by the normalization coefficient, where L' is the logarithmic domain image after the inverse transform, and O is the output image after illumination uniformity correction. During the illumination homogenization correction process, the above processing procedure is performed independently for each spectral channel, and the filter parameters of each channel remain consistent.

[0036] After denoising and illumination homogenization correction, the output standardized multispectral image data retains its original three-dimensional tensor structure (H×W×C), with data type in floating-point format and values ​​normalized to between 0 and 1. This standardized multispectral image data can be directly input into subsequent spectral feature extraction networks for further processing.

[0037] Step 3, Spectral Feature Extraction and Band Association Modeling: Construct a spectral feature extraction network based on an attention mechanism to extract the channel features of each spectral band, establish a long-distance dependency model between bands, and generate multi-scale spectral feature representations.

[0038] The spectral feature extraction network employs a residual channel attention module, which consists of multiple stacked residual structures. This module effectively extracts deep features from multispectral images and adaptively adjusts the weights of each channel feature through a channel attention mechanism. The specific network architecture of the residual channel attention module includes the following components:

[0039] The first layer is a convolutional input layer, which contains a two-dimensional convolutional kernel with a size of 3×3, a stride of 1, and edge-filled padding. The number of input channels of this convolutional layer is C (the number of spectral channels), and the number of output channels is a first preset number of channels, preferably 64. This convolutional layer is followed by a batch normalization layer and a ReLU activation function layer. The momentum parameter of the batch normalization layer is set to 0.9, and the negative slope of the ReLU activation function is set to 0.2.

[0040] Layers two through five are residual attention blocks. Each residual attention block contains two residual units. The structure of each residual unit is, in sequence, a convolutional layer, a batch normalization layer, a ReLU activation function layer, another convolutional layer, another batch normalization layer, and a residual concatenation adder. The number of output channels for each residual attention block is a corresponding preset channel number sequence. The number of output channels for the second layer is set to 64, for the third layer to 128, for the fourth layer to 256, and for the fifth layer to 512. The scaling factor for the residual concatenation is a preset value, preferably 0.5. This scaling factor controls the fusion ratio of the residual path and the identity mapping path. The fusion method is that the output equals the main path output multiplied by (1 minus the scaling factor) plus the residual path output multiplied by the scaling factor. The channel attention module is embedded after the second convolutional layer of each residual unit. The compression ratio of the channel attention module is a preset ratio, preferably 16. The specific meaning of the compression ratio is as follows: the channel attention module first compresses the size of the feature map from H×W×C to 1×1×C through a global average pooling layer, then compresses the number of channels from C to C divided by the compression ratio through a fully connected layer. The activation function of this fully connected layer is ReLU. Next, the number of channels is expanded back to C by dividing C by the compression ratio through another fully connected layer. The activation function of this fully connected layer is Sigmoid. Finally, the channel attention weights are multiplied with the original feature map channel by channel to achieve adaptive calibration of the channel features.

[0041] The output of the residual channel attention module is a multi-scale spectral feature representation, which contains four feature maps at different scales, corresponding to the outputs of layers two through five of the network. The sizes of the four feature maps are H×W×64, H / 2×W / 2×128, H / 4×W / 4×256, and H / 8×W / 8×512, respectively. This multi-scale spectral feature representation can simultaneously capture fine-grained details and high-level semantic information of the image, providing a rich feature foundation for subsequent cross-band feature fusion.

[0042] The long-distance inter-band dependency model is built upon a self-attention mechanism. This model can establish semantic associations between different spectral bands and capture long-distance dependency information between bands. The core idea of ​​the self-attention mechanism is: for each element in the input sequence, calculate its attention weight with all other elements in the sequence, and then perform a weighted sum of the value vectors of all elements according to the attention weights to generate the output. The mathematical expression of the self-attention mechanism is: Attention(Q, K, V) equals Softmax((Q multiplied by the transpose of K) divided by the square root of Dk) multiplied by V, where Q is the query matrix, K is the key matrix, V is the value matrix, and Dk is the dimension of the key vector. The specific implementation of the long-distance inter-band dependency model includes the following operations:

[0043] First, the multi-scale spectral feature representation output by the residual channel attention module is subjected to global average pooling in spatial dimension, compressing the feature map of each channel into a scalar value, and obtaining the global description vector of each spectral channel, with the dimension of the global description vector being C.

[0044] Then, the global description vector is projected to the query space, key space and value space respectively through three independent linear transformation layers. The dimensions of the query vector, the key vector and the value vector are preset. The dimensions of the query vector, key vector and value vector are preferably all set to 256.

[0045] Next, the dot product similarity between the query vector and the key vector is calculated to obtain the attention score matrix. The size of the attention score matrix is ​​C×C, where the element in the i-th row and j-th column represents the attention weight of the i-th spectral channel to the j-th spectral channel.

[0046] Next, the attention score matrix is ​​normalized using the Softmax function to obtain the normalized attention weight matrix. The temperature parameter of the Softmax function is a preset value, which controls the sharpness of the attention weight distribution. The temperature parameter is preferably set to 1.0.

[0047] Then, the inter-band attention enhancement features are calculated based on the normalized attention weight matrix and the value vector. The dimension of the attention enhancement features is the same as the dimension of the value vector.

[0048] Finally, the inter-band attention enhancement features are residually concatenated with the original global description vector and restored to the original channel dimension through a linear transformation layer to obtain the multi-scale spectral feature representation after enhancing the long-distance dependence between bands.

[0049] The number of attention heads in the self-attention mechanism is set to a preset number, preferably 8. The multi-head attention mechanism projects the query, key, and value matrices from the original dimension to multiple low-dimensional subspaces, calculates attention independently in each subspace, and finally concatenates the attention results of each subspace and performs a linear transformation to obtain the final output. The output of the long-distance inter-band dependency model is fused with the output of the residual channel attention module. The fusion method is to concatenate along the channel dimension. The fused feature dimension is the sum of the channel dimension of the multi-scale spectral feature representation and the inter-band attention enhancement feature dimension.

[0050] Step 4, Cross-band feature fusion and enhancement: Design an adaptive feature fusion module to integrate cross-band information of multi-scale spectral feature representation, enhance the response of lesion-related features and suppress irrelevant interference features, and output a fused feature map.

[0051] The adaptive feature fusion module comprises three parallel branches, responsible for local feature aggregation, global feature aggregation, and band-specific feature extraction, respectively. The input to each branch is the multi-scale spectral feature representation output from step 3, and the output channels of each branch are a preset number, preferably set to 256. The detailed design of the three parallel branches is as follows:

[0052] The first branch is the local feature aggregation branch. This branch uses a multi-layer convolutional structure to aggregate local spatial information, containing three convolutional layers and one global pooling layer. The first convolutional layer has a 1×1 kernel size, 128 output channels, and ReLU activation function; the second convolutional layer has a 3×3 kernel size, 256 output channels, ReLU activation function, and edge padding; the third convolutional layer has a 1×1 kernel size, 256 output channels, and Sigmoid activation function, used to generate a local attention weight map; the global pooling layer performs global average pooling on the local attention weight map to obtain the scalar weights for local feature aggregation.

[0053] The second branch is the global feature aggregation branch. This branch uses a self-attention mechanism to aggregate global spatial information, and includes a multi-head self-attention layer and a feedforward network layer. The multi-head self-attention layer has 8 attention heads, and the dimensions of the key vector, query vector, and value vector are all set to 64. The feedforward network layer contains two fully connected layers. The first layer compresses the dimension from 512 to 256 using ReLU activation function, and the second layer expands the dimension from 256 to 512. The output of the global feature aggregation branch obtains the scalar weights for global feature aggregation through a global average pooling layer.

[0054] The third branch is the band-specific feature extraction branch. This branch uses channel-wise convolution and channel blending operations to extract unique features for each spectral band, and includes one channel-wise convolutional layer and one channel blending layer. The channel-wise convolutional layer has a kernel size of 3×3×1, performs spatial convolution operations independently on each spectral channel, keeps the number of output channels consistent with the number of input channels, and uses ReLU as the activation function. The channel blending layer randomly blends the channel dimensions to enhance the interaction between features between bands, with a blending ratio of 70% preservation of the original channel order and 30% permutation of the channel order. The output of the band-specific feature extraction branch is passed through a global max pooling layer to obtain the scalar weights for band-specific feature extraction.

[0055] The fusion weights are dynamically determined through learnable parameters. Specifically, the scalar weights output from the three branches are concatenated to form a 3-dimensional weight vector. This vector is then mapped from 3 to 3 using a fully connected layer, and normalized using the Softmax function to obtain the normalized fusion weights for the three branches. The normalized fusion weights satisfy the condition that the sum of the three weight values ​​is 1. Mathematically, the fusion operation is represented as a weighted sum followed by batch normalization. The formula for calculating the fusion feature is F_fusion = (w1 multiplied by F_local plus w2 multiplied by F_global plus w3 multiplied by F_band) divided by gamma multiplied by beta, where w1, w2, and w3 are the normalized fusion weights of the three branches, F_local, F_global, and F_band are the output features of the three branches, and gamma and beta are the learnable scaling and offset parameters of the batch normalization layer.

[0056] The enhancement of lesion-related feature responses employs a feature gating mechanism, with the gating signal generated by the spectral difference significance detection module. The specific implementation of this module includes the following operations: First, a difference map between adjacent spectral bands in the multispectral image is calculated, using the absolute value of the difference between adjacent bands. Second, the difference map is thresholded, marking regions with difference values ​​higher than a detection threshold as salient regions. The detection threshold is a preset threshold, calculated as the mean of the grayscale values ​​in the difference map plus twice the standard deviation. Then, morphological dilation is performed on the salient regions, with the radius of the dilation structuring element set to 3 pixels to expand the coverage of the salient regions. Finally, the dilated binary map of the salient regions is used as the gating signal to perform element-wise multiplication on the fused feature map. Feature responses below the detection threshold are multiplied by a preset attenuation coefficient, preferably set to 0.3. Through this feature gating mechanism, lesion-related feature responses are enhanced, while irrelevant interference features are effectively suppressed. The output fused feature map has a size of H / 8×W / 8×256. The fused feature map contains rich cross-band semantic information, providing high-quality input for subsequent anti-interference feature enhancement.

[0057] Step 5, Anti-interference feature enhancement: The spatiotemporal convolution module is used to perform temporal consistency modeling on the fused feature map, which effectively suppresses image blurring and geometric distortion caused by camera movement, patient breathing and gastrointestinal peristalsis, and outputs robust feature representation.

[0058] The spatiotemporal convolution module's architecture fully considers the temporal and spatial correlations in the gastroscopy image sequence. The module contains two temporal convolutional layers and three spatial convolutional layers. The temporal convolutional operations are used to capture the temporal correlation between adjacent frames, while the spatial convolutional operations are used to extract spatial features from a single frame. The specific network structure of the spatiotemporal convolution module is as follows:

[0059] The first layer is a temporal convolutional layer. The input data for this layer consists of the fused feature map output from step 4 and the fused feature maps from several preceding frames. The input data size is T×H / 8×W / 8×256, where T is the time length of the input sequence, preferably set to 5. The kernel size of the temporal convolutional layer is a preset size, preferably 3×1×1, meaning the temporal kernel size is 3 and the spatial kernel size is 1×1, with a total of 256 kernels. This layer performs sliding window filtering in the temporal dimension to capture the feature relationships between adjacent frames. The temporal convolutional layer is followed by a batch normalization layer and a ReLU activation function layer.

[0060] The second layer is a spatial convolutional layer, employing a depthwise separable convolutional structure to reduce the number of parameters and computational complexity. The depthwise separable convolution consists of two parts: channel-wise convolution and pointwise convolution. The channel-wise convolution has a 3×3 kernel size, 256 input channels, and the same number of output channels as the input channels, performing spatial convolution independently on each channel. The pointwise convolution has a 1×1 kernel size, 256 input channels, and 512 output channels, used to fuse feature information from each channel. Following the spatial convolutional layer is a batch normalization layer, a ReLU activation function layer, and a dropout layer, with the dropout ratio set to 0.1.

[0061] The third layer is a temporal convolutional layer. The structure of this layer is the same as the first temporal convolutional layer. The input is the concatenation of the output of the second spatial convolutional layer in the temporal dimension. The kernel size is set to 3×1×1 and the number of kernels is set to 512. The temporal convolutional layer is followed by a batch normalization layer and a ReLU activation function layer.

[0062] The fourth and fifth layers are both spatial convolutional layers. The fourth spatial convolutional layer uses a depthwise separable convolutional structure. For channel-wise convolution, the kernel size is set to 3×3, with 512 input channels and 512 output channels; for pointwise convolution, the input channels are 512 and the output channels are 256. The fifth spatial convolutional layer uses a standard convolutional structure with a 1×1 kernel, 256 input channels, and 128 output channels, used to further compress the feature dimension and enhance feature representation capabilities. Both the fourth and fifth spatial convolutional layers are followed by batch normalization layers and ReLU activation function layers.

[0063] The number of convolutional kernels in each layer of the spatiotemporal convolution module forms a preset sequence: 256, 512, 512, 256, 128. The output of the spatiotemporal convolution module is a robust feature representation modeled with temporal consistency, and the size of the robust feature representation is H / 8×W / 8×128.

[0064] The weights of the loss function for temporal consistency modeling are set to preset weights, preferably 0.5, and are jointly optimized with the segmentation loss function. The number of iterations is a preset number, preferably 100, and the initial learning rate is a preset value, preferably 0.001. A cosine annealing strategy is used to adjust the learning rate, with a minimum learning rate of 0.00001. The joint optimization process uses the stochastic gradient descent algorithm, with a batch size of 4, a momentum parameter of 0.9, and a weight decay coefficient of 0.0001.

[0065] Step 6, Mucosal lesion region segmentation: Based on robust feature representation, a lightweight decoder network is constructed, and feature upsampling, boundary detail restoration and probability map generation are performed in sequence to output a binary segmentation mask of the mucosal lesion region.

[0066] The lightweight decoder network comprises four upsampling stages, each with an upsampling factor of a preset factor, preferably set to 2, meaning each upsampling stage enlarges the spatial size of the input feature map by a factor of 2. The specific structure of the four upsampling stages is as follows:

[0067] The first upsampling stage receives the robust feature representation output from step 5 as input, with the input feature map having dimensions of H / 8 × W / 8 × 128. This first upsampling stage includes an upsampling operation and a convolutional fusion operation. The upsampling operation uses bilinear interpolation to enlarge the spatial size of the input feature map from H / 8 × W / 8 to H / 4 × W / 4. Following the upsampling operation is a convolutional layer with a 3 × 3 kernel, 128 input channels, 128 output channels, and ReLU activation. The skip connection fusion method is to concatenate the channel dimensions and then perform convolutional compression. This stage is skip connected with the output feature map (H / 4×W / 4×256) of the fourth layer in step 3. The specific fusion method is as follows: First, the output of the upsampling stage and the feature map of the skip connection are concatenated in the channel dimension. The number of channels after concatenation is 128 plus 256, which equals 384. Then, a convolutional layer is used to compress the number of channels from 384 to 256. The convolutional kernel size is 3×3 and the activation function is ReLU.

[0068] The second upsampling stage receives the output of the first upsampling stage as input, with the input feature map having a size of H / 4×W / 4×256. The upsampling factor in this stage is 2, increasing the spatial size of the input feature map from H / 4×W / 4 to H / 2×W / 2. Following the upsampling operation is a convolutional layer with a 3×3 kernel, 256 input channels, 128 output channels, and ReLU activation. A skip connection is then used to fuse the output feature map (H / 2×W / 2×128) from the third layer in step 3. The fusion method is also channel concatenation followed by convolutional compression. The concatenated channel count is 128 + 128 = 256, and the compressed channel count is 128.

[0069] The third upsampling stage receives the output of the second upsampling stage as input, with the input feature map having a size of H / 2×W / 2×128. The upsampling factor in this stage is 2, enlarging the spatial size of the input feature map from H / 2×W / 2 to H×W. Following the upsampling operation is a convolutional layer with a kernel size of 3×3, 128 input channels, 64 output channels, and ReLU activation. A skip connection is then used to fuse the output feature map (H×W×64) from the second layer in step 3. The fusion method involves concatenating the channel dimensions followed by convolutional compression, resulting in 64 channels (64 + 64 = 128) after concatenation and 64 channels after convolutional compression.

[0070] The fourth upsampling stage receives the output of the third upsampling stage as input, with an input feature map size of H×W×64. This stage is the final segmentation head stage and does not perform upsampling; it only uses a single convolutional layer to adjust feature channels and generate a probability map. The segmentation head contains two convolutional layers: the first convolutional layer has a 3×3 kernel size, 64 input channels, 32 output channels, and uses ReLU activation; the second convolutional layer has a 1×1 kernel size, 32 input channels, 2 output channels, and uses a preset activation function, preferably Softmax, to generate the segmentation probability map. The segmentation probability map has a size of H×W×2, where the first channel represents the probability of the background category and the second channel represents the probability of the lesion category.

[0071] Boundary detail restoration employs a boundary-aware loss function, with its weight coefficients set to a preset value, preferably 0.8. The boundary-aware loss function is weighted in combination with the Dice loss function and the cross-entropy loss function, with their weights meeting a preset ratio. The preferred preset ratio is 0.4 for the boundary-aware loss function, 0.3 for the Dice loss function, and 0.3 for the cross-entropy loss function. The boundary-aware loss function is calculated as follows: first, morphological gradient operations are performed on the ground truth labels to extract label values ​​in the boundary regions; then, the pixel-level difference between the predicted segmentation mask and the ground truth labels in the boundary regions is calculated; finally, the difference values ​​are weighted and summed to obtain the boundary-aware loss value. The Dice loss function is calculated as DICE = 1 minus (2 multiplied by the number of pixels intersecting the predicted result and the ground truth labels) divided by (the total number of pixels in the predicted result plus the total number of pixels in the ground truth labels). The cross-entropy loss function uses a standard pixel-level cross-entropy calculation method. The weighted sum of the three loss functions is used as the total loss function for backpropagation optimization.

[0072] Regarding the activation function of the output layer, the intermediate convolutional layers in the second to fourth upsampling stages use the ReLU activation function, and the final output layer uses the Softmax activation function to generate a segmentation probability map. The segmentation probability map is then subjected to an argmax operation to select the class with the highest probability as the final predicted class, generating a binary segmentation mask of size H×W. In the binary segmentation mask, a pixel value of 0 represents the background region, and a pixel value of 1 represents the lesion region.

[0073] This invention also includes a post-processing step: performing morphological closing operations on the output binary segmentation mask to eliminate small holes in the segmentation result. The radius of the structuring element is set to a preset radius, preferably an integer between 2 and 5 pixels. The morphological closing operation is a combination of dilation and erosion. The dilation operation fills the small holes in the binary segmentation mask, and the erosion operation restores the boundary expansion caused by dilation. After the morphological closing operation is completed, the final segmentation result of the mucosal lesion region is output.

[0074] In practical clinical applications, the process of segmenting mucosal lesion areas in gastroscopic multispectral images using the method of this invention is as follows: Clinicians use a gastroscope equipped with multispectral imaging function to scan the gastric mucosa of the patient to be examined. The gastroscopic multispectral imaging device continuously acquires multispectral image data at a rate of 25 frames per second. During the acquisition process, image data from 12 spectral channels, including 8 channels in the visible light band and 4 channels in the near-infrared band, are collected. The image resolution is 1920×1080. After acquisition, the original multispectral image sequence is input into the method of this invention for processing: First, after data acquisition and format standardization in step 1, a standardized multispectral image sequence with a size of 1920×1080×12 is obtained; then, after denoising and illumination homogenization correction in step 2, the influence of equipment noise and uneven illumination is eliminated; next, after spectral feature extraction and band association modeling in step 3, a multi-scale spectral feature representation containing four scales is generated; then, after cross-band feature fusion and enhancement in step 4, a fused feature map is obtained; then, after processing by the spatiotemporal convolution module in step 5, motion blur and geometric distortion are effectively suppressed; finally, after processing by the lightweight decoder network in step 6, a binary segmentation mask of the lesion area is output, and the segmentation boundary is optimized through post-processing steps. The end-to-end inference time of the entire processing flow does not exceed 200 milliseconds per frame on a computing platform equipped with a graphics processor, meeting the needs of real-time clinical applications. The Dice coefficient of the segmentation result reaches above 0.85 on the validation dataset, and the boundary F1 score reaches above 0.78, which can accurately segment the mucosal lesion area in gastroscopy images.

[0075] Example 2

[0076] This embodiment describes an alternative technical solution for mucosal lesion region segmentation in ultra-high-definition gastroscopy multispectral images. While maintaining the overall framework of steps 1 to 6, this solution adjusts and optimizes some technical details for high-resolution image processing scenarios.

[0077] In step 1, when the resolution of the gastroscopy multispectral image to be processed is an ultra-high definition of 3840×2160, the number of spectral channels is expanded to 24 channels to obtain richer spectral information. At this time, the configuration of the spectral acquisition range is adjusted as follows: 12 discrete spectral bands are set in the visible light band, with center wavelengths of 410 nm, 440 nm, 470 nm, 500 nm, 530 nm, 560 nm, 590 nm, 620 nm, 650 nm, 680 nm, 710 nm, and 740 nm, respectively, and the bandwidth of each band is set to 25 nm; 12 discrete spectral bands are set in the near-infrared band, with center wavelengths of 760 nm, 790 nm, 820 nm, 850 nm, 880 nm, 910 nm, 940 nm, 970 nm, 1000 nm, 1030 nm, 1060 nm, and 1100 nm, respectively, and the bandwidth of each band is set to 30 nm. The overlap rate between adjacent bands is controlled between 12% and 18%. The image sensor adopts a high-sensitivity back-illuminated complementary metal-oxide-semiconductor sensor with an effective pixel count of no less than 3840×2160, a dynamic range of no less than 14 bits, and a frame rate of no less than 30 frames per second. The storage format of the raw multispectral image data is adjusted to a four-dimensional tensor (T×H×W×C), where T is the time series length.

[0078] In step 2, the parameters for denoising are adjusted as follows: the search window size is increased to 31×31×C to adapt to the noise distribution characteristics of high-resolution images; the similar block size remains unchanged at 5×5×C; and the attenuation coefficient control factor is increased to 40 to improve denoising intensity. In illumination homogenization correction, the center frequency of the Gaussian bandstop filter is adjusted to a normalized frequency of 0.015, and the bandwidth is also adjusted to a normalized frequency of 0.015.

[0079] In step 3, the parameters of the residual channel attention module are adjusted as follows: the number of output channels per layer is adjusted to 128, 256, 512, and 1024; the scaling factor of the residual connections is adjusted to 0.6; and the compression ratio of the channel attention module is adjusted to 8. The parameters of the long-distance dependency model between bands are adjusted as follows: the number of attention heads is adjusted to 16; the dimensions of the query vector, key vector, and value vector are all adjusted to 512; and the temperature parameter is adjusted to 1.2.

[0080] In step 4, the parameters of the adaptive feature fusion module are adjusted as follows: the number of output channels for all three parallel branches is adjusted to 512. A gating control mechanism is added to the calculation of fusion weights; when the weight difference among the three branches exceeds a preset threshold, automatic weight equalization is performed. In the spectral difference significance detection module, the detection threshold is adjusted to the mean of the grayscale values ​​of the difference map plus 2.5 times the standard deviation, and the attenuation coefficient is adjusted to 0.2.

[0081] In step 5, the parameters of the spatiotemporal convolution module are adjusted as follows: the kernel size of the temporal convolutional layer is adjusted to 5×1×1 to capture temporal correlations within a larger time window; the time series length T is adjusted to 7; and the number of kernels in each layer is sequentially adjusted to 512, 1024, 1024, 512, and 256. The weight of the loss function for temporal consistency modeling is adjusted to 0.6, the number of iterations is adjusted to 150, and the initial learning rate is adjusted to 0.0005.

[0082] In step 6, the lightweight decoder network is adjusted to include five upsampling stages to accommodate higher input resolutions, with each stage maintaining an upsampling factor of 2. The skip connection fusion method adds a depthwise separable convolutional layer for compression after channel-level concatenation. The weights for boundary detail restoration are adjusted to 0.5 for the boundary-aware loss function, 0.25 for the Dice loss function, and 0.25 for the cross-entropy loss function. In the post-processing steps, the radius of the structuring element for the morphological closing operation is adjusted to 5 to 8 pixels.

[0083] Validation results of this alternative on a high-resolution gastroscopy multispectral image dataset show further improved segmentation accuracy, with a Dice coefficient exceeding 0.88 and a boundary F1 score exceeding 0.82. However, the end-to-end inference time increases to 350 milliseconds per frame. This approach is suitable for clinical diagnostic scenarios requiring extremely high segmentation accuracy.

[0084] The technical solutions of the present invention are not limited to the specific embodiments described above. Any technical modifications made in accordance with the technical solutions of the present invention fall within the protection scope of the present invention.

Claims

1. A method for segmenting mucosal lesion regions from multispectral images of gastroscopy, characterized in that, Includes the following steps: Step 1, acquire gastroscopy multispectral image data: acquire image data of multiple spectral bands of visible light and near infrared light of the area to be detected using a gastroscopy multispectral imaging device, and construct the original multispectral image dataset; Step 2, Multispectral Image Preprocessing: The original multispectral image data is denoised and illumination homogenized to eliminate the effects of equipment noise and uneven ambient illumination, resulting in a standardized multispectral image. Step 3, Spectral feature extraction and band association modeling: Construct a spectral feature extraction network based on an attention mechanism to extract the channel features of each spectral band, establish a long-distance dependency model between bands, and generate multi-scale spectral feature representations; Step 4, Cross-band feature fusion and enhancement: Design an adaptive feature fusion module to integrate cross-band information of the multi-scale spectral feature representation, enhance the response of lesion-related features and suppress irrelevant interference features, and output a fused feature map; Step 5, Anti-interference feature enhancement: The spatiotemporal convolution module is used to perform temporal consistency modeling on the fused feature map, which effectively suppresses image blurring and geometric distortion caused by camera movement, patient breathing and gastrointestinal peristalsis, and outputs robust feature representation; Step 6, Mucosal lesion region segmentation: Based on the robust feature representation, a lightweight decoder network is constructed, and feature upsampling, boundary detail restoration and probability map generation are performed sequentially to output a binary segmentation mask of the mucosal lesion region.

2. The method for segmenting mucosal lesion regions from multispectral images of gastroscopy according to claim 1, characterized in that, In step 1, the spectral acquisition range of the gastroscopy multispectral imaging device covers a preset spectral acquisition interval, includes multiple discrete spectral bands, the bandwidth of each band is a preset width, and the overlap rate between adjacent bands is controlled within a preset range.

3. The method for segmenting mucosal lesion regions from multispectral images of gastroscopy according to claim 1, characterized in that, In step 2, the denoising process adopts a nonlocal mean filtering algorithm based on three-dimensional block matching. The search window size, similar block size, and attenuation coefficient control factor in the filtering parameters are preset values.

4. The method for segmenting mucosal lesion regions from multispectral images of gastroscopy according to claim 1, characterized in that, In step 2, the illumination uniformity correction adopts a frequency domain processing method based on homomorphic filtering. After converting the image from the spatial domain to the frequency domain, Gaussian bandstop filtering is performed. The center frequency of the Gaussian filter is set to a preset center frequency, and the bandwidth is a preset bandwidth value.

5. The method for segmenting mucosal lesion regions from multispectral images of gastroscopy according to claim 1, characterized in that, In step 3, the spectral feature extraction network adopts a residual channel attention module, which contains a multi-layer residual structure. The number of output channels in each layer is a preset channel number sequence, the scaling factor of the residual connection is a preset value, and the compression ratio of the channel attention module is a preset ratio.

6. The method for segmenting mucosal lesion regions from multispectral images of gastroscopy according to claim 1, characterized in that, In step 3, the long-distance inter-band dependency model is constructed based on a self-attention mechanism. The number of attention heads is set to a preset number, the dimension of the key vector is a preset dimension, the dimension of the query vector is a preset dimension, the dimension of the value vector is a preset dimension, and the temperature parameter of the attention weight is a preset parameter value.

7. The method for segmenting mucosal lesion regions from multispectral images of gastroscopy according to claim 1, characterized in that, In step 4, the adaptive feature fusion module contains three parallel branches, which are responsible for local feature aggregation, global feature aggregation and band-specific feature extraction, respectively. The number of output channels of each branch is a preset number of channels. The fusion weight is dynamically determined by learnable parameters. The mathematical representation of the fusion operation is a weighted sum followed by batch normalization processing.

8. The method for segmenting mucosal lesion regions from multispectral images of gastroscopy according to claim 1, characterized in that, In step 4, the enhanced lesion-related feature response adopts a feature gating mechanism. The gating signal is generated by the spectral difference significance detection module, the detection threshold is set to a preset detection threshold, and the feature response attenuation coefficient below the threshold is a preset attenuation coefficient.

9. The method for segmenting mucosal lesion regions from multispectral images of gastroscopy according to claim 1, characterized in that, In step 5, the spatiotemporal convolution module includes two convolutional layers in the time dimension and three convolutional layers in the spatial dimension. The temporal convolution kernel size is a preset size, and the spatial convolution adopts a depth-separable convolutional structure. The number of convolution kernels in each layer is a preset sequence.

10. The method for segmenting mucosal lesion regions from multispectral images of gastroscopy according to claim 1, characterized in that, In step 6, the lightweight decoder network includes four upsampling stages, each with a preset upsampling rate. The skip connection fusion method is channel-level concatenation followed by convolutional compression. The activation function of the output layer is a preset activation function.

Images