Endoscope multi-region semantic perception time domain fusion adaptive white balance algorithm

Abstract

The invention discloses an endoscope multi-region semantic perception time domain fusion adaptive white balance algorithm. The algorithm comprises the following steps: acquiring an endoscope image; performing tissue region division on the endoscope image through a lightweight semantic segmentation network to obtain a plurality of region images; converting each regional image from an RGB space to a YCbCr color space, separating a brightness channel Y from chrominance channels Cb and Cr, and calculating a channel mean value of the chrominance channels Cb and Cr; calculating the average offset of the chrominance channels Cb and Cr according to the calculated channel average of the chrominance channels Cb and Cr; performing white point detection on each regional image to obtain an RGB three-channel mean value; combining the calculated mean offset of the chromaticity channels Cb and Cr, the RGB three-channel mean obtained by white point detection of each regional image and a statistical index to form a multi-dimensional regional color feature vector; and fusing historical frame features with the regional color feature vector to obtain a feature vector based on space-time fusion.

Description

Technical Field

[0001] This invention relates to the field of endoscopy technology, and more particularly to an endoscopy multi-region semantic perception temporal fusion adaptive white balance algorithm. Background Technology

[0002] Endoscopy, as an important medical diagnostic tool, is widely used for examining internal organs of the human body. However, due to the complexity of the internal environment, blood vessels are not clearly visible in images, their color is similar to that of surrounding tissues, and it is difficult to accurately distinguish the fine structures of blood vessels, which to some extent limits the application of endoscopy in the early diagnosis of diseases.

[0003] Existing endoscopic techniques face numerous challenges in image processing, particularly in scenarios with complex lighting or significant differences in tissue type. First, the application of global parameters during image processing by endoscopes fails to effectively meet the varying color reproduction requirements of different tissues (such as mucosa, blood vessels, and diseased tissues), resulting in localized color casts. Second, robustness in dynamic scenes is poor; fluctuations in light sources or instrument reflections can trigger abrupt color changes, and single-frame processing methods struggle to effectively suppress noise. Furthermore, achieving real-time processing of high-precision segmentation networks on embedded devices faces a trade-off between computational power and accuracy. Existing multi-region methods rely on fixed partitioning strategies, lack semantic information fusion for tissue types, and fail to consider the dynamic characteristics of the temporal domain.

[0004] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main objective of this invention is to provide an endoscope multi-region semantic perception temporal fusion adaptive white balance algorithm, which aims to solve the aforementioned technical problems in the prior art.

[0006] To achieve the above objectives, this invention provides an endoscope multi-region semantic perception temporal fusion adaptive white balance algorithm, comprising:

[0007] Acquire endoscopic images;

[0008] The endoscopic image is divided into tissue regions using a lightweight semantic segmentation network to obtain multiple region images;

[0009] Each region image is converted from RGB space to YCbCr color space, and the luminance channel Y and chrominance channels Cb and Cr are separated. The channel mean of chrominance channels Cb and Cr is calculated.

[0010] Based on the calculated channel mean values ​​of chrominance channels Cb and Cr, calculate the mean offset of chrominance channels Cb and Cr.

[0011] White spot detection is performed on each region of the image to obtain the mean value of the RGB three channels;

[0012] The calculated mean offsets of the chromaticity channels Cb and Cr, the mean values ​​of the RGB three channels obtained from the white point detection of each region image, and statistical indicators are combined to form a multidimensional regional color feature vector.

[0013] The historical frame features are fused with the regional color feature vector to obtain a feature vector based on spatiotemporal fusion.

[0014] The white balance gain parameter is calculated independently for each tissue region based on the feature vector after spatiotemporal fusion.

[0015] Preferably, in the endoscopic multi-region semantic perception temporal fusion adaptive white balance algorithm, the step of dividing the endoscopic image into tissue regions using a lightweight semantic segmentation network to obtain multiple region images includes:

[0016] Preprocess the acquired endoscopic images;

[0017] Network-based main feature extraction is performed on the preprocessed endoscopic images to obtain hierarchical feature maps;

[0018] Dual attention optimization is applied to the shallow feature maps extracted from the shallow layers of the network body;

[0019] The shallow and deep features optimized by dual attention are fused to obtain multiple region images.

[0020] Preferably, in the endoscopic multi-region semantic perception temporal fusion adaptive white balance algorithm, the shallow feature map extracted by the shallow layer of the network body is optimized using dual attention, including:

[0021] Apply channel attention mechanism to the input shallow feature map;

[0022] Based on the feature map after channel attention processing, a spatial attention mechanism is applied.

[0023] Preferably, in the endoscopic multi-region semantic perception temporal fusion adaptive white balance algorithm, the step of calculating the mean offset of chroma channels Cb and Cr based on the calculated channel mean values ​​of chroma channels Cb and Cr includes:

[0024] The difference between the calculated mean values ​​of the chromaticity channels Cb and Cr and the ideal neutral value (0) is used as the mean offset of Cb and the mean offset of Cr.

[0025] Preferably, in the endoscopic multi-region semantic perception temporal fusion adaptive white balance algorithm, the step of detecting white points in each region image to obtain the RGB three-channel mean includes:

[0026] Calculate the brightness value of each region of the image, and generate a brightness histogram based on the distribution of brightness values;

[0027] Based on the brightness histogram corresponding to each region image, sort the pixels from high to low brightness values, and select the top 5% of the brightness pixels in each region image as candidate white points.

[0028] Perform connected component analysis on the candidate white points in each region image, calculate the number of pixels in each connected region, and select regions with an area greater than 10 pixels and located in the center of the image as valid white points.

[0029] Take the RGB values ​​of the pixels in all valid white point areas in each region image, and use their RGB average as the reference white point of that region image, thus obtaining the RGB three-channel average of each region image;

[0030] The calculated mean offsets of the chromaticity channels Cb and Cr, the mean values ​​of the RGB three channels obtained from white point detection in each region of the image, and statistical indicators are combined to form a multidimensional regional color feature vector.

[0031] Preferably, in the endoscopic multi-region semantic perception temporal fusion adaptive white balance algorithm, in the step of combining the calculated mean offsets of the chromaticity channels Cb and Cr, the mean values ​​of the RGB three channels obtained from the white point detection of each region image, and statistical indicators to form a multi-dimensional regional color feature vector, the multi-dimensional regional color feature vector is 12-dimensional, including: mean offset of Cb channel, mean offset of Cr channel, mean of R channel, mean of G channel, mean of B channel, brightness contrast, standard deviation of Y channel, standard deviation of Cb channel, standard deviation of Cr channel, contrast of Cb channel, contrast of Cr channel, and percentage of effective region pixels.

[0032] Preferably, in the endoscopic multi-region semantic perception temporal fusion adaptive white balance algorithm, the step of fusing historical frame features with the regional color feature vector to obtain a feature vector based on spatiotemporal fusion includes:

[0033] Historical frame features are fused using an exponentially weighted sliding window with dynamic weights; the window size is adaptively adjusted based on the motion detection results of the current frame.

[0034] Preferably, in the endoscopic multi-region semantic perception temporal fusion adaptive white balance algorithm, the mouth size is adaptively adjusted according to the motion detection result of the current frame, including:

[0035] When the optical flow field variance of the image sequence is greater than the threshold, the window size is set to 10 frames;

[0036] When the variance of the optical flow field in the image sequence is less than or equal to the threshold, the window size is set to 5 frames.

[0037] Preferably, in the endoscopic multi-region semantic perception temporal fusion adaptive white balance algorithm, after the step of fusing historical frame features with the regional color feature vector to obtain a feature vector based on spatiotemporal fusion, and before the step of independently calculating the white balance gain parameter for each tissue region based on the feature vector based on spatiotemporal fusion, the algorithm further includes:

[0038] The spatiotemporal fusion-based feature vector of each organizational region is input into the pre-trained ARIMA time series analysis model. An autoregressive model is established for the mean Cb and Cr series of each region, and the spatiotemporal color features are output as the spatiotemporal fusion-based feature vector.

[0039] Preferably, in the endoscopic multi-region semantic perception temporal fusion adaptive white balance algorithm, the step of independently calculating the white balance gain parameter for each tissue region based on the feature vector after spatiotemporal fusion includes:

[0040] For each tissue region, the spatiotemporally fused feature vector will be input into a pre-trained linear regression model, which will output RGB gain coefficients.

[0041] Soft boundary blending technology is used to eliminate color banding between different tissue regions.

[0042] The present invention has at least the following beneficial effects:

[0043] This invention provides an endoscope multi-region semantic perception temporal fusion adaptive white balance algorithm to acquire endoscope images; the endoscope images are divided into tissue regions using a lightweight semantic segmentation network to obtain multiple region images; each region image is converted from RGB space to YCbCr color space, and the luminance channel Y and chrominance channels Cb and Cr are separated, and the channel mean values ​​of chrominance channels Cb and Cr are calculated; based on the calculated channel mean values ​​of chrominance channels Cb and Cr, the mean offset of chrominance channels Cb and Cr is calculated; white point detection is performed on each region image to obtain RGB three-way signals. The mean values ​​of the chroma channels Cb and Cr, the mean values ​​of the RGB three channels obtained from the white point detection of each region image, are combined with statistical indicators to form a multi-dimensional regional color feature vector. The historical frame features are fused with the regional color feature vector to obtain a feature vector based on spatiotemporal fusion. Based on the feature vector based on spatiotemporal fusion, the white balance gain parameter is calculated independently for each tissue region. This can solve the problems of existing multi-region methods relying on fixed partitioning strategies, lacking semantic information fusion of tissue types, and not considering the dynamic characteristics of the time domain. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of one embodiment of the endoscope multi-region semantic perception temporal fusion adaptive white balance algorithm provided by the present invention;

[0045] Figure 2 This is a schematic diagram of an embodiment of the endoscope system provided by the present invention.

[0046] The objectives, features, and advantages of this invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0047] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The present invention will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0048] In this embodiment of the invention, the term "and / or" describes the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The character " / " generally indicates that the preceding and following associated objects have an "or" relationship.

[0049] It should be noted that the terms "first," "second," etc., in the specification, claims, and drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0050] In this embodiment of the invention, the term "multiple" refers to two or more, and other quantifiers are similar.

[0051] In this invention, unless otherwise stated, directional terms such as "upper," "lower," "top," and "bottom" are generally used in relation to the direction shown in the accompanying drawings, or in relation to the vertical, perpendicular, or gravitational direction of the component itself; similarly, for ease of understanding and description, "inner" and "outer" refer to the inner and outer contours of each component itself, but the above directional terms are not intended to limit this invention.

[0052] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the various embodiments of the present invention will be described in detail below with reference to the accompanying drawings. However, those skilled in the art will understand that many technical details are presented in the embodiments of the present invention to facilitate a better understanding of the invention. However, the technical solutions claimed in the present invention can be implemented even without these technical details and various variations and modifications based on the following embodiments. The division of the following embodiments is for ease of description and should not constitute any limitation on the specific implementation of the present invention. The various embodiments can be combined with and referenced by each other without contradiction.

[0053] White balance algorithms are a key technique in image processing used to eliminate the influence of light source color temperature on image color. Its core objective is to ensure that "white" in an image appears as neutral white (i.e., R=G=B) under different lighting conditions, thereby restoring the true colors of the scene. Differences in color temperature between different light sources (such as incandescent lamps, sunlight, and fluorescent lamps) can cause color casts in images (e.g., yellowish under warm light, bluish under cool light). White balance algorithms compensate for these color casts by adjusting the gain of the RGB channels.

[0054] To address the aforementioned issues, this implementation involves an endoscope multi-region semantic perception temporal fusion adaptive white balance algorithm, which can be applied to endoscope systems.

[0055] The following describes the implementation details of the endoscope multi-region semantic perception temporal fusion adaptive white balance algorithm according to one embodiment of the present invention. The following implementation details are provided for ease of understanding only and are not necessary for implementing this solution. Figure 1 The diagram illustrates the endoscopic multi-region semantic perception temporal fusion adaptive white balance algorithm of the present invention.

[0056] At step S100, an endoscopic image is acquired.

[0057] In step S200, the endoscopic image is divided into tissue regions using a lightweight semantic segmentation network to obtain multiple region images.

[0058] It should be noted that endoscopic devices are embedded devices (such as the endoscopic host used in surgery), with limited computing power and storage space, so overly complex algorithms cannot be used; however, it is necessary to accurately segment different tissues (mucosa, blood vessels, diseased tissue, etc.), otherwise subsequent "personalized color correction for different tissues" would be impossible; at the same time, the segmentation results need to contain "semantic information", that is, it is necessary to clearly label which tissue each region belongs to (e.g., "this part is mucosa" "that part is blood vessel"), so that subsequent modules know which correction strategy to use for this region.

[0059] The present invention employs a "lightweight semantic segmentation network" because traditional high-precision segmentation networks are too resource-intensive (many parameters, large computational load) and unsuitable for endoscopic equipment. In contrast, the "lightweight semantic segmentation network" is both "lightweight" and "precise".

[0060] Specifically, step S200 includes steps S210 to S250.

[0061] In step S210, the acquired endoscopic images are preprocessed. Endoscopic images (such as gastroscopy and colonoscopy images) may have inherent problems, such as equipment reflection (local overbrightness), noise (bubbles, image graininess), uneven lighting (large differences in brightness within the same image), etc.

[0062] Preprocessing of acquired endoscopic images can include, but is not limited to, denoising, reflection suppression, normalization, and resizing. Preprocessing endoscopic images can reduce interference in the original image. Reflection suppression can reduce the brightness of reflective areas by enhancing local contrast, preventing overly bright pixels from interfering with feature extraction. Normalization can scale pixel values ​​from 0-255 to the range required by the network (e.g., 0 to 1 or -1 to 1), ensuring that the brightness and contrast of different images are on the same scale, facilitating stable network learning.

[0063] In step S220, the preprocessed endoscopic image is used to extract the main features of the network, resulting in a hierarchical feature map. The network is constructed using a "MobileNetV3 architecture optimized with depthwise separable convolution." This depthwise separable convolution significantly reduces computational cost (e.g., compressing the computational cost of traditional convolution to 1 / 10) while maintaining feature extraction capabilities, making the network suitable for embedded devices. The depthwise separable convolution is divided into two steps: depthwise convolution and pointwise convolution. Depthwise convolution processes each input channel (e.g., RGB three channels) with a separate 3×3 convolution kernel, extracting only the spatial features within that channel (e.g., the blood vessel orientation in the "red channel"), drastically reducing computational cost. Pointwise convolution uses a 1×1 convolution kernel to fuse features from all channels (e.g., combining "blood vessels in the red channel" and "mucosal texture in the green channel"), preserving cross-channel semantic relationships.

[0064] In step S230, dual attention optimization is applied to the shallow feature map extracted from the shallow layer of the network. A dual attention collaboration module is embedded in the shallow layer of the network. First, the color feature differentiation of tissues such as mucosa and blood vessels is enhanced through the channel attention mechanism (CBAM), and then the spatial attention mechanism is used to focus on the edge details of the lesion tissue.

[0065] Specifically, the channel attention mechanism (CBAM) is first applied to the input shallow feature map. Global average pooling and global max pooling are used to extract global information for each channel. After processing by a shared fully connected layer, channel weight vectors are generated. Channels with significant differences in color features (e.g., pink hue in mucous membranes, red hue in blood vessels) are assigned higher weights to enhance the color feature differentiation between different tissues and suppress the influence of interfering channels such as reflections, resulting in a feature map based on channel attention processing. This process includes extracting global channel information, generating channel weight vectors, and updating the feature map. Extracting global channel information involves performing two pooling operations on the shallow feature map: global average pooling and global max pooling. Generating channel weight vectors involves concatenating the results of the two pooling operations and inputting them into a shared fully connected layer (a small-sized neural network), which learns and outputs a weight vector of length C (each element corresponding to the importance of a channel). Updating the feature map involves applying the weight vector to the original shallow feature map using a channel-wise weighting (channel feature × corresponding weight) to obtain a feature map with "de-interference and strong color".

[0066] Subsequently, based on the feature map processed by channel attention, a spatial attention mechanism is applied. By performing average pooling and max pooling on the channel dimension, the results are concatenated and then passed through a convolutional layer to generate a spatial weight map. Higher weights are assigned to the spatial locations corresponding to the details of the lesion edge, enabling the network to focus on feature extraction in these edge regions and improving the segmentation clarity of the boundary between the lesion and the surrounding tissue.

[0067] This invention first enhances the color feature differences between different tissues through a "channel attention mechanism (CBAM)," such as mucous membranes being pinkish and blood vessels being reddish. Channel attention makes these two color features more prominent, reducing confusion. Then, a "spatial attention mechanism" focuses on the edge details of lesion tissue. Lesion tissue usually has blurred boundaries, and spatial attention allows the network to pay more attention to edge pixels, improving the clarity of segmentation and avoiding confusion between lesions and surrounding mucous membranes.

[0068] In step S240, the shallow and deep features optimized by dual attention are fused to obtain multiple region images. It should be noted that shallow features are typically defined as high resolution (e.g., 128×128) and rich in detail (clear edges), but have weak semantic judgment ability when used alone. Deep features are typically defined as low resolution (e.g., 32×32, due to multiple pooling compression), but have strong semantic information (able to clearly distinguish "this is a lesion" and "that is mucosa"), but have blurred boundaries (loss of detail).

[0069] Specifically, the deep low-resolution features are upsampled using bilinear interpolation and then fused with the shallow high-resolution features. Since the deep features (32×32) and the shallow features (128×128) have different sizes, they cannot be directly fused. Therefore, by upsampling the deep low-resolution features using bilinear interpolation, their sizes can be made completely consistent with those of the shallow features.

[0070] More specifically, the upsampled deep features are added pixel-by-pixel to the shallow features, ultimately outputting a probability map containing 3-5 tissue categories (mucosa, blood vessels, diseased tissue, background, etc.). Specifically, when constructing a multi-level feature fusion structure before the output layer, shallow features (with high resolution, preserving details such as tissue edges) and deep features (lower resolution but containing semantic information such as tissue category) are first selected from the network. The deep semantic features are then upsampled using bilinear interpolation to enlarge their size to match the shallow high-resolution features. Subsequently, the upsampled deep features and shallow high-resolution features are added element-by-element according to pixel position, ensuring that the fused feature simultaneously contains high-resolution details and deep semantic information. Finally, the fused feature is input into the output layer, and a probability map containing 3-5 tissue categories is generated, with each pixel corresponding to a probability value for belonging to each tissue category.

[0071] While lightweight networks require less computation, they are prone to insufficient feature resolution. For example, deep networks can identify "this is mucosa" (semantic information) but cannot clearly see the specific boundaries (detailed information); shallow networks can see clear boundaries (high resolution) but cannot distinguish between mucosa and blood vessels (weak semantics). This invention employs "multi-level feature fusion" to compensate for this deficiency. It adjusts the high-resolution features (including details) of shallow layers and the semantic features (including tissue type) of deep layers to the same size through "bilinear interpolation upsampling" and then adds them element-wise. The fused features have both clear boundary details and accurate tissue category information, ultimately outputting a "probability map" of 3-5 tissue types (the probability that each pixel belongs to a certain tissue type, for example, a pixel has a 90% probability of being mucosa and a 10% probability of being blood vessel).

[0072] The network parameters, after 8-bit quantization, require only 12MB of storage. During computation, only the effective segmented regions (accounting for 15%-20% of the entire image) are classified pixel-by-pixel, reducing computation by 85% compared to full-image processing. After segmentation, the system dynamically merges excessively small regions based on the area proportion of each tissue region: if the area of ​​a certain type of lesion tissue region is less than 3% of the total image area, it is merged into the adjacent mucosal region to avoid increasing the complexity of subsequent processing due to over-segmentation. (To ensure efficient network operation on embedded devices, two optimizations were made: after 8-bit quantization, the network parameters require only 12MB of storage (compared to hundreds of MB for traditional networks), making it suitable for device storage; during computation, only the segmented "effective regions" (occupying 15%-20% of the entire image, i.e., the actual tissue regions) are processed, reducing computation by 85% compared to full image processing and significantly improving speed. After segmentation, excessively small regions are dynamically merged. If the area of ​​a certain type of lesion tissue is less than 3% of the total image area (e.g., tiny lesions), it is merged into the adjacent mucosal region to avoid segmenting too many fragmented regions (oversegmentation), which would require separate calculations for each small region during subsequent processing, increasing complexity.)

[0073] In step S300, each region image is converted from RGB space to YCbCr color space, and the luminance channel Y and chrominance channels Cb and Cr are separated. The channel mean of the chrominance channels Cb and Cr is calculated. Using an improved grayscale world algorithm, the channel mean of the chrominance channels Cb and Cr is calculated, and pixels with saturation higher than 80% are excluded using a dynamic threshold (to avoid interference from highly saturated regions such as blood vessels on the global mean). The dynamic threshold is automatically determined by the peak position of the pixel saturation histogram within the region.

[0074] In step S400, the mean offsets of the chromaticity channels Cb and Cr are calculated based on the calculated mean values ​​of these channels. It should be noted that the difference between the mean values ​​of the chromaticity channels Cb and Cr and the "ideal neutral value (0)" is the mean offset of Cb and the mean offset of Cr, respectively. For example, if the mean value of the Cb channel in a certain mucosal region is 15, then the mean offset of Cb is 15 (reflecting that the region is generally yellowish, since a positive Cb value represents a yellowish tint); if the mean value of the Cr channel is -8, then the mean offset of Cr is -8 (reflecting that the region is generally greenish, since a negative Cr value represents a greenish tint).

[0075] In step S500, white point detection is performed on each region image to obtain the RGB three-channel mean.

[0076] The essence of white balance is to correct areas in an image that "should be white," eliminating color casts caused by lighting or device limitations. Ideally, white objects should have equal (or balanced in a specific ratio) values ​​across the RGB channels. White point detection involves finding the area in an image that is most likely to be "true white," using this area as a benchmark to calculate color correction parameters, ensuring that this area presents the correct white after correction, thereby driving the color restoration of the entire area.

[0077] Specifically, step S500 includes steps S510 to S540.

[0078] In step S510, the brightness value of each region image is calculated, and a brightness histogram is generated based on the brightness value distribution.

[0079] In step S520, the brightness histograms of each region image are sorted from high to low according to the brightness values, and the top 5% of the pixels in brightness of each region image are selected as candidate white points.

[0080] In step S530, connected component analysis is performed on the candidate white points in each region image. The number of pixels in each connected region is calculated, and regions with an area greater than 10 pixels and located in the center region of the image (excluding edge reflection interference) are selected as valid white points.

[0081] In step S540, the RGB values ​​of pixels in all valid white point areas of each region image are taken, and their RGB average is used as the reference white point of the region image, thus obtaining the RGB three-channel average of each region image.

[0082] In step S600, the calculated mean offsets of the chromaticity channels Cb and Cr, the mean values ​​of the RGB three channels obtained from the white point detection of each region image, and statistical indicators are combined to form a multidimensional regional color feature vector.

[0083] In some implementations, the multidimensional regional color feature vector is a 12-dimensional regional color feature vector. The 12-dimensional regional color feature vector includes the mean offset of the Cb channel, the mean offset of the Cr channel, the mean of the R channel, the mean of the G channel, the mean of the B channel, the brightness contrast (the difference between the maximum and minimum brightness of the Y channel, reflecting the brightness and darkness levels of the region), the standard deviation of the Y channel (the dispersion of brightness values, reflecting brightness uniformity), the standard deviation of the Cb channel (the dispersion of Cb chromaticity values, reflecting the uniformity of blue-yellow tones), the standard deviation of the Cr channel (the dispersion of Cr chromaticity values, reflecting the uniformity of red-green tones), the contrast of the Cb channel (the difference between the maximum and minimum Cb channel values, reflecting the blue-yellow tones), the contrast of the Cr channel (the difference between the maximum and minimum Cr channel values, reflecting the red-green tones), and the effective area pixel percentage (reflecting the region's integrity). By setting a 12-dimensional regional color feature vector, the degree of color shift and texture characteristics of the region can be comprehensively described.

[0084] In step S700, the historical frame features are fused with the regional color feature vector to obtain a feature vector based on spatiotemporal fusion. A single frame image may exhibit local color anomalies due to accidental factors (such as random sensor noise or instantaneous instrument reflections), or sudden changes in light source (such as sudden LED flickering) may cause abrupt color shifts. However, in a continuous image sequence (video), the scene and color changes between adjacent frames are usually smooth (for example, in normal surgery, tissue color does not suddenly change from reddish to greenish). Therefore, by fusing the color features of several past frames (historical frames), the abnormal fluctuations in a single frame can be "smoothed out," making the corrected color more consistent in time (for example, if the previous frame is reddish and the current frame is yellowish due to noise, the fused color will more closely approximate a smooth, realistic transition).

[0085] To suppress the effects of single-frame noise and sudden changes in light source, some implementations fuse historical frame features using an exponentially weighted sliding window with dynamic weights. The window size is adaptively adjusted based on the motion detection results of the current frame: when the optical flow field variance of the image sequence is greater than a threshold (indicating movement of the instrument or respiratory motion of the patient), the window size is set to 10 frames to enhance the smoothing effect of historical information; when the optical flow field variance of the image sequence is less than or equal to the threshold (stable scene), the window size is set to 5 frames to improve response speed.

[0086] The fusion weights of the feature vectors for each frame follow an exponential decay law. The exponential decay of the dynamic weights is determined according to the different importance of each frame within the window. For example, the current frame is the "latest information" and has the highest weight (e.g., 0.6); the previous frame is next (e.g., 0.2); and earlier frames have less impact on the current frame (decreasing by 0.1). This ensures the dominance of the current frame while using historical frames to "neutralize" noise from single frames (for example, if the current frame suddenly appears white due to reflection, the normal colors of historical frames will reduce this abnormal impact).

[0087] The number of historical frames can be set according to system settings or preset according to requirements. When the actual number of historical frames is insufficient compared to the preset number of historical frames, a deviation correction can be used for calibration.

[0088] For example, the weight of the current frame is 0.6, the previous frame is 0.2, and the weights of the two previous frames and earlier frames decrease in a geometric progression of 0.1. The initial 5 frames' fusion results are calibrated using a deviation correction formula to avoid parameter offsets caused by insufficient historical data in the initial stage. (When the system has just started (the first 5 frames), the number of historical frames is insufficient (e.g., only one historical frame is available for the second frame), directly using the above weights would lead to result deviations (e.g., noise in the first frame would severely affect the second frame). Therefore, the fusion results of the first 5 frames are calibrated using a "deviation correction formula" (essentially dynamically adjusting the weight ratio to reduce interference from a small amount of historical data) to prevent the initial calibration results from going astray.

[0089] Furthermore, for feature vectors based on spatiotemporal fusion, both high-frequency noise and low-frequency drift can affect the stability of spatiotemporal color feature representation. High-frequency noise refers to rapid, transient color fluctuations, such as a fleeting glare from equipment, causing a sudden spike in the Cr channel in a particular frame. Low-frequency drift refers to slow, continuous color changes, such as the gradual shift in color temperature from cool to warm during light source preheating, causing a continuous shift in the Cb channel.

[0090] To form a stable spatiotemporal color feature representation, it is also necessary to analyze the changing trends of color features. The feature vector of each tissue region based on spatiotemporal fusion is input into a pre-trained ARIMA time series analysis model. An autoregressive model is established for the mean Cb and Cr values ​​of each region to predict the color shift trend of the current frame, effectively filtering high-frequency noise and low-frequency drift, and finally outputting a more stable "spatiotemporal color feature".

[0091] ARIMA is a time series prediction model that establishes a "variation pattern model" based on the mean Cb and Cr values ​​(chromaticity features) of historical frames. It can predict the reasonable color offset range of the current frame, filter out high-frequency fluctuations and low-frequency drifts that do not conform to the rules, and finally output a more stable "spatiotemporal color feature" (which contains information of the current frame and conforms to the smooth change pattern over time).

[0092] In step S800, the white balance gain parameter is calculated independently for each tissue region based on the feature vector after spatiotemporal fusion.

[0093] It should be noted that different tissues (such as mucous membranes, blood vessels, and lesions) have different requirements for the "correct presentation" of colors (for example, mucous membranes need a natural skin tone, blood vessels need a clear red saturation, and lesions need to highlight the color difference from the surroundings). Therefore, the same set of correction parameters cannot be used; they must be "tailor-made." When adjacent areas are corrected using different parameters, obvious color discontinuities may appear at the junction (for example, the mucous membrane is pink, the adjacent blood vessel is bright red, and there is an abrupt change at the boundary). Technical means are needed to make the transition smoother.

[0094] Step S800 includes steps S810 and S820.

[0095] In step S810, for each tissue region, the feature vector after spatiotemporal fusion is input into a pre-trained linear regression model, which outputs RGB gain coefficients. The parameters of the linear regression model are obtained by training on more than 2000 sets of sample data from different light sources and tissue types collected offline. For mucosal regions, the focus is on color temperature correction (adjusting R / B gain balance), for vascular regions, the enhancement of saturation is maintained (fixed G gain benchmark), and for lesion tissue, the contrast is optimized (dynamically adjusting the R / G / B three-channel ratio).

[0096] In this invention, the "gain parameter" refers to a "coefficient" that adjusts the brightness of the RGB three channels (e.g., R gain = 1.2 indicates a 20% increase in the brightness of the red channel). The spatiotemporally fused feature vector contains 12-dimensional color features that are stable over time for that region, such as long-term Cb / Cr shifts and stable reference white points, which serve as the "basis" for calculating the gain. By establishing the correspondence between the feature vector and the RGB gain using a "linear regression model," the linear regression model, after being trained with over 2000 sets of sample data (covering different light sources and tissues), can "remember" how much R, G, and B gain should be used to correct when the feature vector of a certain region shows a "yellowish tint and a reddish reference white point" (e.g., increasing the B channel gain to offset the yellow and decreasing the R channel gain to balance the reddish tint).

[0097] Further, step S810 includes, based on the pre-trained linear regression model using the spatiotemporally fused feature vector input, outputting a gain parameter corresponding to the type of each tissue region.

[0098] The semantic segmentation and feature extraction module of this invention adopts OpenCL parallel computing and utilizes the processor's NEON unit to accelerate matrix operations; the feature vector storage and transmission use a 16-bit fixed-point number format, which reduces memory usage by half compared to floating-point operations; the modules are connected through a circular buffer to achieve pipelined processing, and region feature extraction is started immediately after image partitioning is completed, without waiting for the entire image to be processed. Finally, a processing speed of less than 50ms is achieved at 1080P resolution, which meets the needs of real-time clinical imaging in endoscopy.

[0099] Personalized strategies are applied to different tissues. For example, the mucosal area requires a focus on adjusting the color temperature (the color temperature is determined by the ratio of R and B channels; for instance, a high R / B ratio will result in a warm, yellowish tone, so adjusting the balance between the two will allow the mucosa to present a natural skin tone); the vascular area requires a focus on maintaining saturation (the red saturation of blood vessels is key to identification, so the G channel gain is fixed as a benchmark to avoid the green channel being too strong and diluting the red saturation); and the diseased tissue requires a focus on improving contrast (by dynamically adjusting the ratio of the R / G / B channels, the difference in brightness and color between the diseased area and the surrounding tissue is made more obvious, making it easier for doctors to observe).

[0100] Step S820 uses soft boundary fusion technology to eliminate color banding between different tissue regions.

[0101] Because different gain parameters are used in different areas, color "jumps" may occur at the boundaries. For example, the mucosa may be light pink after correction, while adjacent blood vessels may be dark red, resulting in a clear boundary line. "Soft boundary fusion" aims to eliminate this discontinuity: a 5-10 pixel "transition zone" (similar to a gradient area between two colors) is defined at the boundary of the region; the gain parameter of each pixel within the transition zone is calculated using a "Gaussian kernel": the more similar the features of adjacent regions (e.g., mild lesions and mucosa, with high cosine similarity), the standard deviation of the Gaussian kernel is set to 8 pixels (wide transition range, smoother color gradient); the greater the feature difference (e.g., blood vessels and background, with low cosine similarity), the standard deviation is set to 5 pixels (narrow transition range, avoiding color mixing); the pixel gain within the transition zone is a "weighted average" of the gains of adjacent regions, ultimately allowing the color at the boundary to smoothly transition from one region to another.

[0102] Specifically, a transition band with a width of 5-10 pixels is established centered on the boundary of the tissue region. The standard deviation of the Gaussian kernel is dynamically adjusted based on the cosine similarity of the feature vectors of adjacent tissue regions (set to 8 pixels if the similarity is high, and to 5 pixels if the similarity is low). The gain parameters of the boundary pixels are weighted and averaged to ensure a natural color transition.

[0103] In addition, this invention can also identify the type of light source. Different light sources (LED lights, xenon lights) have different color temperature change patterns, and the algorithm needs to be tailored accordingly. LED lights: color temperature fluctuates rapidly (>100K / second, such as sudden brightening or color change during surgery). In this case, the weight of historical frames is reduced (the weight of the current frame is increased to 0.8), allowing the algorithm to respond more quickly to changes in the current light source (completing the adjustment within 150ms) and avoiding correction lag. Xenon lights: color temperature changes gradually (small fluctuations). The default weight (0.6 for the current frame, with historical frames as an aid) is used to enhance stability through historical frame information and avoid single-frame noise affecting the correction results.

[0104] Specifically, the current light source is determined by the color temperature estimation values ​​of 5 consecutive frames (based on the correlation color temperature calculated from the white point detection results). If an LED light source is detected (color temperature fluctuation range > 100K / second), the weight of the historical frames in the time fusion window is automatically reduced (the weight of the current frame is increased to 0.8), shortening the algorithm's adaptation time to 150ms. If it is a xenon lamp (color temperature changes gradually), the default weight allocation is used to enhance stability.

[0105] To achieve the above objectives, the present invention also provides an endoscope system, such as... Figure 2 As shown, the endoscope includes at least one processor 901; and a memory 902 communicatively connected to at least one processor 901; wherein the memory 902 stores instructions that can be executed by at least one processor 901, and the instructions are executed by at least one processor 901 to enable at least one processor 901 to execute the aforementioned endoscope multi-region semantic perception temporal fusion adaptive white balance algorithm.

[0106] The memory 902 and processor 901 are connected via a bus, which can include any number of interconnecting buses and bridges. The bus connects various circuits of one or more processors 901 and memory 902. The bus can also connect various other circuits, such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. A bus interface provides an interface between the bus and the transceiver. The transceiver can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by processor 901 is transmitted over a wireless medium via an antenna, which further receives data and transmits it to processor 901.

[0107] Processor 901 is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory 902 can be used to store data used by processor 901 during operation.

[0108] To achieve the above objectives, the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor 901, implements the aforementioned endoscope multi-region semantic perception temporal fusion adaptive white balance algorithm.

[0109] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0110] Obviously, the embodiments described above are merely some, not all, embodiments of the present invention. Based on the embodiments of the present invention, those skilled in the art can make other variations or modifications without creative effort, and all such variations or modifications should fall within the scope of protection of the present invention.

Claims

1. An endoscopic multi-region semantic perception temporal fusion adaptive white balance algorithm, characterized in that, include: Acquire endoscopic images; The endoscopic image is divided into tissue regions using a lightweight semantic segmentation network to obtain multiple region images; Each region image is converted from RGB space to YCbCr color space, and the luminance channel Y and chrominance channels Cb and Cr are separated. The channel mean of chrominance channels Cb and Cr is calculated. Based on the calculated channel mean values ​​of chrominance channels Cb and Cr, calculate the mean offset of chrominance channels Cb and Cr. White spot detection is performed on each region of the image to obtain the mean value of the RGB three channels; The calculated mean offsets of the chromaticity channels Cb and Cr, the mean values ​​of the RGB three channels obtained from the white point detection of each region image, and statistical indicators are combined to form a multidimensional regional color feature vector. The historical frame features are fused with the regional color feature vector to obtain a feature vector based on spatiotemporal fusion. The white balance gain parameter is calculated independently for each tissue region based on the feature vector after spatiotemporal fusion.

2. The endoscopic multi-region semantic perception temporal fusion adaptive white balance algorithm as described in claim 1, characterized in that, The step of dividing the endoscopic image into tissue regions using a lightweight semantic segmentation network to obtain multiple region images includes: Preprocess the acquired endoscopic images; Network-based main feature extraction is performed on the preprocessed endoscopic images to obtain hierarchical feature maps; Dual attention optimization is applied to the shallow feature maps extracted from the shallow layers of the network body; The shallow and deep features optimized by dual attention are fused to obtain multiple region images.

3. The endoscopic multi-region semantic perception temporal fusion adaptive white balance algorithm as described in claim 2, characterized in that, The shallow feature map extracted from the shallow layers of the network body is optimized using dual attention, including: Apply channel attention mechanism to the input shallow feature map; Based on the feature map after channel attention processing, a spatial attention mechanism is applied.

4. The endoscopic multi-region semantic perception temporal fusion adaptive white balance algorithm as described in claim 1, characterized in that, The step of calculating the mean offset of chromaticity channels Cb and Cr based on the calculated channel mean values ​​of chromaticity channels Cb and Cr includes: The difference between the calculated mean values ​​of the chromaticity channels Cb and Cr and the ideal neutral value (0) is used as the mean offset of Cb and the mean offset of Cr.

5. The endoscopic multi-region semantic perception temporal fusion adaptive white balance algorithm as described in claim 1, characterized in that, The step of detecting white points in each region of the image to obtain the RGB three-channel mean includes: Calculate the brightness value of each region of the image, and generate a brightness histogram based on the distribution of brightness values; Based on the brightness histogram corresponding to each region image, sort the pixels from high to low brightness values, and select the top 5% of the brightness pixels in each region image as candidate white points. Perform connected component analysis on the candidate white points in each region image, calculate the number of pixels in each connected region, and select regions with an area greater than 10 pixels and located in the center of the image as valid white points. Take the RGB values ​​of the pixels in all valid white point areas in each region image, and use their RGB average as the reference white point of that region image, thus obtaining the RGB three-channel average of each region image; The calculated mean offsets of the chromaticity channels Cb and Cr, the mean values ​​of the RGB three channels obtained from white point detection in each region of the image, and statistical indicators are combined to form a multidimensional regional color feature vector.

6. The endoscopic multi-region semantic perception temporal fusion adaptive white balance algorithm as described in claim 5, characterized in that, In the step of combining the calculated mean offsets of the chroma channels Cb and Cr, the mean values ​​of the RGB three channels obtained from the white point detection of each region image, and statistical indicators to form a multidimensional regional color feature vector, the multidimensional regional color feature vector is 12-dimensional, including: mean offset of Cb channel, mean offset of Cr channel, mean of R channel, mean of G channel, mean of B channel, brightness contrast, standard deviation of Y channel, standard deviation of Cb channel, standard deviation of Cr channel, contrast of Cb channel, contrast of Cr channel, and percentage of effective region pixels.

7. The endoscopic multi-region semantic perception temporal fusion adaptive white balance algorithm as described in claim 1, characterized in that, The step of fusing historical frame features with the regional color feature vector to obtain a feature vector based on spatiotemporal fusion includes: Historical frame features are fused using an exponentially weighted sliding window with dynamic weights; the window size is adaptively adjusted based on the motion detection results of the current frame.

8. The endoscopic multi-region semantic perception temporal fusion adaptive white balance algorithm as described in claim 7, characterized in that, The aperture size is adaptively adjusted based on the motion detection results of the current frame, including: When the optical flow field variance of the image sequence is greater than the threshold, the window size is set to 10 frames; When the variance of the optical flow field in the image sequence is less than or equal to the threshold, the window size is set to 5 frames.

9. The endoscopic multi-region semantic perception temporal fusion adaptive white balance algorithm as described in claim 7, characterized in that, After the step of fusing historical frame features with the regional color feature vector to obtain a feature vector based on spatiotemporal fusion, and before the step of independently calculating the white balance gain parameter for each tissue region based on the feature vector based on spatiotemporal fusion, the method further includes: The spatiotemporal fusion-based feature vector of each organizational region is input into the pre-trained ARIMA time series analysis model. An autoregressive model is established for the mean Cb and Cr series of each region, and the spatiotemporal color features are output as the spatiotemporal fusion-based feature vector.

10. The endoscopic multi-region semantic perception temporal fusion adaptive white balance algorithm as described in claim 1, characterized in that, The step of independently calculating the white balance gain parameter for each tissue region based on the feature vector after spatiotemporal fusion includes: For each tissue region, the spatiotemporally fused feature vector will be input into a pre-trained linear regression model, which will output RGB gain coefficients. Soft boundary blending technology is used to eliminate color banding between different tissue regions.

Images