Image highlight elimination method and device, computer equipment and storage medium

Abstract

The invention relates to the technical field of image processing, and discloses an image highlight elimination method and device, computer equipment and a storage medium, and the method comprises the steps: obtaining an initial image containing a mirror highlight feature; based on the initial image, extracting a frequency domain feature of a distribution rule of the mirror surface highlight feature and a spatial domain feature of a spatial form of the mirror surface highlight feature, and fusing the frequency domain feature and the spatial domain feature to obtain a fused feature map; performing feature optimization on mirror surface highlight features in the fused feature map to obtain an enhanced feature map; according to the method, the global cognition and local form modeling capability of the model on highlight distribution is enhanced through the fusion frequency domain and spatial domain features of the image, semantic structures and texture details covered by highlight are recovered in combination with a feature optimization mechanism, and the specular highlight is eliminated. And the removal effect of the mirror surface highlight in the image and the image reconstruction quality are improved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically to a method, apparatus, computer device, and storage medium for eliminating image highlights. Background Technology

[0002] In scenarios such as intelligent transportation, autonomous driving, and underground parking management, image recognition and perception systems play a crucial role in accurately perceiving the ground environment. However, due to the unique lighting conditions and ground materials of underground parking lots, specular reflection often occurs, resulting in large areas of "spectral highlights" in the images. These highlights severely interfere with the texture information and semantic content of the images, affecting not only the performance of downstream tasks such as object detection, path planning, and parking space recognition, but also potentially leading to system decision-making errors and safety hazards.

[0003] The main approaches to removing specular highlights from images include: traditional image processing methods, deep learning methods based on CNNs (Convolutional Neural Networks), and multi-task learning methods. Traditional methods rely on highlight detection and image inpainting techniques, reconstructing images based on brightness differences and edge information. However, their effectiveness is limited when faced with missing textures or blurred edges, and they lack the ability to model complex semantics. CNN-based methods utilize networks such as U-Net or GANs to learn semantic context; however, limited by the local receptive field of convolutions, they struggle to effectively capture long-distance dependencies and complex spatial relationships, leading to performance degradation when highlight areas are large or structurally complex. Multi-task learning methods attempt to integrate highlight detection and removal to improve consistency, but their model structures are complex, training is unstable, and they lack a systematic design for learning phased features. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a method, apparatus, computer device, and storage medium for removing image specular highlights, in order to address the problem of how to improve the spatial modeling, semantic understanding, and structural reconstruction capabilities of image specular highlight removal technology.

[0005] In a first aspect, embodiments of the present invention provide a method for eliminating image highlights, the method comprising: Obtain an initial image containing specular highlight features; Based on the initial image, the frequency domain features of the distribution pattern of the specular highlight features and the spatial domain features of the spatial morphology of the specular highlight features are extracted, and the frequency domain features and the spatial domain features are fused to obtain a fused feature map. The specular highlight features in the fused feature map are optimized to obtain an enhanced feature map. Based on the enhanced feature map, the image is reconstructed to obtain a restored image with specular highlights removed.

[0006] This invention acquires an initial image containing specular highlight features; extracts frequency domain features of the distribution pattern of specular highlight features and spatial domain features of the spatial morphology of specular highlight features based on the initial image, and fuses the frequency domain features and spatial domain features to obtain a fused feature map; optimizes the specular highlight features in the fused feature map to obtain an enhanced feature map; and reconstructs the image based on the enhanced feature map to obtain a restored image with specular highlights removed. This invention enhances the model's global cognition and local morphological modeling ability of highlight distribution by fusing the frequency domain and spatial domain features of the image, and restores the semantic structure and texture details obscured by highlights by combining feature optimization mechanisms, thereby improving the removal effect of specular highlights in the image and the quality of image reconstruction.

[0007] In conjunction with the first aspect, in one embodiment, acquiring the initial image containing specular highlight features includes: Obtain the image to be processed; Feature extraction is performed on the image to be processed, and a downsampling operation is performed to obtain an initial feature map. Feature extraction is then performed on the initial feature map, and a downsampling operation is performed again to obtain an initial image containing specular highlight features.

[0008] The SHRNet network in this invention uses a minimalist SiCNN module for fast downsampling in the first two layers of the network. The lightweight front end is responsible for quickly capturing and condensing basic visual features (such as edges and colors) to form an initial image rich in highlight information, which greatly reduces the computational complexity and memory usage in the initial stage.

[0009] In conjunction with the first aspect or its corresponding implementation, in one implementation, after acquiring the image to be processed, the method includes: The image to be processed is input into a specular highlight restoration network, wherein the specular highlight restoration network comprises at least two convolutional networks, the convolutional networks comprising a first convolutional network and a second convolutional network, and each convolutional network comprising a convolutional layer, a normalization layer and an activation function; The basic features of the image to be processed are extracted through the convolutional layers in the first convolutional network; The basic features are normalized by the normalization layer in the first convolutional network to obtain a first feature map, and the first feature map is non-linearly activated by the activation function in the first convolutional network to obtain a third feature map. The third feature map is downsampled to obtain the initial feature map; Highlight correlation features of the initial feature map are extracted through the convolutional layers in the second convolutional network; The hyperspectral correlation features are normalized by the normalization layer in the second convolutional network to obtain a stable feature map; The stable feature map is nonlinearly activated by the activation function of the second convolutional network to obtain a deep specular feature map; The deep specular feature map is downsampled to obtain the initial image.

[0010] The highlight restoration network in this invention adopts a cascaded downsampling structure composed of at least two convolutional networks to extract features from the image to be processed in a hierarchical manner, effectively preventing the loss of details in the image.

[0011] In conjunction with the first aspect, in one embodiment, the step of extracting the frequency domain features of the distribution pattern of the specular highlight features and the spatial domain features of the spatial morphology of the specular highlight features based on the initial image includes: The initial image is input into the local blending domain attention module in the specular highlight restoration network. The local blending domain attention module is set after the convolutional network in the specular highlight restoration network and is used to carry out feature processing of the initial image containing specular highlight features obtained by two downsampling operations. The initial image is obtained by performing pixel shifting in the horizontal or vertical direction using the local hybrid domain attention module; The initial image is obtained by performing pixel shifting in the horizontal or vertical direction using the local hybrid domain attention module; Perform a Fourier transform on the pixel-shifted initial image to extract the frequency spectrum of the initial image; The low-frequency and high-frequency structural features of the initial image are obtained based on the frequency spectrum. Based on the low-frequency structural features and the high-frequency structural features, the initial image after pixel shifting is divided into several local windows, and the frequency domain features of the initial image after pixel shifting are extracted in each local window; The initial image after pixel shifting is converted into a three-dimensional feature image, and the spatial domain features of the three-dimensional feature image are extracted.

[0012] This invention introduces a pixel shifting mechanism to overcome the boundary limitations of fixed window division. This allows for the fusion of contextual information from different windows in subsequent local window processing, enhancing the perception of highlight region boundaries and significantly improving the continuity and naturalness of the restored texture, while avoiding artifacts caused by window fragmentation. Secondly, Fourier transform is performed after pixel shifting, ensuring that the subsequently extracted low-frequency structural features and high-frequency detail features inherently contain cross-regional global information. This enhances the integrity of the frequency domain prior. Guided by the enhanced frequency domain features, spatial domain features are extracted. By comprehensively utilizing the structural priors of the frequency domain and the local details of the spatial domain, more accurate and natural restoration and reconstruction of occluded textures can be achieved in complex specular reflection environments, improving the effect and robustness of highlight removal.

[0013] In conjunction with the first aspect or its corresponding implementation, in one implementation, the step of performing a horizontal or vertical pixel shift on the initial image to obtain the pixel-shifted initial image includes: Determine the pixel shift parameters of the initial image, wherein the pixel shift parameters include shift direction and / or shift distance; The initial image is segmented based on the pixel shift parameters of the initial image to obtain the retained region and the region to be filled; The area to be filled is filled with pixel values ​​to generate a filled area; The retained region and the filled region are stitched together along the spatial dimensions of the initial image to obtain the pixel-shifted initial image.

[0014] The pixel shifting mechanism of this invention shifts pixels in the input initial image in the horizontal or vertical direction, which can rearrange the spatial distribution of pixels in the initial image, break the spatial limitations imposed by the fixed local window boundaries, forcibly promote the fusion of contextual information across windows, and improve the interaction capability of semantic information between different windows.

[0015] In conjunction with the first aspect, in one implementation, feature optimization is performed on the specular highlight features in the fused feature map to obtain an enhanced feature map, including: The fused feature map is input into the global attention module of the specular restoration network. This global attention module is positioned after the local hybrid domain attention module in the specular restoration network and is used to perform feature processing on the fused feature map generated by fusing the frequency domain and spatial domain features extracted by the local hybrid domain attention module. The global attention module obtains a three-dimensional vector from the fused feature map and divides this vector into several vector groups. For each vector group, multi-head attention computation is performed to obtain the context enhancement features corresponding to each vector group. The context enhancement features of all vector groups are concatenated to generate the context feature matrix of the fused feature map, and an enhanced feature map is generated based on this context feature matrix.

[0016] This invention divides a 3D vector into several independent vector groups along a specific dimension for parallel processing, reducing the computational complexity and memory consumption of multi-head attention mechanisms. Attention calculations are performed independently within each vector group, enabling the focus on and enhancement of key contextual information from different perspectives. Finally, by concatenating and integrating the optimized results of all vector groups, the generated enhanced feature map retains the complete global context and is rich in multi-angle detail features, thereby significantly improving the ability to discriminate complex specular highlight features and the accuracy and quality of subsequent image reconstruction.

[0017] In conjunction with the first aspect or its corresponding implementation, in one implementation, the step of performing multi-head attention computation for each vector group to obtain context-enhanced features corresponding to each vector group includes: A query matrix, key matrix, and value matrix for each vector group are generated through linear transformation, and the query matrix, key matrix, and value matrix are divided into multiple attention heads. For each attention head, the dot product of the query matrix and the key matrix is ​​calculated to obtain the dot product result. The dot product result is then normalized to obtain the attention weight matrix. Multiply the attention weight matrix by the corresponding value matrix to obtain the output of each attention head. Concatenate the outputs of multiple attention heads to obtain the context-enhanced features of the vector group.

[0018] This invention maps the features of each vector group to multiple attention heads for parallel computation, allowing each attention head to independently focus on different contextual dependencies (such as texture, shape, or lighting), thus capturing complex feature interactions within the specular highlight region more comprehensively and precisely. Secondly, the extracted attention weights are dynamically fused with the original features, thereby enhancing key features related to specular highlights and suppressing irrelevant or interfering information, improving the model's semantic understanding and discrimination capabilities for specular highlights.

[0019] In conjunction with the first aspect or its corresponding implementation, in one implementation, the step of concatenating the context enhancement features of all vector groups to generate the context feature matrix of the fused feature map, and generating the enhanced feature map based on the context feature matrix, includes: The residuals of the context-enhanced features and the three-dimensional vector are added together to obtain the residual result. The residual results are normalized to generate the context feature matrix of the fused feature map, and an enhanced feature map is generated based on the context feature matrix.

[0020] This invention adds the context-enhanced features to the 3D vector using residual addition, effectively preserving the low-level details and original information in the input fused feature map. This ensures that key texture and structural features are not lost or overwritten while introducing global context. Normalizing the residual results stabilizes the training process and accelerates model convergence, improving training efficiency and stability.

[0021] Secondly, embodiments of the present invention provide an image highlight removal device, the device comprising: The image acquisition module is used to acquire an initial image containing specular highlight features; The feature fusion module is used to extract the frequency domain features of the distribution pattern of the specular highlight features and the spatial domain features of the spatial morphology of the specular highlight features based on the initial image, and fuse the frequency domain features and the spatial domain features to obtain a fused feature map. The image enhancement module is used to optimize the specular highlight features in the fused feature map to obtain an enhanced feature map; The image reconstruction module is used to reconstruct the image based on the enhanced feature map to obtain a restored image with specular highlights removed.

[0022] Thirdly, embodiments of the present invention provide a computer device, including: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to perform the image highlight removal method described in the first aspect or any corresponding embodiment.

[0023] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing computer instructions for causing a computer to perform the image highlight removal method described in the first aspect or any corresponding embodiment. Attached Figure Description

[0024] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0025] Figure 1 This is a schematic flowchart of an image highlight removal method according to some embodiments of the present invention; Figure 2 This is a schematic diagram of the network structure in an image highlight removal method according to some embodiments of the present invention; Figure 3 This is a schematic diagram of another network structure in an image highlight removal method according to some embodiments of the present invention; Figure 4 This is a schematic diagram of another network structure in an image highlight removal method according to some embodiments of the present invention; Figure 5 This is a structural block diagram of an image highlight removal device according to an embodiment of the present invention; Figure 6 This is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] According to an embodiment of the present invention, an embodiment of a method for eliminating image highlights is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0028] In scenarios such as intelligent transportation, autonomous driving, and underground parking management, image recognition and perception systems play a crucial role in accurately perceiving the ground environment. However, due to the unique lighting conditions and ground materials of underground parking lots, specular reflection often occurs, resulting in large areas of "spectral highlights" in the images. These highlights severely interfere with the texture information and semantic content of the images, affecting not only the performance of downstream tasks such as object detection, path planning, and parking space recognition, but also potentially leading to system decision-making errors and safety hazards.

[0029] The main approaches to removing specular highlights from images include: traditional image processing methods, deep learning methods based on CNNs (Convolutional Neural Networks), and multi-task learning methods. Traditional methods rely on highlight detection and image inpainting techniques, reconstructing images based on brightness differences and edge information. However, their effectiveness is limited when faced with missing textures or blurred edges, and they lack the ability to model complex semantics. CNN-based methods utilize networks such as U-Net or GANs to learn semantic context; however, limited by the local receptive field of convolutions, they struggle to effectively capture long-distance dependencies and complex spatial relationships, leading to performance degradation when highlight areas are large or structurally complex. Multi-task learning methods attempt to integrate highlight detection and removal to improve consistency, but their model structures are complex, training is unstable, and they lack a systematic design for learning phased features.

[0030] Based on this, this embodiment provides a method for eliminating image highlights. Figure 1 This is a flowchart of an image highlight removal method according to an embodiment of the present invention, such as... Figure 1 As shown, the process includes the following steps: Step S101: Obtain an initial image containing specular highlight features.

[0031] Step S102: Based on the initial image, extract the frequency domain features of the distribution pattern of specular highlight features and the spatial domain features of the spatial morphology of specular highlight features, and fuse the frequency domain features and spatial domain features to obtain a fused feature map.

[0032] Step S103: Perform feature optimization on the specular highlight features in the fused feature map to obtain an enhanced feature map.

[0033] Step S104: Reconstruct the image based on the enhanced feature map to obtain a repaired image with specular highlights removed.

[0034] The image specular highlight removal method provided in this embodiment obtains an initial image containing specular highlight features; extracts frequency domain features of the distribution pattern of specular highlight features and spatial domain features of the spatial morphology of specular highlight features based on the initial image, and fuses the frequency domain features and spatial domain features to obtain a fused feature map; performs feature optimization on the specular highlight features in the fused feature map to obtain an enhanced feature map; and reconstructs the image based on the enhanced feature map to obtain a restored image with specular highlights removed. This invention enhances the model's global cognition and local morphological modeling ability of highlight distribution by fusing frequency domain and spatial domain features of the image, and restores the semantic structure and texture details covered by highlights by combining feature optimization mechanisms, thereby improving the removal effect of specular highlights in the image and the image reconstruction quality.

[0035] Reference Figure 2This invention provides an SHRNet (Specular Highlight RestorationNet) network. The SHRNet uses the U-Net network architecture as its backbone and embeds SiCNN, LHDAT, and CGAT network structure modules into the encoder (downsampling path) and decoder (upsampling path) of the standard U-Net. Specifically, the first and second layers of the SHRNet network are composed of a SiCNN convolutional network, which consists of a single convolutional layer, a normalization layer, and an activation function connected sequentially, used for preliminary shallow feature extraction. The third layer of the encoder and its symmetrical layer in the decoder embed a hybrid attention module (i.e., the LHDAT module) to extract local pixel relationships within the highlight region. Through a hybrid domain (i.e., spatial and frequency domain) attention mechanism, the model's ability to perceive and model local details such as highlight edges and subtle textures is enhanced, preventing the loss of image details during feature transfer. Secondly, a Global Attention (CGAT) module is embedded at the connection between the encoder and decoder. This module weights all channel dimensions to integrate the global illumination context information of the image, enabling the SHRNet network to judge and coordinate the intensity of highlight suppression in different regions as a whole, ensuring the consistency of illumination in the restored image. Furthermore, the LHDAT and CGAT modules include a Frequency Domain Feature Fusion (FFM) module. The FFM module transforms the spatial features of the image to the frequency domain using Fast Fourier Transform, separating and selectively processing high-frequency result features (typically corresponding to texture and edges) and low-frequency structural features (typically corresponding to illumination and color) in the frequency domain. This utilizes frequency information to assist in reconstructing textures masked by highlights. In summary, this embodiment of the invention achieves shallow feature extraction through the SiCNN module and, through the organic cooperation of the LHDAT and CGAT modules in the local, global, and frequency domain dimensions, constructs an SHRNet network capable of accurately modeling highlight characteristics and achieving high-quality image restoration. Specifically, the two SiCNN modules downsample the input initial image twice to obtain an initial image of size N×4C×(H / 4)×(W / 4). The initial image of size N×4C×(H / 4)×(W / 4) is then input into the LHDAT module for processing to obtain a fused feature map. After downsampling the fused feature map by 2 times, the LHDAT module inputs an enhanced feature map of size N×8C×(H / 8)×(W / 8) into the CGAT module.

[0036] For step S101, an initial image containing specular highlight features is obtained.

[0037] The system receives an image containing specular highlights, for example, an RGB three-channel color image of size H×W×3. The image is input into the SHRNet network, which converts the input H×W×3 image size to N×C×H×W. Here, N represents the batch size during training (N=1 during inference prediction), C is the number of feature channels determined after the initial convolution operation, and C is a configurable hyperparameter.

[0038] The resized image to be processed is input into the first layer of the SHRNet network, the SiCNN module. The SiCNN module consists of a single convolutional layer, a normalization layer (such as InstanceNorm or BatchNorm), and a non-linear activation function (such as ReLU) connected sequentially. The SiCNN module performs initial, coarse-grained feature extraction on the resized image, while simultaneously performing a 2x downsampling operation to quickly capture the most basic low-level visual features such as edges and colors, and reduce the spatial resolution. That is, the image size changes from N×C×H×W to N×2C×(H / 2)×(W / 2), thus obtaining the initial feature map output by the first layer of the SiCNN module. This initial feature map is then input into a second, lightweight SiCNN module with the same structure, i.e., the second layer of the SiCNN module. The second layer of the SiCNN module performs deeper feature extraction on the initial feature map and performs another 2x downsampling operation to obtain the final initial image. After the input image undergoes two levels of downsampling through the first and second SiCNN modules, an initial image is obtained with significantly compressed spatial dimensions and an increased number of feature channels. The specific steps include: Obtain the image to be processed.

[0039] Feature extraction is performed on the image to be processed, and downsampling is performed to obtain an initial feature map.

[0040] Feature extraction is performed on the initial feature map, and then downsampling is performed again to obtain an initial image containing specular highlight features.

[0041] In this embodiment of the invention, the SHRNet network uses a minimalist SiCNN module for fast downsampling in the first two layers of the network. The lightweight front end is responsible for quickly capturing and condensing basic visual features (such as edges and colors) to form an initial image rich in highlight information, which greatly reduces the computational complexity and memory usage in the initial stage.

[0042] Furthermore, feature extraction is performed on the image to be processed, and a downsampling operation is performed to obtain an initial feature map. Feature extraction is then performed on the initial feature map, and a downsampling operation is performed again to obtain an initial image containing specular highlight features, including: The image to be processed is input into a specular inpainting network, wherein the specular inpainting network contains at least two convolutional networks, the convolutional networks include a first convolutional network and a second convolutional network, and each convolutional network includes a convolutional layer, a normalization layer and an activation function; The basic features of the image to be processed are extracted through the convolutional layers in the first convolutional network; The basic features are normalized by the normalization layer in the first convolutional network to obtain the first feature map, and the first feature map is non-linearly activated by the activation function in the first convolutional network to obtain the third feature map. The third feature map is downsampled to obtain the initial feature map; Highlight correlation features of the initial feature map are extracted through the convolutional layers in the second convolutional network; The hyperspectral correlation features are normalized by the normalization layer in the second convolutional network to obtain a stable feature map; The stable feature map is non-linearly activated by the activation function of the second convolutional network to obtain the deep specular feature map; The deep specular feature map is downsampled to obtain the initial image.

[0043] This embodiment utilizes a first convolutional network to perform basic feature extraction and normalization on the input image. The downsampling operation reduces computational complexity while preserving effective information in the image. The second convolutional network further extracts highlight-related features and enhances the perception of highlight regions through secondary normalization and activation. This enables the network to more accurately distinguish between highlight and normal regions, enhancing the model's adaptability and generalization ability in complex highlight scenes.

[0044] Step S102: Based on the initial image, extract the frequency domain features of the distribution pattern of specular highlight features and the spatial domain features of the spatial morphology of specular highlight features, and fuse the frequency domain features and spatial domain features to obtain a fused feature map.

[0045] In this embodiment, the third layer of the SHRNet network is a Local Hybrid Domain Attention (LHDAT) module. The LHDAT module operates at a low resolution (e.g., H / 4, W / 4) after spatial size compression to achieve a balance between computational efficiency and model performance. (Refer to...) Figure 3 The diagram shows the network structure of the LHDAT module in the SHRNet network.

[0046] The LHDAT module receives an initial image of size N×4C×(H / 4)×(W / 4) after being downsampled by 2x by the SiCNN module. It then performs pixel shifting on the initial image in the horizontal or vertical direction to eliminate the spatial constraints imposed by fixed local window boundaries. This forces the fusion of contextual information across windows, thereby enhancing the interaction of semantic information between different windows. When processing highlight edge regions in an image, the LHDAT module can obtain broader contextual support, resulting in stronger stability and effectively improving the texture continuity of the restored image.

[0047] The LHDAT module comprises a first image processing branch and a second image processing branch. After pixel shifting, the first image processing branch performs a Fourier transform on the input initial image using a frequency processor, converting it from the spatial domain to the frequency domain to obtain the frequency spectrum of the initial image. Based on this frequency spectrum, the LHDAT module can parse the frequency domain information of the initial image, including low-frequency and high-frequency structural features. Low-frequency structural features correspond to the spectral center, representing the slowly changing overall structure and illumination background of the image. High-frequency structural features correspond to the spectral edges, representing the rapidly changing edges and textures in the image. These features can be used to guide the SHRNet network in restoring diffuse textures, effectively avoiding blurring, texture loss, or artifacts in the restoration results of the image to be processed.

[0048] Furthermore, based on low-frequency and high-frequency structural features, the initial image after pixel shifting is divided into several local windows. In each local window, the frequency domain features of the initial image after pixel shifting are extracted, and the frequency domain features are used as the query vector in the Transformer attention mechanism.

[0049] Specifically, attached Figure 3 The (1×1Conv) in the diagram is a convolutional component used to linearly map the initial image after windowing to the query, key, and value spaces. On the other hand, (1×1Conv) can also reduce or increase the number of feature channels without changing the spatial dimensions of the image.

[0050] Reshape stands for dimensionality reshaping, used to transform a two-dimensional feature matrix into a three-dimensional structure suitable for multi-head parallel computation. For example, a feature matrix with shape [batch size, sequence length, number of channels] can be reshaped to [batch size, sequence length, number of heads, channel dimension per head].

[0051] Reshaping chunks is used to uniformly divide a feature matrix into multiple non-overlapping chunks or segments along a specific dimension (usually the channel dimension). For example, if a feature matrix has C channels and the goal is to segment it into 8 attention heads, it will be split into 8 tensors of the same shape along the channel dimension, with each tensor having (C / 8) channels. After chunking, each chunk is typically reshaped individually to further adjust its dimensions.

[0052] The second image processing branch processes the initial image after pixel shifting in the spatial domain, extracts its spatial shape and contextual information, and generates the key vector and value vector in the Transformer attention mechanism, respectively.

[0053] Finally, the query vector from the frequency domain branch, along with the key and value vectors from the spatial domain branch, are input into the Transformer structure for deep fusion of spatial and frequency domain features. Thus, the SHRNet network can capture and reconstruct high-quality semantic content obscured by specular highlights by comprehensively utilizing spatial context and frequency domain texture in an image, without relying on specular masks or any additional prior knowledge. Specifically, the steps include: The initial image is input into the Local Mixing Domain Attention Module in the Highlight Inpainting Network. The Local Mixing Domain Attention Module is set after the convolutional network in the Highlight Inpainting Network and is used to carry out feature processing of the initial image containing specular highlight features obtained by two downsampling operations. The initial image is obtained by horizontally or vertically shifting pixels in the local blending domain attention module.

[0054] Perform a Fourier transform on the initial image after pixel shifting to extract the frequency spectrum of the initial image.

[0055] Low-frequency and high-frequency structural features of the initial image are obtained based on the frequency spectrum.

[0056] Based on low-frequency and high-frequency structural features, the initial image after pixel shifting is divided into several local windows, and the frequency domain features of the initial image after pixel shifting are extracted in each local window.

[0057] The initial image after pixel shifting is converted into a three-dimensional feature image, and the spatial domain features of the three-dimensional feature image are extracted.

[0058] This embodiment introduces a pixel shifting mechanism to overcome the boundary limitations of fixed window division. This allows for the fusion of contextual information from different windows in subsequent local window processing, enhancing the perception of highlight region boundaries and significantly improving the continuity and naturalness of the restored texture, while avoiding artifacts caused by window fragmentation. Secondly, Fourier transform is performed after pixel shifting, ensuring that the subsequently extracted low-frequency structural features and high-frequency detail features inherently contain cross-regional global information. This enhances the integrity of the frequency domain prior. Guided by the enhanced frequency domain features, spatial domain features are extracted. By comprehensively utilizing the structural prior in the frequency domain and the local details in the spatial domain, more accurate and natural restoration and reconstruction of occluded textures can be achieved in complex specular reflection environments, improving the effect and robustness of highlight removal.

[0059] In one embodiment of the present invention, when performing horizontal or vertical pixel shifting on an initial image, it is first necessary to determine the pixel shift parameters, which include the shift direction (e.g., horizontally to the right or vertically downward) and / or the shift distance (e.g., shifting by a certain number of pixels, which can be preset to half the window size or other empirical values). Next, based on the determined shift direction and distance, the initial image is segmented. Specifically, along the shift direction, the initial image is divided into two parts: a retained region and a region to be filled. The retained region contains pixel data from the initial image that will remain within the effective frame after the shift, while the region to be filled contains blank areas created by the pixel shift that currently lack valid pixel data at the corresponding positions in the original image.

[0060] The area to be filled is assigned a valid pixel value to generate the filling area. In a preferred embodiment, a wraparound filling method can be used, that is, pixels that have exceeded the original boundary due to displacement in the initial image are cyclically filled into the area to be filled. For example, when shifting horizontally to the right, the pixel column that was moved out from the leftmost edge of the image will be used to fill the newly generated blank column on the rightmost edge. Alternatively, zero-value filling or edge copy filling strategies can also be used, and this embodiment is not limited to these.

[0061] After the filled area is generated, the reserved area and the filled area are stitched together according to their relative positions in the initial image to generate the pixel-shifted initial image. Through the pixel shifting mechanism of this embodiment, the spatial distribution of pixels in the initial image can be rearranged, effectively breaking the boundaries of fixed local windows in subsequent processing. Specifically, the steps include the following: Determine the pixel shift parameters of the initial image, including the shift direction and / or shift distance.

[0062] The initial image is segmented based on the pixel shift parameters of the initial image to obtain the retained region and the region to be filled.

[0063] The area to be filled is filled with pixel values ​​to generate the filled area.

[0064] The retained region and the filled region are stitched together along the spatial dimensions of the initial image to obtain the pixel-shifted initial image.

[0065] The pixel shifting mechanism in this embodiment shifts pixels in the input initial image in the horizontal or vertical direction, which can rearrange the spatial distribution of pixels in the initial image, break the spatial limitations imposed by the fixed local window boundaries, forcibly promote the fusion of contextual information across windows, and improve the interaction capability of semantic information between different windows.

[0066] Step S103: Perform feature optimization on the specular highlight features in the fused feature map to obtain an enhanced feature map.

[0067] Reference Figure 4 The fused feature map is downsampled by a factor of 2 and then input into the fourth layer of the SHRNet network, the channel-level global attention module (CGAT module). This transforms the fused feature map from an N×4C×(H / 4)×(W / 4) feature size to an N×8C×(H / 8)×(W / 8) feature size. The resulting 3D vector of the fused feature map is obtained, typically expressed in the form of [C, H, W], where C represents the number of feature channels, and H and W represent the spatial height and width of the fused feature map, respectively. This 3D vector is then divided into several independent vector groups along a specific dimension, for example, along the channel dimension C.

[0068] For each of the resulting vector groups, multi-head attention computation is performed independently. Specifically, each vector group is mapped to generate a query matrix, key matrix, and value matrix through a linear transformation. The relevance weights between the query, key, and value matrices within each vector group are calculated to extract the corresponding context-enhanced features for each group. The context-enhanced features of all vector groups are concatenated along the feature channel dimension to reconstruct and generate a context feature matrix rich in global context. The feature dimensions of the context feature matrix are adjusted using linear projection to output the enhanced feature map. The specific steps include the following: Feature optimization is performed on the specular highlight features in the fused feature map to obtain the enhanced feature map, including: The fused feature map is input into the global attention module in the specular restoration network. The global attention module is set after the local hybrid domain attention module in the specular restoration network and is used to carry out the feature processing of the fused feature map generated by fusing the frequency domain features and spatial domain features extracted by the local hybrid domain attention module. The three-dimensional vector of the fused feature map is obtained through the global attention module, and the three-dimensional vector is divided into several vector groups; For each vector group, perform multi-head attention computation to obtain the context-enhanced features corresponding to each vector group; The context enhancement features of all vector groups are concatenated to generate a context feature matrix of the fused feature map, and the enhanced feature map is generated based on the context feature matrix.

[0069] This embodiment divides the 3D vector into several independent vector groups along a specific dimension for parallel processing, reducing the computational complexity and memory consumption of the multi-head attention mechanism. Attention calculations are performed independently within each vector group, thereby enhancing contextual information from different dimensions. Finally, by concatenating and integrating the optimized results of all vector groups, the generated enhanced feature map retains the complete global context and is rich in multi-angle detail features, thus significantly improving the ability to discriminate complex specular highlight features and the accuracy and quality of subsequent image reconstruction.

[0070] In a specific embodiment, each partitioned vector group is mapped to generate a corresponding query matrix, key matrix, and value matrix through a linear transformation. These three matrices are then uniformly divided into multiple attention heads along the feature dimension. For each attention head obtained after segmentation, the dot product between the transpose of the query matrix and the key matrix is ​​first calculated within that attention head to obtain an initial dot product result. This result is then scaled and normalized using a scaling factor to obtain the attention weight matrix. The attention weight matrix is ​​used to characterize which locations in the feature subspace of the current enhanced feature map require focused attention and aggregation of feature information. Further, the attention weight matrix is ​​multiplied by the corresponding value matrix, and the weighted sum is obtained to get the output result of that attention head. The output results of all attention heads within the same vector group are then concatenated along the feature dimension of the channel to finally obtain the context enhancement feature corresponding to that vector group.

[0071] After calculating the context-enhanced features for all vector groups, they are concatenated along the feature dimensions used to divide the vector groups to obtain the global context-enhanced features. On the other hand, to preserve the original feature information before optimization and alleviate the gradient vanishing problem in deep networks, the concatenated global context-enhanced features are residually added to the 3D vector of the fused feature map. The result of this residual addition is normalized to stabilize the training process and accelerate convergence, ultimately yielding the context feature matrix. Finally, the context feature matrix is ​​processed through linear projection to generate the enhanced feature map.

[0072] Specifically, using the Cross-Transformer, frequency domain features are used as queries, and spatial domain features are used as keys and values. When a frequency domain feature is used as a query, it carries prior information about the image's global structure and frequency distribution. It then searches for and aggregates local textures and spatial details related to frequency patterns within the spatial domain features used as keys / values. In other words, it uses the global structure to guide and restore textures locally damaged by highlights. Specifically, by calculating the similarity (i.e., attention weights) between the query in the frequency domain and the keys in the spatial domain, it dynamically determines which spatial locations and how much information needs to be extracted from the spatial domain to supplement the frequency domain guidance. The specific steps include: The query matrix, key matrix, and value matrix of each vector group are generated through linear transformation, and the query matrix, key matrix, and value matrix are divided into multiple attention heads.

[0073] For each attention head, calculate the dot product of the query matrix and the key matrix to obtain the dot product result. Normalize the dot product result to obtain the attention weight matrix.

[0074] Multiply the attention weight matrix by the corresponding value matrix to obtain the output of each attention head. Concatenate the outputs of multiple attention heads to obtain the context-enhanced features of the vector group.

[0075] This embodiment maps the features of each vector group to multiple attention heads for parallel computation, allowing each attention head to independently focus on different contextual dependencies (such as texture, shape, or lighting), thus capturing more comprehensive complex feature interactions within the specular highlight region. Secondly, the extracted attention weight matrix is ​​fused with the corresponding value matrix, enhancing key features related to specular highlights and suppressing irrelevant or interfering information, thereby improving the model's semantic understanding and discrimination capabilities for specular highlights.

[0076] In a specific embodiment, this application introduces a hybrid domain cross-attention mechanism in the channel-level global attention module to coordinate the representation of spatial and frequency domain information, which are different but complementary, to achieve more accurate perception and reconstruction of specular interference regions. Specifically, spatial and frequency domain features are input simultaneously and further calculated through cross-attention. For example, frequency domain features are used as queries, and spatial domain features are used as keys and values, allowing the restoration process to actively search for and restore matching texture details in the spatial domain, guided by the global structural priors provided by the frequency domain. To further optimize the fusion effect, this embodiment sets a learnable fusion parameter. The fusion parameters, used as network weights, are given a neutral initial value (e.g., 0.5) during the initial training phase, indicating that the spatial and frequency domains are assigned equal weights. During training, the values ​​are adjusted based on the specific input image features and the degree of specular interference they contain. This improves the SHRNet network's ability to reconstruct and recognize the texture continuity in areas with specular highlight interference. The specific formula is as follows:

[0077] in, Represented as 1 / 8 of the feature map of the initial image (size: The projection of ) has dimensions of . , For the number of channels, This indicates the number of attention heads.

[0078] In a specific embodiment, the context enhancement features of all vector groups are concatenated to generate a context feature matrix of the fused feature map, and an enhanced feature map is generated based on the context feature matrix, including: The residual is obtained by adding the context-enhanced features to the 3D vector.

[0079] The residual results are normalized to generate a context feature matrix for the fused feature map, and an enhanced feature map is generated based on the context feature matrix.

[0080] This embodiment adds the context-enhanced features to the 3D vector using residual summation, effectively preserving the low-level details and original information in the input fused feature map. This ensures that key texture and structural features are not lost or overwritten while introducing global context. Normalizing the residual results stabilizes the training process and accelerates model convergence, improving training efficiency and stability.

[0081] Step S104: Reconstruct the image based on the enhanced feature map to obtain a repaired image with specular highlights removed.

[0082] Specifically, the enhanced feature map is upsampled to restore its spatial dimensions to the same size as the initial image, i.e., N×C×H×W, where N is the batch size, C is the number of feature channels, H is the height of the initial image, and W is the width of the initial image. Next, a convolutional layer with C input channels and 3 output channels projects the image size back from N×C×H×W to the original RGB image size. In other words, the dimensions of the enhanced feature map change from N×C×H×W to N×3×H×W, resulting in a restored image with specular highlights removed.

[0083] This embodiment converts the image size so that the final generated repaired image maintains the same spatial size and color channel format as the original input image to be processed. As a result, the repaired image can be used directly, realizing the reconstruction from deep features to usable pixel images.

[0084] This embodiment also provides an image highlight removal device, which is used to implement the above embodiments and preferred embodiments, and will not be repeated as already described. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0085] This embodiment provides a device for eliminating image highlights, such as... Figure 5 As shown, it includes: Image acquisition module 501 is used to acquire an initial image containing specular highlight features.

[0086] The feature fusion module 502 is used to extract the frequency domain features of the distribution pattern of specular highlight features and the spatial domain features of the spatial morphology of specular highlight features based on the initial image, and to fuse the frequency domain features and spatial domain features to obtain a fused feature map.

[0087] Image enhancement module 503 is used to optimize the specular highlight features in the fused feature map to obtain an enhanced feature map.

[0088] Image reconstruction module 504 is used to reconstruct the image based on the enhanced feature map to obtain a restored image with specular highlights removed.

[0089] Image acquisition module 501 includes: The image acquisition unit is used to acquire the image to be processed.

[0090] The feature extraction unit is used to extract features from the image to be processed, perform downsampling operations to obtain an initial feature map, extract features from the initial feature map, and perform downsampling operations again to obtain an initial image containing specular highlight features.

[0091] An image acquisition unit is used to input the image to be processed into a specular highlight restoration network. The specular highlight restoration network comprises at least two convolutional networks, including a first convolutional network and a second convolutional network. Each convolutional network includes a convolutional layer, a normalization layer, and an activation function. The unit extracts basic features of the image to be processed through the convolutional layers in the first convolutional network. The basic features are normalized through the normalization layers in the first convolutional network to obtain a first feature map. The first feature map is then non-linearly activated through the activation function in the first convolutional network to obtain a third feature map. The third feature map is downsampled to obtain an initial feature map. Specular-related features of the initial feature map are extracted through the convolutional layers in the second convolutional network. The specular-related features are normalized through the normalization layers in the second convolutional network to obtain a stable feature map. The stable feature map is then non-linearly activated through the activation function in the second convolutional network to obtain a deep specular feature map. The deep specular feature map is then downsampled to obtain the initial image.

[0092] The feature fusion module 502 is specifically used to input the initial image into the local mixing domain attention module in the specular highlight restoration network. The local mixing domain attention module is located after the convolutional network in the specular highlight restoration network and is used to perform feature processing on the initial image containing specular highlight features obtained through two downsampling operations. The local mixing domain attention module performs horizontal or vertical pixel shifting on the initial image to obtain a pixel-shifted initial image. A Fourier transform is performed on the pixel-shifted initial image to extract its frequency spectrum. Low-frequency and high-frequency structural features of the initial image are obtained based on the frequency spectrum. Based on the low-frequency and high-frequency structural features, the pixel-shifted initial image is divided into several local windows, and frequency domain features of the pixel-shifted initial image are extracted in each local window. The pixel-shifted initial image is converted into a three-dimensional feature image, and its spatial domain features are extracted.

[0093] The feature fusion module 502 is specifically used to determine the pixel shift parameters of the initial image, including the shift direction and / or shift distance. Based on the pixel shift parameters, the initial image is segmented to obtain a retained region and a region to be filled. The region to be filled is filled with pixel values ​​to generate a filled region. The retained region and the filled region are concatenated along the spatial dimensions of the initial image to obtain the pixel-shifted initial image.

[0094] The image enhancement module 503 is specifically used to input the fused feature map into the global attention module in the specular inpainting network. The global attention module is set after the local mixing domain attention module in the specular inpainting network and is used to carry out the feature processing of the fused feature map generated by fusing the frequency domain features and spatial domain features extracted by the local mixing domain attention module. The global attention module obtains the three-dimensional vector of the fused feature map and divides the three-dimensional vector into several vector groups. For each vector group, multi-head attention calculation is performed to obtain the context enhancement features corresponding to each vector group. The context enhancement features of all vector groups are concatenated to generate the context feature matrix of the fused feature map, and the enhanced feature map is generated based on the context feature matrix.

[0095] The image enhancement module 503 is specifically used to generate the query matrix, key matrix, and value matrix of each vector group through linear transformation, and to segment the query matrix, key matrix, and value matrix into multiple attention heads. For each attention head, the dot product of the query matrix and the key matrix is ​​calculated, and the dot product result is normalized to obtain the attention weight matrix. The attention weight matrix is ​​multiplied by the corresponding value matrix to obtain the output result of each attention head. The output results of multiple attention heads are concatenated to obtain the context enhancement features of the vector group.

[0096] The image enhancement module 503 is specifically used to add the context enhancement features and the three-dimensional vector by residual addition to obtain the residual result. The residual result is normalized to generate the context feature matrix of the fused feature map, and the enhanced feature map is generated based on the context feature matrix.

[0097] In this embodiment, the image highlight removal device is presented in the form of a functional unit. Here, a unit refers to an ASIC circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above-mentioned functions.

[0098] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0099] This invention also provides a computer device having the above-described features. Figure 5 The image highlight removal device shown.

[0100] Please see Figure 6 , Figure 6 This is a schematic diagram of the structure of a computer device provided in an optional embodiment of the present invention, such as... Figure 6As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 6 Take a processor 10 as an example.

[0101] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0102] The memory 20 stores instructions executable by at least one processor 10 to cause at least one processor 10 to perform the method shown in the above embodiments.

[0103] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device as shown by a landing page for an app. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, which can be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0104] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0105] The computer device also includes a communication interface 30 for communicating with other devices or communication networks.

[0106] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0107] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A method for eliminating highlights in an image, characterized in that, The method includes: Obtain an initial image containing specular highlight features; Based on the initial image, the frequency domain features of the distribution pattern of the specular highlight features and the spatial domain features of the spatial morphology of the specular highlight features are extracted, and the frequency domain features and the spatial domain features are fused to obtain a fused feature map. The specular highlight features in the fused feature map are optimized to obtain an enhanced feature map. Based on the enhanced feature map, the image is reconstructed to obtain a restored image with specular highlights removed.

2. The method according to claim 1, characterized in that, The process of acquiring an initial image containing specular highlight features includes: Obtain the image to be processed; Feature extraction is performed on the image to be processed, and a downsampling operation is performed to obtain an initial feature map. Feature extraction is then performed on the initial feature map, and a downsampling operation is performed again to obtain an initial image containing specular highlight features.

3. The method according to claim 2, characterized in that, The process of extracting features from the image to be processed, performing a downsampling operation to obtain an initial feature map, extracting features from the initial feature map, and performing a downsampling operation again to obtain an initial image containing specular highlight features includes: The image to be processed is input into a specular highlight restoration network, wherein the specular highlight restoration network comprises at least two convolutional networks, the convolutional networks comprising a first convolutional network and a second convolutional network, and each convolutional network comprising a convolutional layer, a normalization layer and an activation function; The basic features of the image to be processed are extracted through the convolutional layers in the first convolutional network; The basic features are normalized by the normalization layer in the first convolutional network to obtain a first feature map, and the first feature map is non-linearly activated by the activation function in the first convolutional network to obtain a third feature map. The third feature map is downsampled to obtain the initial feature map; Highlight correlation features of the initial feature map are extracted through the convolutional layers in the second convolutional network; The hyperspectral correlation features are normalized by the normalization layer in the second convolutional network to obtain a stable feature map; The stable feature map is nonlinearly activated by the activation function of the second convolutional network to obtain a deep specular feature map; The deep specular feature map is downsampled to obtain the initial image.

4. The method according to claim 1, characterized in that, The extraction of frequency domain features of the distribution pattern of the specular highlight features and spatial domain features of the spatial morphology of the specular highlight features based on the initial image includes: The initial image is input into the local blending domain attention module in the specular highlight restoration network. The local blending domain attention module is set after the convolutional network in the specular highlight restoration network and is used to carry out feature processing of the initial image containing specular highlight features obtained by two downsampling operations. The initial image is obtained by performing pixel shifting in the horizontal or vertical direction using the local hybrid domain attention module; Perform a Fourier transform on the pixel-shifted initial image to extract the frequency spectrum of the initial image; The low-frequency and high-frequency structural features of the initial image are obtained based on the frequency spectrum. Based on the low-frequency structural features and the high-frequency structural features, the initial image after pixel shifting is divided into several local windows, and the frequency domain features of the initial image after pixel shifting are extracted in each local window; The initial image after pixel shifting is converted into a three-dimensional feature image, and the spatial domain features of the three-dimensional feature image are extracted.

5. The method according to claim 4, characterized in that, The step of performing a horizontal or vertical pixel shift on the initial image to obtain the pixel-shifted initial image includes: Determine the pixel shift parameters of the initial image, wherein the pixel shift parameters include shift direction and / or shift distance; The initial image is segmented based on the pixel shift parameters of the initial image to obtain the retained region and the region to be filled; The area to be filled is filled with pixel values ​​to generate a filled area; The retained region and the filled region are stitched together along the spatial dimensions of the initial image to obtain the pixel-shifted initial image.

6. The method according to claim 1, characterized in that, The specular highlight features in the fused feature map are optimized to obtain an enhanced feature map, including: The fused feature map is input into the global attention module in the specular restoration network. The global attention module is set after the local hybrid domain attention module in the specular restoration network and is used to carry out feature processing of the fused feature map generated by fusing the frequency domain features and spatial domain features extracted by the local hybrid domain attention module. The three-dimensional vector of the fused feature map is obtained through the global attention module, and the three-dimensional vector is divided into several vector groups; For each vector group, multi-head attention computation is performed to obtain the context enhancement features corresponding to each vector group; The context enhancement features of all vector groups are concatenated to generate the context feature matrix of the fused feature map, and the enhanced feature map is generated based on the context feature matrix.

7. The method according to claim 6, characterized in that, For each of the vector groups, multi-head attention computation is performed to obtain the context-enhanced features corresponding to each vector group, including: A query matrix, key matrix, and value matrix for each vector group are generated through linear transformation, and the query matrix, key matrix, and value matrix are divided into multiple attention heads. For each attention head, the dot product of the query matrix and the key matrix is ​​calculated to obtain the dot product result. The dot product result is then normalized to obtain the attention weight matrix. Multiply the attention weight matrix by the corresponding value matrix to obtain the output of each attention head. Concatenate the outputs of multiple attention heads to obtain the context-enhanced features of the vector group.

8. The method according to claim 6, characterized in that, The process of concatenating the context enhancement features of all vector groups to generate the context feature matrix of the fused feature map, and generating the enhanced feature map based on the context feature matrix, includes: The residuals of the context-enhanced features and the three-dimensional vector are added together to obtain the residual result. The residual results are normalized to generate the context feature matrix of the fused feature map, and an enhanced feature map is generated based on the context feature matrix.

9. An image highlight removal device, characterized in that, The device includes: The image acquisition module is used to acquire an initial image containing specular highlight features; The feature fusion module is used to extract the frequency domain features of the distribution pattern of the specular highlight features and the spatial domain features of the spatial morphology of the specular highlight features based on the initial image, and fuse the frequency domain features and the spatial domain features to obtain a fused feature map. The image enhancement module is used to optimize the specular highlight features in the fused feature map to obtain an enhanced feature map; The image reconstruction module is used to reconstruct the image based on the enhanced feature map to obtain a restored image with specular highlights removed.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing a computer to perform the method of any one of claims 1 to 8.

Images