Image processing method and device, electronic equipment and storage medium

Abstract

The invention relates to an image processing method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a to-be-processed feature vector of a to-be-processed image; processing the to-be-processed feature vector based on a pre-trained first deep learning model to obtain an initial non-highlight feature vector, and processing the to-be-processed feature vector and the initial non-highlight feature vector to obtain a highlight feature vector; connecting the to-be-processed feature vector, the initial non-highlight feature vector and the highlight feature vector to obtain a target feature vector; inputting the target feature vector into a pre-trained second deep learning model to obtain a middle non-highlight feature vector; and processing the intermediate non-highlight feature vector and the to-be-processed feature vector to obtain a target non-highlight feature vector, and obtaining a target non-highlight image of the target non-highlight feature vector. According to the technical scheme, the image highlight removal efficiency and effect are improved through a two-stage progressive highlight removal mechanism.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to an image processing method, apparatus, electronic device and storage medium. Background Technology

[0002] Typically, in scenarios such as intelligent transportation, autonomous driving, and underground parking management, image recognition and perception systems play a crucial role in accurately perceiving the ground environment.

[0003] However, due to their unique lighting conditions and surface materials, underground parking lots often exhibit specular reflection, resulting in large areas of "spectral highlights" in images. These highlights severely interfere with the texture information and semantic content of the image, affecting not only the performance of downstream tasks such as object detection, path planning, and parking space recognition, but also potentially leading to system decision-making errors.

[0004] In related technologies, traditional image processing-based methods are typically used for specular highlight removal, such as highlight detection combined with image inpainting (e.g., based on Poisson reconstruction or edge-guided filling). These methods rely on explicit edge information and brightness differences, and perform poorly when there is severe texture loss or blurred edges, lacking the ability to model complex semantic scenes. Summary of the Invention

[0005] In order to solve the above-mentioned technical problems, or at least partially solve the above-mentioned technical problems, at least one embodiment of the present disclosure provides an image processing method, apparatus, electronic device and storage medium.

[0006] In a first aspect, this disclosure provides an image processing method, comprising: acquiring an image to be processed and acquiring a feature vector to be processed from the image to be processed; processing the feature vector to be processed based on a pre-trained first deep learning model to obtain an initial no-highlight feature vector, and subtracting the feature vector to be processed from the initial no-highlight feature vector element-wise to obtain a highlight feature vector; concatenating the feature vector to be processed, the initial no-highlight feature vector, and the highlight feature vector to obtain a target feature vector; inputting the target feature vector into the pre-trained second deep learning model to obtain an intermediate no-highlight feature vector; wherein the pre-trained second deep learning model and the pre-trained first deep learning model share model parameters; adding the intermediate no-highlight feature vector and the feature vector to be processed element-wise to obtain a target no-highlight feature vector, and acquiring a target no-highlight image based on the target no-highlight feature vector.

[0007] Secondly, this disclosure provides an image processing apparatus, comprising: a first acquisition module, configured to acquire an image to be processed and acquire a feature vector to be processed from the image to be processed; a first processing module, configured to process the feature vector to be processed based on a pre-trained first deep learning model to obtain an initial no-highlight feature vector; a subtraction module, configured to subtract the feature vector to be processed from the initial no-highlight feature vector element-wise to obtain a highlight feature vector; a connection module, configured to connect the feature vector to be processed, the initial no-highlight feature vector, and the highlight feature vector to obtain a target feature vector; a second processing module, configured to input the target feature vector into the pre-trained second deep learning model to obtain an intermediate no-highlight feature vector; wherein the pre-trained second deep learning model and the pre-trained first deep learning model share model parameters; an addition module, configured to add the intermediate no-highlight feature vector to the feature vector to be processed element-wise to obtain a target no-highlight feature vector; and a second acquisition module, configured to acquire a target no-highlight image of the target no-highlight feature vector.

[0008] Thirdly, this disclosure provides an electronic device, including: a processor and a memory;

[0009] The processor executes any of the image processing methods provided in the embodiments of this disclosure by calling the programs or instructions stored in the memory.

[0010] Fourthly, this disclosure provides a computer-readable storage medium storing a program or instructions that cause a computer to perform any of the image processing methods provided in the embodiments of this disclosure.

[0011] Fifthly, this disclosure provides a computer program product for executing any of the image processing methods described in the embodiments of this disclosure.

[0012] The technical solution provided in this disclosure has at least the following advantages compared with the prior art:

[0013] In this embodiment, an image to be processed is acquired, and a feature vector to be processed from the image to be processed is acquired. The feature vector to be processed is processed based on a pre-trained first deep learning model to obtain an initial no-highlight feature vector. The feature vector to be processed and the initial no-highlight feature vector are then subtracted element-wise to obtain a highlight feature vector. The feature vector to be processed, the initial no-highlight feature vector, and the highlight feature vector are concatenated to obtain a target feature vector. The target feature vector is input into a pre-trained second deep learning model to obtain an intermediate no-highlight feature vector. The pre-trained second deep learning model and the pre-trained first deep learning model share model parameters. The intermediate no-highlight feature vector and the feature vector to be processed are then added element-wise to obtain a target no-highlight feature vector, and a target no-highlight image is acquired based on the target no-highlight feature vector. By adopting the above technical solution, two pre-trained deep learning models are used to perform coarse separation of specular highlights and collaborative optimization of highlight image inpainting in a two-stage progressive highlight removal method, thereby improving the efficiency and effect of image highlight removal. At the same time, the two pre-trained deep learning models share model parameters, which enhances the model's edge deployment capability and reduces the model's memory cost. Attached Figure Description

[0014] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0015] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 A schematic flowchart of an image processing method provided in an exemplary embodiment of this disclosure;

[0017] Figure 2 A schematic flowchart of an image processing method provided for another exemplary embodiment of this disclosure;

[0018] Figure 3 Example diagram of an image processing method provided as an exemplary embodiment of this disclosure;

[0019] Figure 4 Example diagram of a model network architecture provided for an exemplary embodiment of this disclosure;

[0020] Figure 5 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of the present disclosure. Detailed Implementation

[0021] To better understand the above-described objectives, features, and advantages of this disclosure, the present disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. It is understood that the described embodiments are only some, not all, of the embodiments of this disclosure. The specific embodiments described herein are merely for explaining this disclosure and not for limiting it. Unless otherwise specified, the embodiments of this disclosure and the features within them can be combined with each other. All other embodiments obtained by those skilled in the art based on the described embodiments of this disclosure are within the scope of protection of this disclosure.

[0022] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0023] Existing methods for specular highlight removal rely on explicit edge information and brightness differences by combining highlight detection with image inpainting. However, these methods perform poorly when there is severe texture loss or blurred edges, and lack the ability to model complex semantic scenes. Alternatively, deep learning methods based on CNNs (Convolutional Neural Networks) can be used for highlight inpainting. While these methods can learn some semantic context, their ability to model long-distance dependencies and complex spatial relationships is limited by the local receptive field of convolutions. They struggle to handle situations with large highlight occlusion areas or complex semantic structures. Multi-task learning methods, which simultaneously perform highlight detection and removal to improve overall consistency, are also available, but these models are typically structurally complex, training is unstable, and they lack a systematic design for learning phased features.

[0024] Therefore, existing specular removal methods have obvious limitations. Traditional image processing methods are insufficient for modeling complex semantics and structures. CNN methods have limited receptive fields and are difficult to handle large-area occlusion and long-distance dependencies. Multi-task learning methods (spectrum detection and specular restoration) are difficult to train, have poor stability, lack clear stage-based collaborative mechanisms, and have the problem of difficult deployment on edge devices. That is, when CNN models are deployed on edge devices, they cannot perform long-distance modeling and therefore perform poorly.

[0025] To address the aforementioned problems, this disclosure provides an image processing method. The method involves acquiring an image to be processed and obtaining its feature vector; processing the feature vector based on a pre-trained first deep learning model to obtain an initial no-highlight feature vector; subtracting the initial no-highlight feature vector from the feature vector to be processed element-wise to obtain a highlight feature vector; concatenating the feature vector to be processed, the initial no-highlight feature vector, and the highlight feature vector to obtain a target feature vector; inputting the target feature vector into a pre-trained second deep learning model to obtain an intermediate no-highlight feature vector; wherein the pre-trained second deep learning model and the pre-trained first deep learning model share model parameters; and adding the intermediate no-highlight feature vector and the feature vector to be processed element-wise to obtain the target no-highlight feature vector, and acquiring the target no-highlight image based on the target no-highlight feature vector. By adopting the above technical solution, two pre-trained deep learning models are used to perform coarse separation of specular highlights and collaborative optimization of highlight image inpainting in a two-stage progressive highlight removal method, thereby improving the efficiency and effect of image highlight removal. At the same time, the two pre-trained deep learning models share model parameters, which enhances the model's edge deployment capability and reduces the model's memory cost.

[0026] The specific embodiments of the image processing method, apparatus, electronic device, and storage medium of this disclosure are explained in detail below with reference to the accompanying drawings.

[0027] Figure 1 This is a schematic flowchart of an image processing method provided in an exemplary embodiment of the present disclosure. The method can be executed by an image processing apparatus provided in the embodiments of the present disclosure. The image processing apparatus can be implemented in software and / or hardware and is generally integrated into an electronic device.

[0028] like Figure 1 As shown, the image processing method may include the following steps:

[0029] Step 101: Obtain the image to be processed and obtain the feature vector of the image to be processed.

[0030] The image to be processed is an image with highlights; highlights refer to the white areas in the image to be processed that reach the maximum brightness value, and the color level value is usually constant at 255.

[0031] In this embodiment, there are many ways to acquire the image to be processed. As an example, the image acquisition device based on the vehicle acquires the image to be processed in a preset area. That is, some areas, such as the lighting conditions and ground material of the parking lot, can be preset to acquire images containing highlight areas, so that the images acquired in the preset area are used as the images to be processed.

[0032] As another example, the driving conditions of the road can be determined based on ground information such as whether there is water accumulation or weather forecast during the vehicle's journey, such as whether it is raining or snowing. This allows for the selection of one or more target driving routes. When the vehicle is traveling on the target driving route, the images captured by the vehicle's image acquisition equipment are used as images to be processed, further improving the accuracy of vehicle control.

[0033] In this embodiment of the disclosure, after acquiring the image to be processed, the image to be processed can be processed to obtain the feature vector to be processed. Specifically, the number of channels, height and width of the image to be processed are acquired as the feature vector to be processed.

[0034] Step 102: Based on the pre-trained first deep learning model, process the feature vector to be processed to obtain the initial feature vector without highlights, and subtract the feature vector to be processed from the initial feature vector without highlights element by element to obtain the highlight feature vector.

[0035] In this embodiment of the disclosure, after obtaining the feature vector to be processed, the first deep learning model is pre-trained to perform no-highlight feature extraction processing on the feature vector to be processed, so as to obtain the initial no-highlight feature vector, that is, to obtain the no-highlight image corresponding to the image to be processed.

[0036] It should be noted that if the number of channels in the feature vector to be processed is 3, it can be converted to 1 through convolution operation before inputting the feature vector to be processed into the first deep learning module of the pre-training, which can further improve the efficiency and effect of subsequent processing.

[0037] In this embodiment of the disclosure, a pre-trained first deep learning model is obtained. The pre-trained first deep learning model can be selected and set as needed. As an example, image samples and corresponding feature vector samples are obtained. The image samples include a no-highlight label. The feature vector samples are input into a pre-constructed deep learning neural network for processing to obtain a predicted no-highlight feature vector, and the predicted no-highlight image corresponding to the predicted no-highlight feature vector is obtained. The network parameters of the pre-constructed deep learning neural network are adjusted based on the loss value between the image samples and the predicted no-highlight image and a preset loss threshold until the loss value is less than or equal to the preset loss threshold, thus obtaining the pre-trained first deep learning model.

[0038] In this embodiment of the disclosure, after obtaining the pre-trained first deep learning model, the feature vector to be processed is input into the pre-trained first deep learning model for feature extraction to obtain an initial no-highlight feature vector. The feature vector to be processed and the initial no-highlight feature vector are subtracted element by element to obtain the highlight feature vector.

[0039] Element-by-element subtraction refers to performing independent operations (subtraction) on corresponding elements in the feature vector to be processed and the initial non-highlight feature vector to obtain the highlight feature vector. In other words, it is to obtain the highlight image region from the image to be processed.

[0040] Step 103: Connect the feature vector to be processed, the initial non-highlight feature vector, and the highlight feature vector to obtain the target feature vector.

[0041] In this process, after obtaining the highlight feature vector, the feature vector to be processed, the initial non-highlight feature vector, and the highlight feature vector are concatenated to obtain the target feature vector. Specifically, the concat function (used to concatenate strings or arrays) can be used to merge multiple inputs into a single output, that is, to merge the feature vector to be processed, the initial non-highlight feature vector, and the highlight feature vector into a single target feature vector.

[0042] In this embodiment of the disclosure, after obtaining the initial no-highlight feature vector and the highlight feature vector, the feature vector to be processed, the initial no-highlight feature vector, and the highlight feature vector can be connected to obtain a target feature vector.

[0043] Step 104: Input the target feature vector into the pre-trained second deep learning model to obtain the feature vector without highlights in the middle; wherein, the pre-trained second deep learning model and the pre-trained first deep learning model share model parameters.

[0044] In this embodiment of the disclosure, after obtaining the target feature vector, the target feature vector is processed by pre-training a second deep learning model to extract the non-highlight features, thereby obtaining the non-highlight features in the middle. In other words, the non-highlight image is obtained from the image corresponding to the target feature vector.

[0045] It should be noted that if the number of channels in the target feature vector is 3, it can be converted to 1 through convolution operation before inputting the target feature vector into the pre-trained second deep learning module, which can further improve the efficiency and effect of subsequent processing.

[0046] In this embodiment of the disclosure, a pre-trained second deep learning model is obtained. The pre-trained second deep learning model can be selected and set as needed. As an example, image samples and corresponding feature vector samples are obtained. The image samples include a no-highlight label. The feature vector samples are input into a pre-constructed deep learning neural network for processing to obtain a predicted no-highlight feature vector, and the predicted no-highlight image corresponding to the predicted no-highlight feature vector is obtained. The network parameters of the pre-constructed deep learning neural network are adjusted based on the loss value between the image samples and the predicted no-highlight image and a preset loss threshold until the loss value is less than or equal to the preset loss threshold, thus obtaining the pre-trained second deep learning model.

[0047] Among them, the pre-trained second deep learning model and the pre-trained first deep learning model share model parameters, which reduces the memory cost of the model and improves the edge deployment capability of the model, thereby improving the efficiency and effect of image processing.

[0048] In this embodiment of the disclosure, after obtaining the pre-trained second deep learning model, the target feature vector is input into the pre-trained second deep learning model for feature extraction to obtain the feature vector without highlights in the middle.

[0049] The image processing method of this disclosure includes: acquiring an image to be processed and acquiring a feature vector to be processed from the image to be processed; processing the feature vector to be processed based on a pre-trained first deep learning model to obtain an initial no-highlight feature vector; subtracting the feature vector to be processed from the initial no-highlight feature vector element-wise to obtain a highlight feature vector; concatenating the feature vector to be processed, the initial no-highlight feature vector, and the highlight feature vector to obtain a target feature vector; inputting the target feature vector into a pre-trained second deep learning model to obtain an intermediate no-highlight feature vector; wherein the pre-trained second deep learning model and the pre-trained first deep learning model share model parameters; adding the intermediate no-highlight feature vector to the feature vector to be processed element-wise to obtain a target no-highlight feature vector; and acquiring a target no-highlight image with the target no-highlight feature vector. By adopting the above technical solution, two pre-trained deep learning models are used to perform coarse separation of specular highlights and collaborative optimization of highlight image inpainting in a two-stage progressive highlight removal method, thereby improving the efficiency and effect of image highlight removal. At the same time, the two pre-trained deep learning models share model parameters, which enhances the model's edge deployment capability and reduces the model's memory cost.

[0050] In one optional embodiment of this disclosure, before processing based on a pre-trained first deep learning model and a pre-trained second deep learning model, it is necessary to pre-train a deep learning model. The training methods for the first and second deep learning models are the same. Taking the first deep learning model as an example, image samples and corresponding feature vector samples are obtained; wherein, the image samples include a no-highlight label; the feature vector samples are input into a pre-constructed deep learning neural network for processing to obtain a predicted no-highlight feature vector, and the predicted no-highlight image corresponding to the predicted no-highlight feature vector is obtained; the network parameters of the pre-constructed deep learning neural network are adjusted based on the loss value between the image samples and the predicted no-highlight image and a preset loss threshold until the loss value is less than or equal to the preset loss threshold, thereby obtaining the pre-trained first deep learning model.

[0051] Specifically, image samples including highlights can be obtained, and corresponding no-highlight labels can be obtained for the image samples. The no-highlight labels represent the real no-highlight images. There are usually multiple image samples. As an example, the real no-highlight images are processed to obtain image samples including highlights. The real no-highlight images are used as the no-highlight labels for the image samples.

[0052] Specifically, after acquiring image samples, feature vector samples corresponding to the image samples are acquired, and the feature vector samples are input into a pre-constructed deep learning neural network for processing to obtain the predicted no-highlight feature vector, and the predicted no-highlight image corresponding to the predicted no-highlight feature vector is obtained; where the predicted no-highlight image represents the image after the predicted image sample has no highlights.

[0053] The pre-constructed deep learning neural network includes multiple frequency domain processing deep learning networks. Feature vector samples are input into the pre-constructed deep learning neural network for processing to obtain a predicted feature vector without highlights. This process includes downsampling the feature vector samples at least once using at least one first frequency domain processing deep learning network to obtain a first image feature vector sample; processing the first image feature vector sample using a second frequency domain processing deep learning network to obtain a second image feature vector sample; and then element-wise adding the first and second image feature vector samples and inputting them into at least one third frequency domain processing deep learning network for upsampling to obtain the predicted feature vector without highlights. The number of first and third frequency domain processing deep learning networks is the same.

[0054] In this embodiment, the number of deep learning networks processed in the first frequency domain is the same as the number of deep learning networks processed in the third frequency domain. That is, the number of downsampling times and the number of upsampling times are the same. The number of downsampling times and the number of upsampling times can be selected and set according to the actual application scenario. This embodiment does not impose any restrictions.

[0055] For example, taking a scenario where both downsampling and upsampling are performed three times (i.e., there are three deep learning networks for first and third frequency domain processing), the first frequency domain processing deep learning network downsamples the feature vector sample to obtain the first downsampled feature vector sample. Another first frequency domain processing deep learning network then downsamples the first downsampled feature vector sample to obtain the second downsampled feature vector sample. Yet another first frequency domain processing deep learning network processes the second downsampled feature vector sample to obtain the third downsampled feature vector sample, which serves as the first feature vector sample. Finally, the second frequency domain processing deep learning network processes the first feature vector sample... The first feature vector sample is processed to obtain the second feature vector sample. The first and second feature vector samples are then added element-wise and input into a third frequency domain processing deep learning network for upsampling to obtain the first upsampled feature vector sample. The second downsampled feature vector sample and the first upsampled feature vector sample are then added element-wise and input into another third frequency domain processing deep learning network for upsampling to obtain the second upsampled feature vector sample. The first downsampled feature vector sample and the second upsampled feature vector sample are then added element-wise and input into yet another third frequency domain processing deep learning network for upsampling to obtain the predicted feature vector without specular highlights.

[0056] The frequency domain processing deep learning network includes a frequency domain transformation module, a deep learning coding module, and a cross-scale feature fusion module. Taking the first frequency domain processing deep learning network as an example, the feature vector samples are processed by the frequency domain transformation module to obtain frequency domain feature vector samples. The frequency domain feature vector samples and the feature vector samples are input into the deep learning coding module for processing to obtain multiple coded feature vector samples. The multiple coded feature vector samples are input into the cross-scale feature fusion module for processing to obtain the first feature vector sample.

[0057] In this embodiment of the disclosure, after obtaining the predicted image without highlights corresponding to the predicted feature vector without highlights, the network parameters of the pre-constructed deep learning neural network are adjusted based on the loss value between the no-highlight label and the predicted image without highlights and a preset loss threshold until the loss value is less than or equal to the preset loss threshold, thereby obtaining a pre-trained first deep learning model.

[0058] Specifically, there are many ways to calculate the loss value between an image sample and the predicted image without highlights. In some embodiments, the absolute error value between the no-highlight label corresponding to the image sample and the predicted image without highlights is directly calculated as the loss value. In other embodiments, the tone difference value between the no-highlight region of the image sample and the predicted image without highlights is obtained, the absolute error value between the no-highlight label and the predicted image without highlights is calculated, and the loss value is determined based on the preset tone coefficient, tone difference value, and absolute error value.

[0059] It is understandable that the specular highlight-free images in the training data are usually darker than the input image samples. This bias stems from the shortcomings of current specular highlight removal dataset creation methods (usually based on cross-polarizer technology, multiple light sources, and dichroic reflection model principles). Removing a light source at a certain angle will darken the overall environment. Therefore, the actual specular highlight-free images created are darker than the ideal specular highlight-free images. This will result in the problem of missing texture information and global brightness being too dark, even if the model has fully learned the mapping relationship from the original specular highlight image to the specular highlight-free image.

[0060] In this embodiment of the disclosure, in order to improve the image processing effect, only the loss value of the highlight region is considered when calculating the loss value during the training process. That is, the tonal difference value between the image sample and the predicted image without highlights is obtained, the absolute error value between the image without highlights label and the predicted image without highlights is calculated, and the loss value is determined based on the preset tonal coefficient, the tonal difference value and the absolute error value.

[0061] In some embodiments, obtaining the non-highlight region of an image sample and the hue difference value of the predicted non-highlight image includes: determining the non-highlight region of the image sample based on a preset maximum inter-class variance method, calculating the intensity value of the predicted non-highlight image in the HSV color space based on a preset hue transformation function, and calculating the hue difference value based on the intensity value of the predicted non-highlight image and the intensity value of the non-highlight region.

[0062] Specifically, the image sample Image and the predicted non-highlight image Output are thresholded using the Otsu's method to divide the image sample into a highlight region Hregion and a non-highlight region FHregion. Among them, the non-highlight region FHregion better reflects the true tone and is also the key target area for tone correction.

[0063] Specifically, a linear transformation is performed on the thinned image in the HSV (Hue, Saturation, Value) color space. A hue transformation function T is defined to minimize the hue difference E between the image without highlights and the input image in the FHregion region. The calculation is shown in formula (1).

[0064]

[0065] Where FHregion represents all pixels in the image without highlight areas, and T(·) is the modulation transform function, as shown in formula (2).

[0066] T(p) = M*(p) h ,p s ,p v ,1) T (2).

[0067] Where, p = (p h ,p s ,p v (i.e., HSV three-channel values) are the intensity values ​​of the Output in the HSV color space; This is the tone correction vector, containing all the parameters required for the transformation; * indicates vector multiplication; (·) T This is the transpose of a vector.

[0068] Therefore, the HSV color space can more clearly separate luminance and color information. Highlights mainly affect saturation and luminance (S and V channels). Thus, modeling in the HSV color space can more effectively control and correct hue deviations, and improve the generalization and robustness of the model.

[0069] Specifically, the loss value is determined based on the preset hue coefficient, hue difference value, and absolute error value, as shown in formula (3).

[0070]

[0071] Where D is the actual image without highlights, i.e., the image without highlights, Output is the estimated value of the image without highlights, i.e., the predicted image without highlights, α is the preset tone coefficient, and E is the tone difference value.

[0072] Based on the description of the foregoing embodiments, existing methods suffer from problems such as weak generalization, insufficient global modeling capabilities, and distortion in repairing structures. This disclosure proposes an image processing method that achieves global feature modeling by introducing a deep learning model composed of a frequency domain processing deep learning network. A two-stage progressive specular highlight removal mechanism is designed to complete the coarse separation of specular highlights and the collaborative optimization of specular image inpainting, effectively solving the comprehensive challenges of spatial modeling, semantic understanding, and structural reconstruction in specular highlight removal. At the same time, in order to improve the edge deployment capability of the model, a model parameter (weight) sharing mechanism is used to reduce the model's memory cost and improve the efficiency and effect of image specular highlight removal.

[0073] Figure 2 This is a flowchart illustrating an image processing method provided in another exemplary embodiment of the present disclosure. The method can be executed by an image processing apparatus provided in the embodiments of the present disclosure. The image processing apparatus can be implemented in software and / or hardware and is generally integrated into an electronic device.

[0074] like Figure 2 As shown, the image processing method may include the following steps:

[0075] Step 201: The vehicle-based image acquisition device acquires the image to be processed in a preset area and obtains the feature vector to be processed of the image.

[0076] Step 202: Perform at least one downsampling process on the feature vector to be processed based on at least one first frequency domain processing deep learning network to obtain the first image feature vector.

[0077] Step 203: Process the first image feature vector based on the second frequency domain processing deep learning network to obtain the second image feature vector.

[0078] Step 204: The first image feature vector and the second image feature vector are added element by element and then input into at least one third frequency domain processing deep learning network for upsampling to obtain an initial feature vector without highlights; wherein, the number of the first frequency domain processing deep learning network and the third frequency domain processing deep learning network are the same.

[0079] In this embodiment, the number of deep learning networks processed in the first frequency domain is the same as the number of deep learning networks processed in the third frequency domain. That is, the number of downsampling times and the number of upsampling times are the same. The number of downsampling times and the number of upsampling times can be selected and set according to the actual application scenario. This embodiment does not impose any restrictions.

[0080] For example, taking three downsampling and three upsampling operations as an example, that is, there are three deep learning networks for both the first and third frequency domain processing. The first frequency domain processing deep learning network performs downsampling on the feature vector to be processed, obtaining the first downsampled image feature vector. Another first frequency domain processing deep learning network then performs downsampling on the first downsampled image feature vector, obtaining the second downsampled image feature vector. Yet another first frequency domain processing deep learning network processes the second downsampled image feature vector, obtaining the third downsampled image feature vector, which is then used as the first image feature vector. Finally, the second frequency domain processing deep learning network processes the first downsampled image feature vector... The image feature vector is processed to obtain the second image feature vector. The first and second image feature vectors are then added element-wise and input into a third frequency domain processing deep learning network for upsampling to obtain the first upsampled image feature vector. The second downsampled image feature vector and the first upsampled image feature vector are then added element-wise and input into another third frequency domain processing deep learning network for upsampling to obtain the second upsampled image feature vector. The first downsampled image feature vector and the second upsampled image feature vector are then added element-wise and input into yet another third frequency domain processing deep learning network for upsampling to obtain the predicted feature vector without highlights.

[0081] In this embodiment of the disclosure, the frequency domain processing deep learning network includes a frequency domain transformation module, a deep learning coding module, and a cross-scale feature fusion module. The network performs downsampling processing on the feature vector to be processed to obtain a first image feature vector, including: processing the feature vector to be processed based on the frequency domain transformation module to obtain a frequency domain feature vector; inputting the frequency domain feature vector and the feature vector to be processed into the deep learning coding module for processing to obtain multiple coded feature vectors; and inputting the multiple coded feature vectors into the cross-scale feature fusion module for processing to obtain the first image feature vector.

[0082] It should be noted that the network structure of each frequency domain processing deep learning network is the same. The process of inputting any feature vector into the frequency domain processing deep learning network is the same as the process of processing the feature vector to be processed based on the frequency domain transformation module to obtain the frequency domain feature vector.

[0083] Step 205: Connect the feature vector to be processed, the initial feature vector without highlights, and the feature vector with highlights to obtain the target feature vector.

[0084] Step 206: Input the target feature vector into the pre-trained second deep learning model to obtain the feature vector without highlights in the middle; wherein, the pre-trained second deep learning model and the pre-trained first deep learning model share model parameters.

[0085] It should be noted that the model structure and model parameters of the pre-trained second deep learning model are the same as those of the pre-trained first deep learning model. In other words, step 206 can be achieved through the aforementioned steps 202 to 205.

[0086] Step 207: Add the intermediate no-highlight feature vector and the feature vector to be processed element by element to obtain the target no-highlight feature vector, and obtain the target no-highlight image of the target no-highlight feature vector.

[0087] In this embodiment of the disclosure, it is necessary to remove highlights from the image to be processed. The image processing method of this disclosure is as follows: Figure 3 As shown, step 1 involves the coarse separation of the specular residual and the non-spectral image, and step 2 involves the refinement mechanism for the non-spectral image. Specifically, given an input image (e.g., a specular image of a mirror surface in an underground parking lot), the corresponding feature vector is obtained with dimensions N×C×H×W (N is the batch size, C is the number of input channels, H is the image height, and W is the image width). Then, the first deep learning model, Transformer1, is pre-trained. Figure 3The encoder E1 and decoder D1 shown decompose the image into a non-highlight image D1 (an image without highlight areas, represented by a feature vector of size N×C×H×W) and a highlight-residual image S1 (referring only to the highlight parts of the image, represented by a feature vector of size N×C×H×W). The highlight-residual image S1 is obtained by subtracting the non-highlight image D1 from the image element by element. Both the non-highlight image D1 and the highlight-residual image S1 have dimensions of N×C×H×W.

[0088] Further refinement of this coarse result is needed to mitigate visual artifacts, such as color distortion. Specifically, this involves: first, performing a concat operation on the original input Image, the image without highlights D1, and the image with residual highlights S1, resulting in a dimension of N×3C×H×W; then, obtaining a feature F1 with the same scale as the Image through a 1×1 convolution Conv and a 3×3 depthwise separable convolution Dwconv, and inputting F1 into the pre-trained second deep learning model Transformer2. Figure 3 The encoder E2 and decoder D2 shown in the figure are used to obtain a refined specular-free image D2 (represented by a feature vector of size N×C×H×W as shown in the figure). Finally, the specular-free image D2 is added element-wise to the original input image Image through a residual connection, and then passed through a mapping module of 1×1 convolution and 3×3 depthwise separable convolution to obtain the output real specular-free image Output. In order to reduce the model parameters and improve the network's deployment capability on edge devices, Transformer1 and Transformer2 share weights, that is, share model parameters.

[0089] Specifically, such as Figure 3 As shown, both stages have an encoder-decoder structure. Transformer1 is responsible for transforming the input image Image into a specular-free image D1 and a specular-residual image S1. Transformer2 fine-tunes F1 to obtain a refined specular-free image D2. Transformer1 and Transformer2 need to share weights, therefore their model structures are identical, as shown in the specific structure below. Figure 4 As shown.

[0090] Specifically, in this architecture, the encoder-decoder structure runs through the entire two-stage processing flow, undertaking the tasks of feature extraction, separation, and reconstruction. Each encoder-decoder module adopts a U-Net-style symmetrical design, extracting feature semantics at different scales through layer-by-layer downsampling, and gradually restoring spatial resolution in the upsampling stage to generate the corresponding image results.

[0091] Specifically, in the first stage, the encoder-decoder (E1-D1) obtains a very coarse, specular-free image D1. In the second stage, the encoder-decoder (E2-D2) further refines the image and corrects its color. In the third stage, the coarse specular-free image, the specular residue image, and the original input image are combined to further refine and improve the image's naturalness and detail fidelity, thereby generating the final high-quality specular-free image. This optimizes image quality by leveraging the collaborative work of deep semantic representation and shallow spatial information, effectively improving the stability and accuracy of the specular removal task.

[0092] like Figure 4 In this network architecture, the FPTransformer module is one of the core components, combining the advantages of frequency processing (FP) and the Transformer architecture to extract image features more efficiently and enhance multi-scale modeling capabilities. Its structure typically consists of three key sub-modules: a frequency transformation module, a Transformer encoding module, and a cross-scale feature fusion module. First, the frequency transformation module (FP) maps the input features from the spatial domain to the frequency domain, usually employing learnable frequency filters or Fourier transform-like mechanisms to extract features with local details (high frequencies) and structural information (low frequencies) from the image. Through frequency domain operations, the network can more clearly distinguish details in the image, such as edges and highlights, which are difficult to capture in the spatial domain. Subsequently, the frequency domain features are fed into the standard Transformer encoding module, which includes multi-head self-attention (MHSA) and a feedforward network. Network (FFN) models the dependencies between long-distance pixels through an attention mechanism, thereby compensating for the limitations of the receptive field of the convolution kernel; the FPTransformer module can enhance local and global perception by utilizing frequency domain information, while maintaining the modeling advantages of the Transformer architecture in handling complex image tasks, achieving high-precision image de-highlighting and enhancement.

[0093] in addition, Figure 4 The Reshape operation in the model is used to adjust the shape of a tensor without changing the number of its elements to suit the needs of different network layers; Softmax is a normalized exponential function used to transform the input vector into a probability distribution; the FeedForward operation performs feature extraction and dimensionality transformation on the input data through linear transformation and nonlinear activation functions, enhancing the nonlinear expressive power of the model.

[0094] Therefore, a Transformer structure with shared model parameters is introduced to reduce the number of parameters while improving global modeling capabilities. This is the first time a Transformer structure with shared model parameters (i.e., weights) has been introduced in image specular removal tasks. It effectively models long-distance dependencies and global semantic information across regions in the image, significantly improving the restoration quality under large-area specular occlusion and the deployment capability of edge devices. The specular removal task is divided into two stages: coarse separation of the specular-free image and specular residue, and fine-tuning of the specular image. Each stage performs its specific function, progressing layer by layer, and is optimized overall through a unified objective function to ensure the unity of physical consistency, detail restoration, and color accuracy.

[0095] Furthermore, using frequency domain information as input to the Transformer enhances interpretability and stability. In other words, frequency domain processing operations are introduced to transform the Transformer's input into frequency domain information, enabling the model to capture the global illumination distribution based on frequency domain priors. Additionally, tone correction is performed in the HSV color space to improve the realism and consistency of the results. In the second stage, a tone mapping strategy based on regions without highlights is proposed to perform linear tone transformation in the HSV space, accurately correcting the brightness and color imbalance caused by training data bias, making the final output image more natural and closer to the real scene in subjective vision.

[0096] The image processing method of this disclosure embodiment acquires an image to be processed in a preset area using an image acquisition device of a vehicle, and acquires a feature vector to be processed from the image to be processed. It then performs at least one downsampling operation on the feature vector to be processed using at least one first frequency domain processing deep learning network to obtain a first image feature vector. Next, it processes the first image feature vector using a second frequency domain processing deep learning network to obtain a second image feature vector. Finally, it adds the first and second image feature vectors element-wise and inputs them into at least one third frequency domain processing deep learning network for upsampling to obtain an initial no-highlight feature vector. The number of first and third frequency domain processing deep learning networks is the same. The feature vector to be processed, the initial no-highlight feature vector, and the highlight feature vector are concatenated to obtain a target feature vector. The target feature vector is then input into a pre-trained second deep learning model to obtain an intermediate no-highlight feature vector. The pre-trained second deep learning model and the pre-trained first deep learning model share model parameters. The intermediate no-highlight feature vector and the feature vector to be processed are added element-wise to obtain a target no-highlight feature vector, and a target no-highlight image is acquired based on the target no-highlight feature vector. Therefore, through a two-stage divide-and-conquer design (coarse separation of specular highlights and fine refinement of images without highlights), each sub-module focuses on a specific problem, avoiding target interference in multi-task learning, while improving the model's convergence speed and generalization ability. The global modeling capability and staged semantic enhancement mechanism of Transformer make the model more stable and reliable when facing real underground parking lot scenes with large highlight areas and complex image structures. The highlight removal quality is significantly improved, and detail preservation and color consistency are balanced. From structural reconstruction to tone correction, the model parameter sharing mechanism enhances the model's edge deployment capability and reduces the model's memory cost.

[0097] To implement the above embodiments, this disclosure also provides an image processing apparatus, which can be implemented in software and / or hardware.

[0098] Figure 5 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of the present disclosure, as shown below. Figure 5 As shown, the image processing apparatus 50 may include:

[0099] The first acquisition module 510 is used to acquire the image to be processed and to acquire the feature vector to be processed of the image to be processed.

[0100] The first processing module 520 is used to process the feature vector to be processed based on the pre-trained first deep learning model to obtain an initial feature vector without highlights.

[0101] The subtraction module 530 is used to perform element-wise subtraction between the feature vector to be processed and the initial feature vector without highlights to obtain the highlight feature vector.

[0102] The connection module 540 is used to connect the feature vector to be processed, the initial feature vector without highlights, and the feature vector with highlights to obtain the target feature vector;

[0103] The second processing module 550 is used to input the target feature vector into the pre-trained second deep learning model to obtain a feature vector without highlights in the middle; wherein the pre-trained second deep learning model and the pre-trained first deep learning model share model parameters;

[0104] The addition module 560 is used to perform element-wise addition of the intermediate no-highlight feature vector and the feature vector to be processed to obtain the target no-highlight feature vector.

[0105] The second acquisition module 570 is used to acquire the target image without highlights by acquiring the target without highlight feature vector.

[0106] Optionally, before processing the feature vector to be processed based on the pre-trained first deep learning model, the method further includes: a third acquisition module, used to acquire image samples and feature vector samples corresponding to the image samples; wherein the image samples have corresponding no-highlight labels; a third processing module, used to input the feature vector samples into a pre-constructed deep learning neural network for processing to obtain a predicted no-highlight feature vector and acquire a predicted no-highlight image corresponding to the predicted no-highlight feature vector; and a training module, used to adjust the network parameters of the pre-constructed deep learning neural network based on the loss value between the image samples and the predicted no-highlight image and a preset loss threshold until the loss value is less than or equal to the preset loss threshold, thereby obtaining the pre-trained first deep learning model.

[0107] Optionally, the device further includes: a third acquisition module, configured to acquire the non-highlight region of the image sample and the hue difference value of the predicted non-highlight image; a calculation module, configured to calculate the absolute error value between the non-highlight label and the predicted non-highlight image; and a determination module, configured to determine the loss value based on a preset hue coefficient, the hue difference value, and the absolute error value.

[0108] Optionally, the third acquisition module is specifically used to: determine the region without highlights of the image sample based on a preset maximum inter-class variance method; calculate the intensity value of the predicted region without highlights in the HSV color space based on a preset tone transformation function; and calculate the tone difference value based on the intensity value of the predicted region without highlights and the intensity value of the region without highlights.

[0109] Optionally, the pre-trained first deep learning model includes multiple frequency domain processing deep learning networks. The first processing module 520 includes: a first processing unit, used to perform at least one downsampling process on the feature vector to be processed based on at least one first frequency domain processing deep learning network to obtain a first image feature vector; a second processing unit, used to process the first image feature vector based on a second frequency domain processing deep learning network to obtain a second image feature vector; and a third processing unit, used to add the first image feature vector and the second image feature vector element-wise and then input them into at least one third frequency domain processing deep learning network for upsampling to obtain the initial no-highlight feature vector; wherein the number of the first frequency domain processing deep learning network and the number of the third frequency domain processing deep learning network are the same.

[0110] Optionally, the frequency domain processing deep learning network includes a frequency domain transformation module, a deep learning coding module, and a cross-scale feature fusion module. The first processing unit is specifically used for: processing the feature vector to be processed based on the frequency domain transformation module to obtain a frequency domain feature vector; inputting the frequency domain feature vector and the feature vector to be processed into the deep learning coding module for processing to obtain multiple coded feature vectors; and inputting the multiple coded feature vectors into the cross-scale feature fusion module for processing to obtain the first image feature vector.

[0111] Optionally, the first acquisition module 510 is specifically used to: acquire the image to be processed in a preset area based on the vehicle's image acquisition device, and acquire the feature vector to be processed of the image to be processed.

[0112] The image processing apparatus provided in this disclosure can execute the image processing method provided in this disclosure, and has the corresponding functional modules and beneficial effects for executing the method. Content not described in detail in the apparatus embodiments of this disclosure can be referred to the description in any method embodiment of this disclosure.

[0113] This disclosure also provides an electronic device, including a processor and a memory; the processor executes the steps of the foregoing image processing method embodiments by calling programs or instructions stored in the memory. To avoid repetition, these steps will not be repeated here.

[0114] This disclosure also provides a computer-readable storage medium that is non-transitory and stores a program or instructions that cause a computer to perform the steps of the foregoing embodiments of the image processing method. To avoid repetition, these steps will not be repeated here.

[0115] This disclosure also provides a computer program product for performing the steps of the foregoing embodiments of the image processing methods.

[0116] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0117] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image processing method, characterized in that, include: Obtain the image to be processed, and obtain the feature vector of the image to be processed; The feature vector to be processed is processed based on the pre-trained first deep learning model to obtain an initial feature vector without highlights. The feature vector to be processed and the initial feature vector without highlights are then subtracted element by element to obtain a highlight feature vector. The target feature vector is obtained by concatenating the feature vector to be processed, the initial feature vector without specular highlights, and the feature vector with specular highlights. The target feature vector is input into a pre-trained second deep learning model to obtain a feature vector without highlights in the middle; wherein, the pre-trained second deep learning model and the pre-trained first deep learning model share model parameters; The intermediate feature vector without highlights and the feature vector to be processed are added element by element to obtain the target feature vector without highlights, and the target image without highlights based on the target feature vector without highlights is obtained.

2. The method according to claim 1, characterized in that, Before processing the feature vector to be processed based on the pre-trained first-stage deep learning model, the method further includes: Obtain image samples and corresponding feature vector samples; wherein, the image samples have corresponding no-highlight labels; The feature vector samples are input into a pre-constructed deep learning neural network for processing to obtain a predicted feature vector without highlights, and the predicted image without highlights corresponding to the predicted feature vector without highlights is obtained. Based on the loss value between the image sample and the predicted image without highlights and a preset loss threshold, the network parameters of the pre-constructed deep learning neural network are adjusted until the loss value is less than or equal to the preset loss threshold, thereby obtaining the pre-trained first deep learning model.

3. The method according to claim 2, characterized in that, The loss value between the image sample and the predicted specular-free image includes: Obtain the non-highlight regions of the image sample and the tonal difference value of the predicted non-highlight image; Calculate the absolute error between the no-highlight label and the predicted no-highlight image; The loss value is determined based on the preset hue coefficient, the hue difference value, and the absolute error value.

4. The method according to claim 3, characterized in that, The step of obtaining the tonal difference value between the non-highlight regions of the image sample and the predicted non-highlight image includes: The region without highlights in the image sample is determined based on the preset maximum inter-class variance method; The intensity value of the predicted image without highlights in the HSV color space is calculated based on a preset tone transformation function; The hue difference value is calculated based on the intensity value of the predicted image without highlights and the intensity value of the region without highlights.

5. The method according to claim 1, characterized in that, The pre-trained first deep learning model includes multiple frequency domain processing deep learning networks. The process of processing the feature vector to be processed based on the pre-trained first deep learning model to obtain an initial feature vector without highlights includes: The feature vector to be processed is downsampled at least once based on at least one first frequency domain processing deep learning network to obtain a first image feature vector. The first image feature vector is processed by a second frequency domain deep learning network to obtain a second image feature vector; The first image feature vector and the second image feature vector are added element by element and then input into at least one third frequency domain processing deep learning network for upsampling to obtain the initial no-highlight feature vector; wherein, the number of the first frequency domain processing deep learning network and the number of the third frequency domain processing deep learning network are the same.

6. The method according to claim 5, characterized in that, The frequency domain processing deep learning network includes a frequency domain transformation module, a deep learning coding module, and a cross-scale feature fusion module. Based on the first frequency domain processing deep learning network, it performs downsampling processing on the feature vector to be processed to obtain a first image feature vector, including: The frequency domain transformation module processes the feature vector to be processed to obtain a frequency domain feature vector. The frequency domain feature vector and the feature vector to be processed are input into the deep learning coding module for processing to obtain multiple coded feature vectors; The multiple encoded feature vectors are input into the cross-scale feature fusion module for processing to obtain the first image feature vector.

7. The method according to claim 1, characterized in that, The process of acquiring the image to be processed includes: The vehicle-based image acquisition device acquires the image to be processed in a preset area.

8. An image processing apparatus, characterized in that, The device includes: The first acquisition module is used to acquire the image to be processed and to acquire the feature vector to be processed of the image to be processed; The first processing module is used to process the feature vector to be processed based on the pre-trained first deep learning model to obtain an initial feature vector without highlights. The subtraction module is used to perform element-wise subtraction between the feature vector to be processed and the initial feature vector without specular highlights to obtain the specular highlight feature vector. A connection module is used to connect the feature vector to be processed, the initial non-highlight feature vector, and the highlight feature vector to obtain the target feature vector; The second processing module is used to input the target feature vector into the pre-trained second deep learning model to obtain a feature vector without highlights in the middle; wherein the pre-trained second deep learning model and the pre-trained first deep learning model share model parameters; The addition module is used to add the intermediate no-highlight feature vector and the feature vector to be processed element by element to obtain the target no-highlight feature vector; The second acquisition module is used to acquire the target image without highlights by acquiring the target without highlight feature vector.

9. An electronic device, characterized in that, include: Processor and memory; The processor executes the image processing method as described in any one of claims 1 to 7 by calling the program or instructions stored in the memory.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program or instructions that cause a computer to perform the image processing method as described in any one of claims 1 to 7.

Images