Polarization and visible light image fusion texture extraction method and device

Abstract

The application provides a polarization and visible light image fusion texture extraction method and device. A visible light reflection component is first extracted to screen a reliable linear polarization degree image texture and generate a preliminary fusion texture image. Then, a deep learning network model is trained by taking the preliminary fusion texture as a supervised true value, combining a reasonable data enhancement method and a scientific training strategy, so as to eliminate residual block effects and noises in artificial textures. Finally, a fusion texture with high information quantity, high precision, low noise and low block effect is output. In this way, the complementary characteristics of the polarization and visible light images are fully utilized, the texture extraction result is clear and free of artifacts, and the method is suitable for image fusion, edge detection and other real complex scene image detail enhancement and quality optimization.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method and apparatus for extracting textures from polarized and visible light images. Background Technology

[0002] Polarization imaging technology can acquire polarization characteristic information of objects, making up for the shortcomings of traditional visible light intensity images. To address the lack of detailed information in complex scenes, the Degree of Linear Polarization (DoLP) image can reflect unique properties of a target surface, such as material and roughness, and can extract hidden texture information that is difficult to distinguish in visible light images. Furthermore, in real-world environments, for areas with insufficient lighting, DoLP images can eliminate the effects of degraded illumination and restore the true polarization properties of objects. Therefore, the fusion of polarization and visible light images for texture extraction has been widely used in image fusion, edge detection, and other fields.

[0003] Currently, existing technologies for texture extraction from polarized and visible light images have the following limitations:

[0004] On one hand, there are traditional manual texture extraction schemes. Traditional operator-based texture extraction methods, such as the Sobel operator, Prewitt operator, and Laplacian operator, can only directly extract textures from linearly polarized (DoLP) images and visible light images based on gradient magnitudes. However, in real-world scenes, due to insufficient lighting, DoLP images are not entirely reliable and contain a significant amount of noise. Directly extracting gradients from DoLP images using operators will inevitably introduce noise. Furthermore, manual texture extraction schemes largely rely on meticulous parameter design, requiring manual adjustment of algorithm parameters for the data. Despite this, they still struggle to completely separate real textures from noise, failing to meet the demands of high-precision applications.

[0005] On the other hand, most existing deep learning-based polarization image fusion methods use an end-to-end network architecture to directly fuse images, ultimately only obtaining a fused color image. They do not specifically extract the fused texture of polarization and visible light images, nor do they fully utilize the complementary characteristics of DoLP images and visible light images to specifically distinguish and filter reliable textures and unreliable noise in DoLP images. This leads to the network easily misjudging noise as texture and incorrectly retaining it. The final texture extraction result still has the defects of blurred details and noise interference.

[0006] Therefore, there is an urgent need for a fusion texture extraction technique that can fully utilize the complementary characteristics of polarization and visible light images. Through a manually guided deep learning training scheme, high-precision, low-noise fusion texture extraction can be achieved, thus making up for the shortcomings of existing technologies. Summary of the Invention

[0007] This application proposes a method and apparatus for extracting textures from polarized and visible light images, which can solve one of the problems existing in the background art.

[0008] To achieve the above objectives, this application adopts the following technical solution:

[0009] Firstly, a method for extracting textures from polarized and visible light images is provided, including:

[0010] Obtain a linear polarization degree image, a visible light image, and a reflection component image of the visible light image, wherein the linear polarization degree image and the visible light image are of the same origin;

[0011] Furthermore, the linear polarization image, the visible light image, and the reflection component image are input into the trained fusion texture extraction model for processing to obtain a fusion texture map;

[0012] The fused texture extraction model is trained using the preliminary fused texture map as supervised ground truth. The preliminary fused texture map is obtained through the following process: for all local blocks constituting the linear polarization image, the visible light image, and the reflection component image of the visible light image, several local blocks that meet the preset linear polarization texture confidence discrimination logic requirements are determined, and the textures of the determined several local blocks are extracted to obtain the preliminary fused texture map.

[0013] In one possible design approach of the first aspect, the linear polarization degree texture confidence discrimination logic is constructed based on the image texture amplitude extracted by the Sobel operator.

[0014] In one possible design approach of the first aspect, the linear polarization degree texture reliability discrimination logic includes:

[0015] The texture intensity of the linearly polarized image in the local block is examined to see if it is significantly stronger than that of the visible light image. If so, the local block in the linearly polarized image is preliminarily determined to be reliable. The texture intensity is constructed based on the image texture amplitude extracted by the Sobel operator.

[0016] In one possible design approach of the first aspect, the linear polarization degree texture reliability discrimination logic further includes:

[0017] If the texture intensity of the linear polarization image in a local block is significantly stronger than the texture intensity of the visible light image, determine whether the texture intensity of the reflection component image in the local block is stronger than the texture intensity of the visible light image. If so, further improve the credibility of the local block in the linear polarization image.

[0018] In one possible design approach of the first aspect, the linear polarization degree texture reliability discrimination logic further includes:

[0019] When the texture intensity of both the linear polarization degree image and the reflection component image in a local block is stronger than the texture intensity of the visible light image, it is determined whether the texture direction of the linear polarization degree image in the local block is consistent with the texture direction of the visible light image. If so, the texture of the linear polarization degree image of the local block is finally considered reliable.

[0020] In one possible design of the first aspect, the method for extracting textures by fusing polarization and visible light images further includes:

[0021] The regions in the visible light image where the texture amplitude of a pixel is greater than zero are identified as valid texture mask regions. Based on these valid texture mask regions, the linear polarization degree texture confidence discrimination logic is executed.

[0022] In one possible design approach of the first aspect, the preliminary fused texture map is obtained by extracting the textures of several determined local blocks, specifically as follows:

[0023] Once the linear polarization image texture in the local block is deemed reliable, the preliminary fused texture map T is extracted using the following method. f (x,y):

[0024] in, This represents the texture amplitude of an image of type i, where i takes S0 for the visible light image, Ret for the reflectance component image, and DoLP for the linear polarization degree image. The image texture amplitude after linear stretching to the [0,255] interval. The function restricts the final value to the range [0,255], where (x,y) are the pixel coordinates and Mask represents the effective texture mask area;

[0025] When the linear polarization image texture in a local block is ultimately deemed unreliable, a preliminary fused texture map T is extracted using the following method. f (x,y): , .

[0026] In one possible design approach of the first aspect, the loss function Ltotal of the fused texture extraction model is:

[0027] Among them, Lchar For pixel-level reconstruction loss, L grad For gradient consistency loss, L SSIM The structural similarity loss is used; the weight coefficients are set empirically as follows: , and N is the total number of pixels. This represents the model's predicted texture map. This represents the supervised ground truth, i.e., the artificially fused texture image extracted in step S2, where i represents the i-th pixel in the image, and is a constant. Set the smoothing term to 10. -6 ; and These are Sobel convolution kernels in the horizontal and vertical directions, respectively. This term, through nested Charbonnier penalty functions, achieves robust constraints on image gradient differences; K represents the number of segments into which a single image is divided. The total number of partial blocks in the sliding window. and They represent the first At each spatial location Local image patch.

[0028] In one possible design of the first aspect, the texture intensity includes: average texture intensity and total texture energy; the linear polarization image, the visible light image, and the reflection component image of the visible light image are y-channel images; the method further includes: randomly sampling the training data of the fused texture extraction model.

[0029] In a second aspect, an electronic device is provided, comprising: a processor, and a memory coupled to the processor, the memory for storing a computer program; the processor for executing the computer program stored in the memory to cause the electronic device to perform the polarization and visible light image fusion texture extraction method as described in any possible implementation of the first aspect.

[0030] Beneficial effects:

[0031] Based on the above technical solution, the visible light reflectance component is first extracted to screen reliable textures in the linearly polarized image and generate a preliminary fused texture map. Then, using the preliminary fused texture as the supervised ground truth, a deep learning network model is trained using appropriate data augmentation methods and a scientific training strategy to eliminate residual block artifacts and noise in the artificial texture. Finally, a fused texture with high information content, high precision, low noise, and low block artifacts is output. This fully utilizes the complementary characteristics of polarization and visible light images, resulting in clear texture extraction without artifacts, suitable for image detail enhancement and quality optimization in real-world complex scenarios such as image fusion and edge detection. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 This is a flowchart of the texture extraction scheme for the fusion of polarization and visible light images provided in the embodiments of this application;

[0034] Figure 2 This application provides preliminary manually extracted texture effects and image, The reflection component of the image, the Sobel texture comparison map of the DoLP image, and the schematic diagram of the source of the artificially fused texture. Figure 2 (a) to Figure 2 (f) are schematic diagrams of the texture sources of the visible light image, the visible light image texture map, the visible light image reflection component texture map, the linear polarization degree image texture map, the preliminary artificial texture map, and the artificial texture map, respectively.

[0035] Figure 3 This is a structural diagram of the fusion texture extraction model provided in the embodiments of this application;

[0036] Figure 4 This is a comparison image of the texture extraction model effect and the artificial texture extraction effect provided in the embodiments of this application, wherein... Figure 4 (a) to Figure 4 (d) represent the visible light image, the linear polarization image, the preliminary artificial texture map, and the model texture map, respectively. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0038] It should be noted that although functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification and the above-mentioned figures are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0039] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0040] This embodiment provides a texture extraction method that fuses polarization and visible light images. It fully utilizes the complementary characteristics of polarization and visible light images to achieve high-precision, low-noise fused texture extraction through a manually guided deep learning training scheme. This method overcomes the shortcomings of existing technologies and is suitable for scenarios such as target detail enhancement and image quality optimization in real and complex scenes.

[0041] The main objective of this embodiment is to provide a method for extracting fused textures from polarized and visible light images. It uses a preliminary fused texture obtained through manual extraction rules as a priori guide, which is then used as the supervised ground truth (GT) to train the fused texture extraction model. This leverages the nonlinear feature mapping capability of the deep learning model to eliminate block artifacts and noise in the preliminary fused texture. First, manual extraction rules for the fused texture are designed, utilizing Retinex theory to extract the reflection component of the visible light image (denoted as Retinex). This method distinguishes between reliable textures and unreliable noise in DoLP images, manually extracts a preliminary fused texture map, and then uses this preliminary texture map as supervised information to train a texture extraction model. Through reasonable data augmentation, model design, and training strategies, it specifically eliminates the block effect and residual noise caused by manual extraction, ultimately achieving high-precision, low-noise fused texture extraction. This solves the defect of noisy fused textures in existing technologies and fills the gap in existing technologies for fused texture extraction methods of polarized images and visible light images.

[0042] To achieve the objectives of the above embodiments, such as Figure 1 As shown, this embodiment provides a method for extracting textures from polarized and visible light images, comprising the following steps:

[0043] S1. For the acquired polarization and visible light homologous images, analyze them to obtain the DoLP image and visible light... The image was extracted using the existing Retinex theoretical model SCI (with a weight of difficulty). Retinex component of an image This provides a data foundation for subsequent texture selection;

[0044] S2. For DoLP images, Retinex Images and The image design uses an artificial texture extraction scheme to screen reliable strong textures in DoLP images, remove unreliable noise, and generate a preliminary fused texture map.

[0045] S3. Using the preliminary fused texture map obtained in step S2 as the ground truth (GT), construct a training dataset and train the fused texture extraction model using appropriate data augmentation methods and training strategies. This is achieved through methods such as DoLP and Retinex. and The texture extraction model is trained by inputting the y-channel image and outputs a high-precision fused texture map that eliminates block artifacts and noise.

[0046] As described in step S1, for the acquired polarization and visible light homologous image, the DoLP image and visible light are analyzed. Image, and obtain The reflection component of an image is determined through the following steps:

[0047] 1) Acquire original images of the same scene in multiple polarization directions using a split-focus plane polarization camera to obtain intensity images of the four polarization directions: 0°, 45°, 90°, and 135°.

[0048] 2) Based on the Stokes parameter calculation method, the linear polarization degree DoLP image is calculated from the intensity images of the four polarization directions, and the total intensity of visible light is also calculated. Images are used to ensure spatial registration accuracy between two images, achieving pixel-level positional alignment.

[0049] 3) Extract visible light using the existing Retinex theoretical model SCI (weighted as difficult). Retinex image of the reflection component This image enhances the realistic edge and texture details in low-light areas of the scene, thereby eliminating the interference of insufficient lighting and making the textures in poorly lit areas more clearly visible.

[0050] Step S2, based on DoLP and Retinex and The image design employs a manual texture extraction scheme, screening reliable strong textures in DoLP images, removing unreliable noise, and generating a preliminary fused texture map. The specific steps are as follows:

[0051] 1) Apply DoLP and Retinex respectively and The images were converted from RGB format to YCbCr format to achieve brightness and color separation. Because the y-channel image concentrates the geometric edges and texture structure, extracting texture from the y-channel can fully capture the true physical structure of the image. Therefore, subsequent manual and model texture extraction are both based on the y-channels of the three images. For the y-channels of the three images, we first perform edge detection using the Sobel operator to extract the texture amplitude response maps of the three images. The formula for extracting the texture amplitude T(x,y) is:

[0052] in and Use Sobel convolution kernels. This represents the input image. (x, y) are the coordinates of different pixels in the input image. The resulting amplitude map... , and The original physical signal strength is preserved as the basis for subsequent discrimination.

[0053] 2) Determination of effective masking region and global enhancement prior estimate: Select the region with the highest confidence and the purest, lowest noise texture. The texture, as a valid texture mask, is denoted as Mask. The determination rule is: in... In a texture, if the texture magnitude of a pixel is greater than zero, then that pixel is marked as a valid texture mask region. The specific formula is as follows:

[0054] in express The texture magnitude of the modality at different pixel locations. Subsequent local statistics calculations and fusion logic are performed within the valid texture mask. For non-valid texture areas, since they do not contain any texture information, zero values ​​are directly selected for filling. For each scene... Retinex We divide the DoLP modal texture image into corresponding 64*64 blocks. (i,j represent different positions of the small blocks). Depending on the mode, one can take... , and For each small block Calculate the following two statistics within its effective texture mask region:

[0055] in To represent different small blocks, This represents the valid texture mask, ensuring that only the texture mean and texture energy of the valid texture regions are calculated. This indicates the small block Within the region, the Sobel texture magnitude of different pixels in the effective texture mask, where T can take different values ​​depending on the mode. Retinex The texture amplitude of the DoLP mode, i.e. , and . This represents the average texture intensity of the small region, which can be taken from different texture modalities. , and . This represents the total texture energy of the effective texture mask region within a small area. It is sensitive to high-frequency energy and can be determined based on different texture modes. , and N represents the small block. The total number of pixels representing the effective texture mask within the region. Due to Retinex... Not only can significantly enhance The texture in low-light areas of an image can also enhance the texture in other areas, resulting in Retinex. Overall texture amplitude intensity ratio of the image Higher. To eliminate this effect, ensure Retinex... Image texture can correctly guide subsequent judgments; this method constructs a global enhancement scaling factor. and As Retinex Compared to Global enhancement priors for images. Unlike local block statistics, and In the entire image The global enhancement scaling factor calculated within the effective texture mask area. The calculation formula is:

[0056] Based on the formula for calculating average texture intensity mentioned above, it can be expanded as follows:

[0057] The calculation formula is:

[0058] Based on the formula for calculating the total texture energy mentioned above, it can be expanded as follows:

[0059] in Retinex The average texture intensity of the entire texture image within the effective texture mask. express The average texture intensity of the entire texture image within the effective texture mask. Retinex The total texture energy of the entire texture image. express The total texture energy of the entire texture image. To avoid the extremely small constant of division by zero, it is 10 here. -6 . Retinex Image texture relative The ratio of the image texture to the mean across the entire image. Retinex Image texture relative The energy ratio of image texture across the entire image. When or When the value is zero, it indicates that there is no valid texture in the entire image. or The default value is 1. The two global enhancement scaling coefficients are adaptively calculated based on the input image and are used to measure Retinex. Image texture compared to The global average enhancement level of image texture, thereby eliminating Retinex. The effect of enhanced overall texture brightness.

[0060] 3) Polarization image credibility judgment logic;

[0061] DoLP can significantly eliminate the illumination effects in low-light areas, restoring the true polarization characteristics of objects in these areas and thus enhancing texture information. However, under low-light conditions, the incident light intensity approaches the sensor's low-noise level, causing weak hardware noise in the DoLP image to be drastically amplified, appearing as high-energy, high-frequency random patches with edge-like features. This noise is randomly distributed between textured and non-textured areas in the low-light region, constituting invalid interference information that needs to be removed. Retinex... The image only significantly enhances textured areas in low-light regions, without significantly enhancing non-textured areas in low-light regions, and does not introduce strong noise textures. Therefore, we chose to use Retinex. The image is used to determine the credibility of DoLP textures, filtering out significantly enhanced and reliable textures from the DoLP images without introducing noise from the DoLP images. Considering the heterogeneity of local features in different images, we adopt a block processing strategy (block size of 64×64) to divide the texture images of the three modalities into non-overlapping local blocks. (where i and j represent different positions of the local block). Depending on the mode, one can take... , and For each local patch, we calculate the mean texture value of that local patch across the three modalities. , and and the texture energy of local blocks in the three modes. , and Next, for each local block, the following three-layer saliency check needs to be performed to determine whether its DoLP texture is saliency and reliable:

[0062] 1. Significance test of polarization texture;

[0063] First, use the following formula to check whether the DoLP image texture intensity in the local block is significantly stronger than... image:

[0064] in The significance threshold is represented, and in this paper, it is set to 1.5 based on experience. and DoLP and Texture image in local blocks Average texture intensity in and DoLP and Texture image in local blocks The formulas for calculating the total texture energy, average texture intensity, and total texture energy have been explained above. This is a criterion used to indicate whether the polarization texture saliency test is valid. When... If the condition is not met, it indicates that the DoLP texture within that local block does not generally have [something]. The image has strong texture, so there is no need to introduce DoLP texture information for that local block. If If the condition is met, it indicates that the DoLP image texture is significantly stronger than that in this local block. Image texture. At this point, the local block could be either a region with significantly enhanced texture or a noise block in the DoLP image, so a significance test of the reflection component is needed for further analysis.

[0065] 2. Significance test of reflection component texture;

[0066] Targeting the satisfaction For local texture patches under certain conditions, the significance of reflection components is further tested using the following formula, aided by Retinex. The image determines whether the local patch is a significantly enhanced texture patch or a noise-interference patch:

[0067] in This represents the significance threshold, which is still set to 1.5. It is used to determine the significance of Retinex in a local block. Is the texture of the image greater than... The more intense the image texture, the higher the value, and the stricter the requirements for judging it as a significant enhancement. To avoid the extremely small constant of division by zero, it is 10 here. -6 . and Retinex and Texture image in local blocks Average texture intensity in and Retinex and Texture image in local blocks The formulas for calculating the total texture energy, average texture intensity, and total texture energy have been explained above. and These represent the mean ratio and energy ratio determined above, respectively. Since in Reinex... In the image, the texture intensity in areas with insufficient light will be significantly enhanced, while the texture intensity in other areas will only be slightly enhanced, thus the overall texture density will be higher. The image is more intense, depending on the settings. and Effectively eliminates Retinex The stronger overall texture of the image allows the model to focus more on recognizing areas with insufficient light. The condition is used to indicate whether the saliency test of the reflection component texture is valid. When a local block does not satisfy the condition... When a condition is met, it indicates that the local block is in Retinex. The image shows no significant enhancement, indicating that this local block is more likely a noise block than a valid texture block in a DoLP image. If the local block satisfies... The conditions then specify the use of DoLP images and Retinex. In the image, the texture of the local block is significantly enhanced, indicating that the local block is more likely to be a texture block.

[0068] It's important to note that in poorly lit areas, noise in the DoLP image is randomly distributed between textured and non-textured areas. Since noise in non-textured areas exists only in the DoLP image and not in the Retinex image itself... In the image, the significance test of the reflectance component can effectively filter out this type of noise; however, for noise existing in low-light texture areas, Retinex... Since it has also been enhanced, the significance test of the reflection component is insufficient to eliminate it, thus introducing noise. Therefore, a third layer of significance test, namely the structural consistency test, is needed.

[0069] 3. Structural consistency check;

[0070] When the local block passes and After the initial inspection, structural consistency testing will proceed. Because the noise distribution in the DoLP image is chaotic and differs from the geometric structure of the real texture, structural consistency testing can determine whether the DoLP and the actual texture mask are identical within the valid texture mask of that local block. We screen for DoLP textures with realistic texture structures by checking whether the texture gradient directions of each pixel in the image are similar, thus filtering out noise in the DoLP image. We perform a structural consistency check by calculating the mean cosine similarity between the gradient directions of each pixel. The specific judgment rules are as follows:

[0071] in and Represent The orientation vectors of the image and the DoLP image texture, where (x, y) represent the pixels located within the effective texture mask. The structural similarity threshold is set to 0.70 based on experience. To avoid the extremely small constant of division by zero, it is 10 here. -6 . The condition is used to indicate whether the structural consistency check is valid. In the valid texture mask within a local block, when DoLP and... The mean cosine similarity between textures is greater than or equal to At that time, we considered DoLP images and If the image has a similar texture structure within this local patch, then the DoLP texture within that local patch is considered a reliable strong texture. When the DoLP and... The mean cosine similarity between textures is less than At that time, we considered the texture of the DoLP image and Since the textures of the images are not uniform, the DoLP local block is more likely to contain messy noise blocks than ordered texture blocks, so this local block is not introduced.

[0072] 4) Pixel-level texture extraction;

[0073] In the texture fusion extraction stage, to eliminate the differences in the absolute values ​​of amplitudes between different modalities and enhance the contrast of the fused texture map, we first perform DoLP... and Retinex The image amplitude is linearly stretched and mapped to a continuous pixel range of [0, 255]. The calculation formula is shown below:

[0074] in These are represented as DoLP, and Retinex The texture amplitude of the image, The value represents the texture amplitude after linear stretching, and i can be taken as DoLP. Or Retinex Three modes. The function restricts the final value to the range [0, 255]. Based on the judgment rules mentioned above, this paper uses the following method for texture extraction:

[0075] When local blocks simultaneously satisfy , and Under certain conditions, this indicates that the DoLP texture in the local block is reliable, and the texture intensity is significantly stronger than... Image. For all pixels in the effective texture mask of this local patch, select DoLP, Retinex. and The maximum value of the texture after linear stretching is used as the final blended texture amplitude:

[0076] When local blocks simultaneously satisfy and The conditions are met, but not satisfied. When the condition is met, it indicates that the DoLP image texture in the local block is significant, and that the local block is within the Retinex range. The texture structure of DoLP is also significantly enhanced, but its DoLP texture structure and Since image textures are not uniform, local blocks in DoLP are more likely to be noise blocks than texture blocks with realistic structure. And when a local block only satisfies... Conditions not met When the condition is met, it indicates that the local block only has strong texture in the DoLP image, while in Retinex... The absence of a corresponding strong texture in the image further suggests that this local block is more likely to be a noise block. If this local block does not satisfy... The conditions are not met. The condition specifies whether the texture of the local block is in DoLP or Retinex. There is no significant enhancement in the image, so no enhancement is needed. For all three cases above, we uniformly use the following formula to extract local block texture:

[0077] Based on the above fusion texture extraction rules, we can fully utilize the strong texture supplementation of DoLP images in low-light areas. The image addresses shortcomings in dark areas while avoiding the introduction of noise from the DoLP image. The final fused texture extraction result is shown in the attached figure. Figure 2 As shown, it can be observed that compared to DoLP and Retinex From single-source images, the extracted fused textures are clearer and more comprehensive, for example in... Figure 2 In automotive scenarios, our artificial texture map, while satisfying texture richness, correctly enhances the texture of the automotive grid region by leveraging the strong texture in the linearly polarized image. Simultaneously, it combines... Figure 2 (f) In this sub-figure, red represents the texture of the point from the linear polarization image, blue represents the texture of the point from the visible light reflection component image, and green represents the texture of the point from the visible light image. It can be found that in low-light areas with insufficient illumination, such as the shadow area of ​​a car, the shadow area inside the car grid is mainly red texture, indicating that the fused texture of these shadow areas mostly comes from the DoLP image, which further verifies the correctness of our rule design.

[0078] While artificial texture extraction algorithms are simple to implement, require no training, and have low algorithmic complexity, their extraction results are highly dependent on the reasonable design of parameters, making it difficult to perfectly achieve fused texture extraction for all scenes. Some samples will exhibit block artifacts and physical noise. To eliminate the block artifacts and noise in the artificial texture extraction results and make the fused texture closer to the real natural texture of the image, this method trains a fused texture extraction model to extract the final fused texture. The specific steps for training the texture extraction model in step S3 are as follows:

[0079] 1) Structural design of the fusion texture extraction model;

[0080] like Figure 3As shown, this model is an improvement on the UNet model. The input features are sequentially processed through four downsampling and four upsampling modules, outputting a final fused texture map. The downsampling module consists of a convolutional layer with a stride of 2 and a kernel size of 3×3, along with a LeakyReLU activation function. The upsampling module is implemented using 2×2 bilinear interpolation. The convolutional modules are responsible for feature extraction and feature map channel transformation. Skip connections concatenate the features from the downsampling and upsampling processes, achieving the fusion of shallow detail features and deep semantic features. Finally, a Softplus activation function is introduced into the model's output layer, combining its output with the learnable... After multiplying the parameters, a truncation operation is performed to ensure that the final output of the model remains in the [0,1] interval. Finally, during inference, it is linearly transformed into a texture map distributed in [0,255] and saved. Figure 3 In this code, Conv represents the convolutional layer module, stride represents the stride, BN represents the batch normalization module, LReLU represents the Leaky ReLU activation function, Dropout represents the random deactivation module, Softplus represents the Softplus activation function, and Clamp represents the numerical truncation module.

[0081] 2) Training data construction;

[0082] This method uses manually extracted texture maps as supervised ground truth (GT) to train a fusion texture extraction model. To prevent the model from forcibly memorizing block artifacts and noisy regions from the GT data during training, this method selectively removes some manually extracted texture samples with obvious block artifacts or noise, and divides the remaining data into a training set:test set ratio of 8:2. Simultaneously, we artificially place data with weak block artifacts but no noise into the test set. Although these samples have some block artifacts, they only exhibit slight variations in texture brightness and can still guide the model's correct training. Placing them in the training set would lead to the model overfitting the block artifacts and noise in the data. Furthermore, during training, this method performs 16 overlapping 512*512 random samplings for each training sample, and adds data augmentation operations such as random rotation, horizontal flipping, and vertical flipping to avoid the model overfitting the block artifacts in the data, thus allowing the model to more specifically learn the general rules of texture extraction. During the verification process, to avoid the randomness of verification, we chose to divide the verification data into 512*512 data for full sampling verification. To improve the training accuracy of the model, we wrote the judgment and extraction logic of artificial texture maps into the data loading code (dataloader), so that online extraction can be performed during model training, preventing the truncation error that occurs when artificial texture maps are saved offline as image data.

[0083] Random cropping refers to randomly cropping 512*512 pixels from a 2048*2048 sample. The location of the cropped pixels is not fixed, ensuring the model learns different data in each training iteration. This effectively prevents the model from memorizing noise and block artifacts present in artificial textures. For example, if the model is trained for 200 epochs with fixed cropping, it will learn the same sample 200 times, potentially memorizing all information, including both realistic textures and noise. With random cropping, the model learns different randomly cropped data in each epoch, preventing forced memorization. Therefore, random cropping effectively improves the model's texture extraction performance.

[0084] 3) Loss function design;

[0085] This method uses a composite loss function to constrain the training process. This function consists of three parts: pixel-level reconstruction loss L... char Gradient consistency loss L grad and structural similarity loss L SSIM The overall objective function is defined as follows:

[0086] The weighting coefficients are set empirically as follows: , and Below are the specific definitions of each loss function:

[0087] 1. Pixel-level reconstruction loss L char :

[0088] With standard Compared to other loss methods, we use Charbonnier loss to constrain the intensity of the texture maps predicted by the model, thereby improving the model's robustness to outliers:

[0089] Where N is the total number of pixels in each patch (patch represents a sample after cropping a 2048*2048 image to 512*512). This represents the model's predicted texture map. This represents the supervised ground truth, i.e., the artificially fused texture image extracted in step S2, where i represents the i-th pixel in the image, and is a constant. Set the smoothing term to 10. -6 .

[0090] 2. Gradient consistency loss L grad ;

[0091] To enable the fusion texture extraction model to better learn high-frequency texture details of the ground truth, we introduce a gradient consistency loss based on the Sobel operator:

[0092] in and These are Sobel convolution kernels in the horizontal and vertical directions, respectively. This term, through nested Charbonnier penalty functions, achieves robust constraints on image gradient differences.

[0093] 3. Structural similarity loss L SSIM ;

[0094] To enhance the visual perception of the fused textures extracted by the model, this method introduces the average structural similarity loss (SSIM), which measures texture quality from three dimensions: brightness, contrast, and structure. This is particularly relevant for predicting images. and monitoring truth value We use an 11×11 sliding window local block to calculate its SSIM value. The SSIM loss function is defined as:

[0095] Where K is the number of segments into which each patch image data is divided. The total number of partial blocks in the sliding window. and They represent the first At each spatial location Local image patch.

[0096] 4) Training strategy design;

[0097] The model employs the Adam optimizer with an initial learning rate of 0.0001 and a single-cycle cosine annealing strategy. The parameter T_max covers the total number of steps in the entire training cycle, and eta_min is set to 10. -7 The training rounds are 300, the batch size is set to 6, and the weights with the lowest loss in the validation set are selected as the optimal weights to achieve incremental model tuning.

[0098] 5) Integrating texture reasoning;

[0099] DoLP, and Retinex The y-channels of the image are concatenated as input to the model. The best weights with the lowest loss on the validation set during training are selected for inference, resulting in a fused texture map that is free of block artifacts, has low noise, and is rich in texture.

[0100] The advantages of this embodiment are as follows:

[0101] 1. Corrected using Retinex The image accurately distinguishes strong textures from noise in DoLP images, effectively avoiding noise being misjudged as texture, and significantly improving the accuracy and reliability of DoLP image texture extraction in real-world scenes.

[0102] 2. A scheme combining manual prior knowledge construction and deep learning is adopted. This scheme not only utilizes the prior knowledge extracted manually to ensure the targeting and physical rationality of texture selection, but also eliminates the block artifacts and residual noise that are unavoidable in manual extraction through deep learning models. It balances the accuracy of prior knowledge and the optimization capability of models, and significantly improves the extraction effect of polarization and visible light image fusion texture in real scenes.

[0103] 3. The model has strong adaptability and can be applied to scenes with complex lighting distribution in real-world environments. It uses the strong texture of DoLP images in low-light areas to supplement the lack of texture in visible light images in low-light areas, and can be widely used in image processing fields such as image fusion and semantic segmentation.

[0104] 4. The main measures taken to prevent overfitting during model training are:

[0105] (1) In the model architecture, different dropout layers were added during the model downsampling and upsampling processes to prevent the model from overfitting the block effect and noise in the artificially fused texture map.

[0106] (2) In the dataloader, for the training set samples, this method selects to randomly crop each training set sample and adds random rotation, horizontal flip and vertical flip to increase generalization, so that the model will not forcibly memorize the noise and block effect of the training set data in repeated training.

[0107] (3) Because the model adopts the UNet architecture, which is a model architecture that first downsamples and then upsamples, the resolution of the data sampling also has an impact. In the final model, this method selects a sampling resolution of 512*512 for both the training and validation sets. In each epoch, for each training set sample, this method selects 16 randomly cropped 512*512 images online in the dataloader as training data. For the validation set, 16 non-overlapping 512*512 images are selected. Since the dataset resolution used in this method when training the model is 2048*2048, it is actually fully sampled in the validation set.

[0108] The following example illustrates the method's texture extraction scheme for fusing polarization and visible light images.

[0109] Still Figure 1As shown, the flowchart of the polarization and visible light image fusion texture extraction scheme of this method includes the following steps:

[0110] S1. For the acquired polarization and visible light homologous images, analyze them to obtain the DoLP image and visible light... The image was extracted using the existing Retinex theoretical model SCI (with a weight of difficulty). The reflection component of the image provides the data basis for subsequent texture selection;

[0111] S2. For DoLP images, Retinex Images and The image design uses an artificial texture extraction scheme to screen reliable strong textures in DoLP images, remove unreliable noise, and generate a preliminary fused texture map.

[0112] S3. Using the preliminary fused texture map obtained in step S2 as the ground truth (GT), construct a training dataset, employ appropriate data augmentation methods and training strategies to train the texture extraction model, and integrate DoLP and Retinex... and The texture extraction model is trained by inputting the y-channel image and outputs a high-precision fused texture map that eliminates block artifacts and noise.

[0113] The following examples illustrate this point:

[0114] This case study uses the CPIF dataset collected from a school for artificial texture extraction. Ten images with obvious block artifacts or noise were removed, and the remaining 90 images were used as ground truth for supervision. The dataset was split into training and test sets in an 8:2 ratio. The test set consisted of manually selected data, the reasons for which are explained above. The Monno dataset was chosen for testing the fusion texture extraction of linearly polarized and visible light images.

[0115] S1. For the acquired polarization and visible light homologous images, analyze them to obtain the DoLP image and visible light... The image was extracted using the existing Retinex theoretical model SCI (with a weight of difficulty). The reflection component of an image;

[0116] S2. Based on the artificial texture extraction scheme designed above, reliable strong textures in the DoLP image are screened, unreliable noise in the DoLP image is removed, and reliable strong textures in the DoLP image are extracted to obtain a preliminary fused texture map. This paper uses nine no-reference metrics to comprehensively evaluate different texture maps. Specifically, MI and SD are used to measure texture response intensity and grayscale dispersion. Sobel-AG and SF are used to measure texture edge sharpness and spatial detail activity. EN is used to measure grayscale information richness. GLCM Contrast and GLCM Dissimilarity are used to measure second-order neighborhood texture structure. NLE is used to estimate noise level. Blockiness-64 is used to measure the blockiness caused by the 64×64 sliding window decision in the artificial texture map. Through the above metrics, we can comprehensively evaluate the model texture map, the artificial texture map, and... Retinex And the quality of DoLP image textures, among which Retinex Textures from both the DoLP image and the image itself are extracted using the Sobel operator, as shown in the formula below. Artificial texture maps, on the other hand, are extracted using the fusion rules described above.

[0117]

[0118] in and Use Sobel convolution kernels. The input image is represented by [image name]. The final performance metrics for each modality image and artificial texture are shown in Table 1 (red indicates optimal, blue indicates suboptimal):

[0119] Table 1. Average metrics of artificial textures and source textures on the CPIF dataset.

[0120] Observing the results in Table 1, it can be seen that artificial texture maps exhibit significant advantages in texture response intensity and spatial detail representation. Specifically, the MI of the manual texture reaches 8.19052, which is higher than... Texture, DoLP Texture and Retinex The Manual Texture score indicates that it possesses a stronger average texture response. Regarding grayscale dispersion, the Manual Texture's SD is 16.71926, also the best among all modal textures, indicating a richer texture response distribution and the ability to distinguish texture regions of different intensities over a wider dynamic range. Furthermore, the Manual Texture achieves SDs of 26.56554 and 9.89892 on Sobel-AG and SF respectively, both superior to other modal textures, demonstrating that the Manual Texture has a stronger local gradient response and more active spatial frequency variations, thus better preserving edges, details, and local texture structure.

[0121] From the perspective of second-order texture statistical features, Manual Texture also exhibits stronger local grayscale variation capabilities. Its GLCM Contrast and GLCM Dissimilarity reach 1.30012 and 0.34753 respectively, both being the best values ​​among all modalities, indicating that the manual texture map has more obvious grayscale differences and richer texture levels between neighboring pixels. In contrast, The GLCM metrics for both Texture and DoLP Texture are generally low, indicating relatively weak texture response. (Retinex) While the Texture class can enhance some dark areas, increasing its GLCM Contrast to 1.19603, it is still lower than that of the manual texture. In terms of entropy, the Manual Texture's EN is 3.56096, which is similar to Retinex. Texture's 3.57599 is close to, and higher than, that of other materials. The scores of 2.93502 for Texture and 3.49101 for DoLP Texture indicate that artificial texture maps enhance texture responsiveness while maintaining sufficient grayscale information complexity.

[0122] It's important to note that the NLE metric in this method cannot be simply interpreted as the true noise level of the texture. The NLE for Manual Texture is 0.66284, which is higher than... Texture, DoLP Texture and Retinex Texture, however, does not necessarily mean that artificial texture maps introduce more realistic noise. The reason is that NLE is a no-reference noise estimation method, which has difficulty distinguishing between realistic detail textures and high-frequency noise perturbations. DoLP and Retinex have relatively weak texture response in many areas, so their lower NLE values ​​are likely due to insufficient texture information rather than a cleaner image texture. Texture and Manual Texture extract a more comprehensive texture structure, thus generating more high-frequency responses and leading to higher NLE values. Therefore, in this method, NLE is only used as an auxiliary reference for high-frequency noise perturbations and not as a strict criterion for determining the true noise content of the texture.

[0123] On the other hand, the main drawback of Manual Texture lies in its significant blockiness. Its Blockiness-64 reaches 0.18287, significantly higher than S0 Texture, DoLP Texture, and Retinex S0 Texture, indicating stronger discontinuities at block boundaries with a 64-pixel stride. Since Manual Texture is obtained by local texture determination and stitching based on 64×64 sliding window blocks, some stitching artifacts are inevitably introduced at block boundaries. In summary, Manual Texture provides richer and stronger texture priors, but it also contains the 64×64 blockiness resulting from the sliding window construction process. Therefore, using Manual Texture as a priori guide in subsequent model training allows the model to learn its effective and universal texture response patterns, while further suppressing the blockiness and noise perturbations remaining from the manual construction process.

[0124] This method will Texture map, DoLP texture map, Retinex Qualitative comparison of texture maps and artificially merged texture maps, such as... Figure 2 As shown, where Figure 2 In (f), the different colors represent different source images in the artificially fused texture map. Red indicates that the texture here comes from a DoLP image, and blue indicates that the texture here comes from a Retinex image. The image shows that the green color represents the texture where the image originates. Image. Observation. Figure 2 It can be observed that artificial textures are more comprehensive and clearer than any single-modal texture. For example, in automotive scenes, our artificial texture map, while satisfying texture richness, correctly enhances the texture of that region by leveraging the strong texture in the car grid area of ​​the linear polarization image. Simultaneously, it combines... Figure 2 (f) It can be observed that in low-light areas with insufficient illumination, such as the shadow areas of cars and the shadow areas of car grids, the texture is mainly red. This indicates that the blended textures in these shadow areas mostly come from the DoLP image, meaning that the artificial textures have effectively absorbed the strong texture parts in DoLP. This is exactly the same as the original design intention of this method. Furthermore, due to the reasonable design of the artificial texture extraction in this method, the artificial textures have not introduced strong noise from DoLP over a large area. However, at the same time, observation... Figure 4(c) It can be found that there is some block effect in the artificial texture map, which is why the S3 step is needed to train the fusion texture extraction model.

[0125] S3. Using the preliminary fused texture map obtained in step S2 as the supervised ground truth, with DoLP, Retinex... and Using the y-channel of the image as input, and employing appropriate data augmentation methods and training strategies, a texture extraction model is trained. The model structure is as follows: Figure 3 As shown. After training, the y-channel images of the three modalities are input into the trained texture extraction model to obtain a fused texture map that eliminates blockiness and noise. This paper calculates the index of the texture extraction results of the fused texture extraction model on the test set of the CPIF dataset. Calculating the index only on the test set ensures that the model does not overfit the training set data, leading to artificially high indexes. This method combines the model's texture map with... Texture, Retinex A comparison of textures, DoLP textures, and artificial textures is shown in Table 2:

[0126] Table 2 shows the average metrics of the model texture map and the source texture map on the test set of the CPIF dataset.

[0127] As shown in Table 2, the model texture map achieves a good balance between texture expressiveness and noise perturbation suppression. Compared to Compared to DoLP's textures, the average texture response of the model's texture map is significantly enhanced, with an MI of 9.42716, which is significantly higher than that of DoLP's texture map. The EN values ​​of 6.46977 and DoLP of 5.32732 indicate that the model can generate a stronger overall texture response. In terms of information complexity, the model's texture map achieves an EN of 4.39936, the best value among all texture maps, indicating a richer grayscale distribution and more sufficient texture information capacity. Meanwhile, the Sobel-AG of the model's texture map is 26.98311, significantly higher than... The scores of 20.65965 for the model and 16.80766 for DoLP are slightly higher than those of Retinex (26.74260), indicating that the model has achieved a slight improvement over Retinex in terms of local edge and texture gradient response. The level. Although the model texture map's SF is 8.69667, it is lower than Retinex's. The value is 9.55962, but still higher than S0 and DoLP, indicating that while the model enhances spatial texture variations, it does not achieve the same effect as Retinex. This results in stronger overall high-frequency fluctuations. The GLCM Dissimilarity of the model texture map is 0.35652, slightly higher than Retinex. 0.35163, and significantly higher than The results, along with DoLP, indicate that the model possesses a strong ability to represent local gray-level differences. Meanwhile, the model's texture map NLE is 0.52341, significantly lower than Retinex. 0.63337, and quite close to The level indicates that the model effectively suppressed Retinex while achieving richer texture representation. A subtle pseudo-texture is introduced during the enhancement process. Therefore, compared to DoLP and Retinex The model texture map does not simply amplify texture or high-frequency information, but achieves a better balance between enhancing effective texture response, increasing information complexity, and controlling noise sample perturbation.

[0128] Compared to the artificial texture map used as a supervisory signal, the model texture map significantly reduced noise-sample perturbations and the 64×64 block artifact present in the artificial texture map while maintaining a similar average texture intensity. Specifically, the MI of the model texture map was 9.42716, almost on par with the 9.44617 of the artificial texture map, indicating that the model learned the overall texture response intensity of the artificial texture map well. The SD, Sobel-AG, SF, and GLCM Contrast of the model texture map were 14.60763, 26.98311, 8.69667, and 0.98862, respectively, all lower than those of the artificial texture map (16.22858, 29.93954, 10.03521, and 1.26873), indicating that the model did not mechanically learn the strong gradients, strong local mutations, and block artifact boundaries in the artificial texture map, but rather suppressed the unnatural high-frequency responses. More importantly, the NLE of the model texture map decreased from 0.66910 to 0.52341, and the Blockiness-64 decreased significantly from 0.13650 to 0.05989. This indicates that while learning from the prior knowledge of the human texture map, the model effectively mitigated the noise introduced by the local sliding window decision and stitching process, as well as the blockiness boundary of the 64×64 texture map. Notably, the EN of the model texture map reached 4.39936, higher than the 4.07457 of the human texture map, indicating that the model maintained richer grayscale levels and information complexity while reducing noise perturbation and blockiness.

[0129] In summary, the model texture map is not a mechanical learning of the artificial texture map, but rather, based on inheriting its effective texture response rules, it suppresses noise and block artifacts in the artificial texture map. Therefore, the model texture map achieves a more stable overall performance in terms of texture response intensity, information richness, noise control, and block artifact suppression, demonstrating the model's ability to effectively learn from and further optimize prior artificial textures.

[0130] This method compares the preliminary artificial texture map with the texture obtained from the fused texture extraction model to qualitatively demonstrate that the fused texture extraction model can effectively eliminate block artifacts and noise present in artificial textures. Figure 4 As shown, it is quite evident that the blended texture extracted by the model does not introduce the blockiness and noise present in artificial textures. For example, in Figure 4 In the preliminary artificial texture map (c), the brightness variations of the square textures are clearly observable. This is partly due to the block effect caused by the difficulty of adapting artificial texture extraction rules to all data. Figure 4 In the model texture map (d), it is clearly observed that this block effect has been eliminated. Furthermore, observation of the preliminary artificial texture map reveals that the artificial texture extraction method also struggles to completely remove noise from the linearly polarized image, leaving some noise residue. However, observation of the model texture map clearly shows that the model texture map introduces virtually no noise in these scenarios. To further verify the generalization ability of the proposed algorithm on unknown data, we selected the model trained on the CPIF dataset and used the optimal weights with the lowest loss function, conducting comparative experiments on the Monno dataset. The experimental results are shown in Table 3:

[0131] Table 3 shows the average metrics of the model texture map and the source texture map on the Monno dataset.

[0132] Observing the results in Table 3, it can be seen that the model texture map exhibits strong generalization ability and stable texture reconstruction performance on the unseen Monno dataset. In terms of texture expressiveness, the model texture map achieves an MI of 14.03976, which is higher than... Texture, DoLP Texture and Retinex The presence of texture data indicates that the model can still generate stronger overall texture responses on unknown datasets. Furthermore, the SD of the model's texture map is 21.24213, also the best value among all methods, indicating that the model's texture map has richer gray-level dispersion and texture intensity variations. The Sobel-AG of the model's texture map reaches 45.29493, slightly higher than Retinex. Texture's 45.11605, and significantly higher Texture and DoLP Texture indicate that the model maintains the strongest performance in local edge and detail gradient response. The model texture map has an SF of 17.72980, which is similar to Retinex. The Texture value of 17.77995 is basically the same, indicating that the model achieves Retinex's level of spatial frequency activity. Modal enhancement of texture level. Meanwhile, the information entropy (EN) of the model texture map reaches 4.77120, the highest among all methods, GLCMContrast also reaches the highest at 3.77936, and GLCM Dissimilarity is 0.79728, very close to Retinex. The Texture score of 0.80646 indicates that the model can reconstruct rich grayscale levels and local texture differences in unknown scenes. More importantly, while achieving strong texture response, the model's texture map has an NLE of 1.38788, significantly lower than DoLPTexture's 2.01340 and Retinex's 1.38788. The texture score of 1.65138 indicates that the model does not simply amplify high-frequency textures or enhance noise perturbations, but rather achieves better noise control while maintaining high-intensity texture extraction capabilities. It's important to note that the Blockiness-64 metric is primarily used to analyze whether the model can effectively eliminate the blockiness introduced by the 64×64 sliding window decision in artificial texture maps during training on the CPIF dataset. The Monno dataset is only used for cross-dataset testing and does not participate in supervised training. Therefore, the Blockiness-64 result in the Monno dataset is only used as an auxiliary diagnostic metric and not as the primary basis for evaluating the model's generalization performance. Combined with the model's strengths on MI, SD, Sobel-AG, EN, and GLCM Contrast, it can be demonstrated that the model can effectively learn and transfer texture priors on the Monno dataset, achieving a good balance between rich texture extraction, information integrity preservation, and noise perturbation suppression in unknown and complex scenes, demonstrating strong robustness and generalization performance.

[0133] This application also provides an electronic device, including: a processor, and a memory coupled to the processor, the memory being used to store a computer program; the processor being used to execute the computer program stored in the memory, so that the electronic device performs the method as described in any of the above embodiments.

[0134] Electronic devices can be computing devices such as desktop computers, laptops, handheld computers, and cloud servers. These electronic devices may include, but are not limited to, processors and memory.

[0135] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the electronic device, connecting various parts of the device via various interfaces and lines.

[0136] The memory can be used to store the computer program, and the processor implements various functions of the electronic device by running or executing the computer program stored in the memory and calling the data stored in the memory.

[0137] The memory may primarily include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function, etc.; the data storage area may store data created based on the use of the mobile phone, etc. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0138] This application also provides a storage medium, which is a computer-readable storage medium. The computer program is stored in the computer-readable storage medium, and when executed by a processor, the computer program can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0139] This application also provides a computer program product, including: a computer program or instructions that, when the computer program or instructions are run on a computer, cause the computer to perform any of the above possible implementation methods.

[0140] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications are also considered to be within the scope of protection of this application.

Claims

1. A method for extracting textures from polarized and visible light images, characterized in that, include: Obtain a linear polarization degree image, a visible light image, and a reflection component image of the visible light image, wherein the linear polarization degree image and the visible light image are of the same origin; Furthermore, the linear polarization image, the visible light image, and the reflection component image are input into a trained fusion texture extraction model for processing to obtain a fusion texture map. The fused texture extraction model is obtained by training with the preliminary fused texture map as supervised ground truth. The preliminary fused texture map is obtained through the following process: for all local blocks that constitute the linear polarization image, the visible light image, and the reflection component image of the visible light image, several local blocks that meet the preset linear polarization texture confidence discrimination logic requirements are determined, and the textures of the determined several local blocks are extracted to obtain the preliminary fused texture map. The linear polarization degree texture reliability discrimination logic includes: The texture intensity of the linear polarization image in the local block is examined to see if it is significantly stronger than that of the visible light image. If so, the local block in the linear polarization image is preliminarily determined to be reliable. The texture intensity is constructed based on the image texture amplitude extracted by the Sobel operator. If the texture intensity of the linear polarization image in a local block is significantly stronger than the texture intensity of the visible light image, determine whether the texture intensity of the reflection component image in the local block is stronger than the texture intensity of the visible light image. If so, further improve the credibility of the local block in the linear polarization image. When the texture intensity of both the linear polarization degree image and the reflection component image in a local block is stronger than the texture intensity of the visible light image, it is determined whether the texture direction of the linear polarization degree image in the local block is consistent with the texture direction of the visible light image. If so, the texture of the linear polarization degree image in the local block is finally considered reliable.

2. The method for extracting textures from polarized and visible light images as described in claim 1, characterized in that, The linear polarization degree texture credibility discrimination logic is constructed based on the image texture amplitude extracted by the Sobel operator.

3. The method for extracting textures from polarized and visible light images as described in claim 1, characterized in that, The method for extracting textures by fusing polarization and visible light images further includes: The regions in the visible light image with a texture amplitude greater than zero are identified as valid texture mask regions. The linear polarization degree texture confidence discrimination logic is then executed based on these valid texture mask regions.

4. The method for extracting textures from polarized and visible light images as described in claim 3, characterized in that, The preliminary fused texture map is obtained by extracting the textures of several determined local blocks, specifically as follows: Once the linear polarization image texture in the local block is deemed reliable, the preliminary fused texture map T is extracted using the following method. f (x,y): in, This represents the texture amplitude of image type i, where i takes S0 for the visible light image, Ret for the reflection component image, and DoLP for the linear polarization degree image. The image texture amplitude after linear stretching to the [0,255] interval. The function restricts the final value to the range [0,255], where (x,y) are the pixel coordinates and Mask represents the effective texture mask area; When the linear polarization image texture in a local block is ultimately deemed unreliable, a preliminary fused texture map T is extracted using the following method. f (x,y): .

5. The method for extracting textures from polarized and visible light images as described in claim 1, characterized in that, The loss function Ltotal of the fused texture extraction model is: Among them, L char For pixel-level reconstruction loss, L grad For gradient consistency loss, L SSIM The structural similarity loss is used; the weight coefficients are set empirically as follows: , and N is the total number of pixels in each patch. This represents the model's predicted texture map. This represents the supervised ground truth, i.e., the artificially fused texture image extracted in step S2, where i represents the i-th pixel in the image, and is a constant. Set the smoothing term to 10. -6 ; and These are Sobel convolution kernels in the horizontal and vertical directions, respectively. This term, through nested Charbonnier penalty functions, achieves robust constraints on image gradient differences; K represents the number of segments into which each patch image data is divided. The total number of partial blocks in the sliding window. and They represent the first At each spatial location Local image patch.

6. The method for extracting textures from polarized and visible light images as described in claim 1, characterized in that, Texture intensity includes: average texture intensity and total texture energy; the linear polarization image, the visible light image, and the reflection component image of the visible light image are y-channel images; the method further includes: randomly sampling the training data of the fused texture extraction model.

7. An electronic device, characterized in that, The electronic device includes: a processor, and a memory coupled to the processor. The memory is used to store computer programs; The processor is configured to execute the computer program stored in the memory, so that the electronic device performs the polarization and visible light image fusion texture extraction method as described in any one of claims 1-6.

Images