Imaging method and system based on polarization imaging lens and visual collaborative optimization, and medium

Abstract

The invention provides an imaging method and system based on a polarization imaging lens and visual collaborative optimization, and a medium, and the method comprises the steps: integrating a four-way polarization filter array on the surface of an imaging sensor, and synchronously collecting a four-channel polarized light intensity image through single-frame exposure based on the imaging sensor; obtaining a three-channel RGB image, and inputting the four-channel polarized light intensity image and the three-channel RGB image into a polarization-RGB fusion neural network; polarization features and RGB features are extracted, heterogeneous information coupling optimization is carried out, and fusion features are obtained; constructing a polarization-color joint decoding network, carrying out polarization and RGB information collaborative decoding on the fusion features, training a polarization decoding process and a material classification task together based on an end-to-end joint optimization strategy, and outputting a material classification result; high-efficiency acquisition of original polarization information on a hardware level is guaranteed through the polarization filtering array; and an end-to-end joint training strategy is adopted to realize optimal decoupling and fusion of perception information.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and more specifically, to an imaging method, system, and medium based on polarization imaging lens and visual co-optimization. Background Technology

[0002] With the development of computer vision and artificial intelligence, the perception capability of imaging systems has become an increasingly crucial factor restricting performance improvement in many fields such as object detection, material recognition, and 3D reconstruction. Especially in complex lighting environments or for tasks involving the recognition of objects with similar textures / colors, traditional RGB imaging often struggles to meet the requirements of high precision and robustness. In such cases, polarization imaging is gradually becoming an effective way to enhance the perception dimension and overcome the limitations of RGB.

[0003] Polarization information reflects the direction of the electric field vector vibration of light waves in the plane perpendicular to their propagation direction. By capturing this information, key properties such as surface roughness, refractive index, and material type can be revealed. For example, the difference in polarization reflection characteristics between metals and plastics is far greater than their difference in visible light reflectance or color characteristics; therefore, the introduction of polarization information can significantly improve the reliability of material classification. However, current polarization imaging solutions on the market suffer from the following bottlenecks: The optical structure is complex: traditional polarization imaging often relies on rotating polarizers, liquid crystal modulators or multiple camera modules, resulting in large structural volume, difficulty in synchronization and high cost.

[0004] Algorithm fragmentation: Polarization information and RGB information are usually acquired by different sensors, which fails to achieve collaborative optimization of the algorithm. The fusion strategy is simple and it is difficult to explore the deep coupling characteristics of cross-modal information.

[0005] Poor real-time performance: High-precision polarization reconstruction algorithms are mostly based on complex physical modeling and large-scale convolutional networks, resulting in high runtime latency and making them difficult to deploy on edge devices or mobile platforms. Summary of the Invention

[0006] The purpose of this application is to provide an imaging method, system, and medium based on polarization imaging lens and visual co-optimization, which ensures efficient acquisition of original polarization information at the hardware level through polarization filter array; and achieves optimal decoupling and fusion of perceived information through end-to-end joint training strategy.

[0007] This application also provides an imaging method based on the coordinated optimization of polarization imaging lens and vision, including: A four-way polarization filter array is integrated onto the surface of an imaging sensor, and a four-channel polarized light intensity image is simultaneously acquired based on the imaging sensor during a single-frame exposure. Acquire a three-channel RGB image, construct a polarization-RGB fusion neural network, and input a four-channel polarization intensity image and a three-channel RGB image into the polarization-RGB fusion neural network; The polarization features of a four-channel polarized light intensity image and the RGB features of a three-channel RGB image are extracted based on a polarization-RGB fusion neural network. Based on spatial guided attention mechanism and channel fusion attention mechanism, heterogeneous information coupling optimization of polarization feature and RGB feature is performed to obtain fused feature; A polarization-color joint decoding network is constructed. Based on the polarization-color joint decoding network, polarization and RGB information are jointly decoded for fused features. Based on an end-to-end joint optimization strategy, the polarization decoding process and the material classification task are trained together to output the material classification result.

[0008] Optionally, in the imaging method based on the coordinated optimization of polarization imaging lens and vision described in the embodiments of this application, the four-way polarization filter array modeling method is as follows: Parallel sensing of polarization angle based on Malus's law, the intensity of polarized light after passing through a linear polarizer. Represented as: ; in: Total intensity of incident light; : Polarizing plate direction; The direction of light polarization; To obtain omnidirectional polarization information in a single exposure, a four-directional polarization array sampling function is constructed, as shown in the following formula: ; The trigonometric identity is obtained using the following formula: ; The sampling function of the four-directional polarization array is uniformly represented by the following formula: ; Two polarization descriptors, polarization degree and polarization angle, are defined. The formula for polarization degree is as follows: ; The formula for polarization angle is as follows: ; According to the definition of the Stokes parameter: ; Indicates total light intensity. Indicates the light intensity in the 0° polarization direction. This indicates the light intensity in the 45° polarization direction. This indicates the light intensity in the 90° polarization direction. This indicates the light intensity in the 135° polarization direction. U represents the linear polarization component. Indicates the degree of polarization, measuring the extent to which light is polarized. This represents the polarization angle; based on the Stokes parameters, four polarizer directions are arranged on the surface of the image sensor with a period of 4 pixels to obtain a four-way polarization filter array: .

[0009] Optionally, in the imaging method based on polarization imaging lens and visual co-optimization described in the embodiments of this application, the polarization-RGB fusion neural network design method is as follows: The network structure is constructed as follows: The feature extraction module extracts the geometric reflection features of the polarization image and the texture and color distribution features of the RGB image. The fusion module, based on the attention mechanism, introduces a spatially guided attention module and a channel fusion attention module to optimize the coupling of geometric reflection features with texture and color distribution features. The classification and discrimination module performs material identification and target detection based on the coupled and optimized features; Define the fusion function as follows: ; in: Polarization characteristics; RGB characteristics; Feature splicing operation; SA: Spatial Guided Attention Module; CFA: Channel Fusion Attention Module.

[0010] Optionally, in the imaging method based on the coordinated optimization of polarization imaging lens and vision described in the embodiments of this application, the method for constructing a four-way polarization filter array further includes: The polarization angles of the four-way polarizing filter include 0°, 45°, 90°, and 135°, which are used to acquire the resolution of the image sensor. Set every four pixels as a group of sampling units: , in: : Sampling pixels with a polarizer orientation of 0°; The direction is 45°; The direction is 90°; The direction is 135°; The sampling unit tiles the entire image sensor in 2×2 pixel increments to achieve spatial multiplexing encoding; the Stokes vector is constructed as follows: .

[0011] Optionally, in the imaging method based on polarization imaging lens and visual co-optimization described in the embodiments of this application, the core function of the polarization-RGB fusion neural network is expressed as: , in: Polarization image coding network; RGB image encoding network; Feature fusion module, including spatial attention and channel attention mechanisms; The following channel attention mapping mechanism is adopted: ; in: The channel attention weight vector is a vector with the same number of channels as the input feature map. Each channel corresponds to a weight value, which is used to emphasize or suppress the feature response of different channels. This represents the Sigmoid activation function, which compresses the output to the range (0, 1), and represents the importance weight of each channel; , These are the weight matrices of two fully connected layers, used to perform linear transformation and dimensionality reduction / increase on the features after global average pooling in order to learn the nonlinear relationships between channels; This represents the ReLU activation function, used to introduce non-linearity and enhance the expressive power of the model; GAP stands for Global Average Pooling, which compresses the feature map of each channel into a scalar, representing the global response strength of that channel; The input feature map is typically a three-dimensional tensor with the shape of... ,in It is the number of channels. and It is height and width; GAP indicates global average pooling; Indicates ReLU activation; It represents Sigmoid; : is the weight matrix; The final fusion features are: ; Based on the material classification task, the final discrimination loss is defined as: ; The total loss function takes the form of a joint optimization: ; in , For hyperparameter weights.

[0012] Optionally, in the imaging method based on polarization imaging lens and vision co-optimization described in the embodiments of this application, an end-to-end physical-semantic network is constructed based on an end-to-end joint optimization strategy. The physical polarization decoding task and the material recognition task are jointly constructed into a multi-task network framework, and the joint loss function of the end physical-semantic network is obtained, as shown in the following formula: , in: Physical decoding supervision; : Classification cross-entropy loss; : Polarization-semantic consistency regularization term.

[0013] The consistency term is constructed as follows: .

[0014] Secondly, embodiments of this application provide an imaging system based on the coordinated optimization of polarization imaging lens and vision. The system includes a memory and a processor. The memory includes a program for an imaging method based on the coordinated optimization of polarization imaging lens and vision. When the program for the imaging method based on the coordinated optimization of polarization imaging lens and vision is executed by the processor, it implements the following steps: A four-way polarization filter array is integrated onto the surface of an imaging sensor, and a four-channel polarized light intensity image is simultaneously acquired based on the imaging sensor during a single-frame exposure. Acquire a three-channel RGB image, construct a polarization-RGB fusion neural network, and input a four-channel polarization intensity image and a three-channel RGB image into the polarization-RGB fusion neural network; The polarization features of a four-channel polarized light intensity image and the RGB features of a three-channel RGB image are extracted based on a polarization-RGB fusion neural network. Based on spatial guided attention mechanism and channel fusion attention mechanism, heterogeneous information coupling optimization of polarization feature and RGB feature is performed to obtain fused feature; A polarization-color joint decoding network is constructed. Based on the polarization-color joint decoding network, polarization and RGB information are jointly decoded for fused features. Based on an end-to-end joint optimization strategy, the polarization decoding process and the material classification task are trained together to output the material classification result.

[0015] Optionally, in the imaging system based on polarization imaging lens and vision co-optimization described in the embodiments of this application, the four-way polarization filter array modeling method is as follows: Parallel sensing of polarization angle based on Malus's law, the intensity of polarized light after passing through a linear polarizer. Represented as: ; in: Total intensity of incident light; : Polarizing plate direction; The direction of light polarization; To obtain omnidirectional polarization information in a single exposure, a four-directional polarization array sampling function is constructed, as shown in the following formula: ; The trigonometric identity is obtained using the following formula: ; The sampling function of the four-directional polarization array is uniformly represented by the following formula: ; Two polarization descriptors, polarization degree and polarization angle, are defined. The formula for polarization degree is as follows: ; The formula for polarization angle is as follows: ; According to the definition of the Stokes parameter: ; Based on the Stokes parameters, four polarizer orientations are arranged on the surface of the image sensor with a period of 4 pixels to obtain a four-directional polarization filter array: .

[0016] Optionally, in the imaging system based on polarization imaging lens and visual co-optimization described in this application embodiment, the polarization-RGB fusion neural network design method is as follows: The network structure is constructed as follows: The feature extraction module extracts the geometric reflection features of the polarization image and the texture and color distribution features of the RGB image. The fusion module, based on the attention mechanism, introduces a spatially guided attention module and a channel fusion attention module to optimize the coupling of geometric reflection features with texture and color distribution features. The classification and discrimination module performs material identification and target detection based on the coupled and optimized features; Define the fusion function as follows: ; in: Polarization characteristics; RGB characteristics; Feature splicing operation; SA: Spatial Guided Attention Module; CFA: Channel Fusion Attention Module.

[0017] Thirdly, embodiments of this application also provide a computer-readable storage medium, which includes an imaging method program based on polarization imaging lens and vision co-optimization. When the imaging method program based on polarization imaging lens and vision co-optimization is executed by a processor, it implements the steps of the imaging method based on polarization imaging lens and vision co-optimization as described in any of the above claims.

[0018] As can be seen from the above, the imaging method, system, and medium based on polarization imaging lens and visual co-optimization provided in this application integrates a four-way polarization filter array onto the surface of an imaging sensor. Based on the imaging sensor, four-channel polarization light intensity images are simultaneously acquired during single-frame exposure. Three-channel RGB images are obtained, and the four-channel polarization light intensity images and the three-channel RGB images are input into a polarization-RGB fusion neural network. Polarization features and RGB features are extracted, and heterogeneous information coupling optimization is performed to obtain fused features. A polarization-color joint decoding network is constructed to perform collaborative decoding of polarization and RGB information on the fused features. Based on an end-to-end joint optimization strategy, the polarization decoding process and material classification task are trained together to output material classification results. The polarization filter array ensures efficient acquisition of the original polarization information at the hardware level. The end-to-end joint training strategy achieves optimal decoupling and fusion of perceived information. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A flowchart illustrating the imaging method based on the coordinated optimization of polarization imaging lens and vision provided in this application embodiment; Figure 2 A schematic diagram of the structure of an imaging system based on polarization imaging lens and vision co-optimization provided in an embodiment of this application; Figure 3 A pixel-level structural arrangement diagram of a four-directional polarization filter array for an imaging system based on polarization imaging lens and vision co-optimization provided in an embodiment of this application; Figure 4 A flowchart illustrating the Stokes parameter decoding process of an imaging system based on polarization imaging lens and vision co-optimization provided in this application embodiment; Figure 5 A structural framework diagram of the polarization-RGB fusion network (PRFN) for an imaging system based on polarization imaging lens and vision co-optimization provided in the embodiments of this application; Figure 6 This is a schematic diagram illustrating the polarization and RGB channel fusion of an imaging system based on polarization imaging lens and visual co-optimization provided in an embodiment of this application. Detailed Implementation

[0021] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0022] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0023] Please refer to Figure 1 , Figure 1 This is a flowchart of an imaging method based on the coordinated optimization of polarization imaging lens and vision, as described in some embodiments of this application. This imaging method based on the coordinated optimization of polarization imaging lens and vision is used in a terminal device and includes the following steps: S101 integrates a four-way polarization filter array onto the surface of an imaging sensor, and acquires four-channel polarized light intensity images simultaneously in a single-frame exposure based on the imaging sensor. S102, acquire a three-channel RGB image, construct a polarization-RGB fusion neural network, and input the four-channel polarization intensity image and the three-channel RGB image into the polarization-RGB fusion neural network; S103, based on a polarization-RGB fusion neural network, extracts the polarization features of a four-channel polarized light intensity image and the RGB features of a three-channel RGB image. S104, Based on the spatial guided attention mechanism and the channel fusion attention mechanism, heterogeneous information coupling optimization of polarization features and RGB features is performed to obtain fused features; S105. Construct a polarization-color joint decoding network. Based on the polarization-color joint decoding network, perform collaborative decoding of polarization and RGB information on the fused features. Based on the end-to-end joint optimization strategy, train the polarization decoding process and the material classification task together to output the material classification result.

[0024] It should be noted that the physical mapping decoding network for polarization information is designed as follows: Obtained from the optical layer , , , Using the image as input, a neural network submodule is further designed to perform fine-grained modeling of the polarization state, constructing a polarization information encoding network 𝒫, whose output is the estimated Stokes component. , , : ; The network structure employs a multi-scale feature fusion mechanism and uses Spatial Pyramid Pooling (SPP) to enhance the receptive field. A decoding supervision loss is also introduced. .

[0025] Furthermore, the polarization-RGB information collaborative decoding mechanism is as follows: Conventional polarization imaging systems and RGB systems sample independently. Due to differences in sensor response characteristics, accurate registration and feature sharing between the two systems are impossible. This invention proposes a fusion mechanism, constructing a polarization-color joint decoding network (PRFN), to achieve semantic layer alignment of heterogeneous modal data.

[0026] This network architecture is jointly modeled through the following steps: Two parallel backbone networks are used to extract RGB features respectively. With polarization characteristics ,in This represents the number of channels.

[0027] The spatial attention module extracts salient structural regions, and the spatial attention mask is defined as follows: , in: ; For the Sigmoid function; This mask is used to guide the network to focus on edges or reflective regions with prominent polarization differences.

[0028] Channel Blending Attention (CFA) is used to cross-reference the RGB and polarization channel responses, and the channel blending weights are defined as follows: , The final fusion features are:

[0029] This method fully leverages the advantages of polarization in enhancing material boundaries, while utilizing the texture semantic capabilities of RGB to achieve deep complementarity between the two heterogeneous information types.

[0030] According to an embodiment of the present invention, the modeling method for a four-way polarizing filter array is as follows: Parallel sensing of polarization angle based on Malus's law, the intensity of polarized light after passing through a linear polarizer. Represented as: ; in: Total intensity of incident light; : Polarizing plate direction; The direction of light polarization; To obtain omnidirectional polarization information in a single exposure, a four-directional polarization array sampling function is constructed, as shown in the following formula: ; The trigonometric identity is obtained using the following formula: ; The sampling function of the four-directional polarization array is uniformly represented by the following formula: ; Define two polarization descriptors: degree of polarization and polarization angle. The formula for degree of polarization is as follows: ; The formula for polarization angle is as follows: ; According to the definition of the Stokes parameter: ; Indicates total light intensity. Indicates the light intensity in the 0° polarization direction. This indicates the light intensity in the 45° polarization direction. This indicates the light intensity in the 90° polarization direction. This indicates the light intensity in the 135° polarization direction. U represents the linear polarization component. Indicates the degree of polarization, measuring the extent to which light is polarized. This represents the polarization angle; based on the Stokes parameters, four polarizer directions are arranged on the surface of the image sensor with a period of 4 pixels to obtain a four-way polarization filter array: .

[0031] It should be noted that the design and arrangement of the four-way polarizing filter array are as follows: Traditional polarization imaging systems typically employ rotating linear polarizers or liquid crystal variable polarizers (LCPRs), requiring multiple shots or time-division scanning to sample different polarization angles. This approach severely limits the real-time performance of the imaging system and is susceptible to interference from target motion or vibration, leading to the loss of inter-frame information.

[0032] This invention introduces a four-directional polarization filter array (FPFA) to periodically lay out four polarization angle filters (0°, 45°, 90°, and 135°) on the image sensor array, enabling multi-angle polarization information sampling within a single frame. The array arrangement period is 2×2, as shown below: | 0° | 45° | | 90° | 135° |; Each 2×2 pixel unit is called a "Micro Polar Unit" (MPU), and the entire surface of the image sensor is periodically covered. This allows for spatial encoding of the polarization angle distribution of incident light without the need for mechanically rotating polarizers or employing complex control logic. Its advantages are as follows: Achieve high frame rate single-frame capture; Eliminate motion artifacts caused by multi-frame synthesis; Reduce structural complexity and power consumption.

[0033] The design theoretically follows Malus's law and is implemented with a pixel-level hardware structure. The corresponding Stokes parameters are calculated as follows: , Furthermore, the polarization state distribution of fully polarized light can be reconstructed using four-directional light intensity information (i.e., , ): ; This microstructure array integration method provides pixel-level hardware parallel support for polarization information sensing.

[0034] According to an embodiment of the present invention, the polarization-RGB fusion neural network design method is as follows: The network structure is constructed as follows: The feature extraction module extracts the geometric reflection features of the polarization image and the texture and color distribution features of the RGB image. The fusion module, based on the attention mechanism, introduces a spatially guided attention module and a channel fusion attention module to optimize the coupling of geometric reflection features with texture and color distribution features. The classification and discrimination module performs material identification and target detection based on the coupled and optimized features; Define the fusion function as follows: ; in: Polarization characteristics; RGB characteristics; Feature splicing operation; SA: Spatial Guided Attention Module; CFA: Channel Fusion Attention Module.

[0035] According to an embodiment of the present invention, the method for constructing a four-way polarizing filter array further includes: The polarization angles of the four-way polarizing filter include 0°, 45°, 90°, and 135°, which are used to acquire the resolution of the image sensor. Set every four pixels as a group of sampling units: ; in: : Sampling pixels with a polarizer orientation of 0°; The direction is 45°; The direction is 90°; The direction is 135°; The sampling unit tiles the entire image sensor in 2×2 pixel increments to achieve spatial multiplexing encoding; the Stokes vector is constructed as follows: .

[0036] It should be noted that this vector describes the state of fully or partially polarized light, and its decoding process can be completed through hardware preprocessing or neural network decoupling.

[0037] To ensure stability during high frame rate shooting, a Local Block Registration (LBR) strategy is introduced to fully correct geometric misalignments caused by lens distortion and micro-vibrations. A local registration mapping function is defined for each polarization pixel partition: , in This represents the subpixel registration function. , These represent local pixel offsets. This function is optimized using the maximum correlation criterion for edge features and is defined as follows: ; Where NCC represents the normalized cross-correlation index.

[0038] According to an embodiment of the present invention, the core function of the polarization-RGB fusion neural network is expressed as follows: , in: Polarization image coding network; RGB image encoding network; Feature fusion module, including spatial attention and channel attention mechanisms; The following channel attention mapping mechanism is adopted: ; in: GAP indicates global average pooling; Indicates ReLU activation; It represents Sigmoid; : is the weight matrix; The final fusion features are: ; Based on the material classification task, the final discrimination loss is defined as: ; The total loss function takes the form of a joint optimization: ; in , For hyperparameter weights.

[0039] According to an embodiment of the present invention, an end-to-end physical-semantic network is constructed based on an end-to-end joint optimization strategy. The physical polarization decoding task and the material recognition task are jointly constructed into a multi-task network framework, and the joint loss function of the end-to-end physical-semantic network is obtained, as shown in the following formula: , in: Physical decoding supervision; : Classification cross-entropy loss; : Polarization-semantic consistency regularization term.

[0040] The consistency term is constructed as follows: ; This represents the gradient operator, which calculates the changes in the feature map along its spatial dimensions. Indicates the first Polarization feature map of each sample Indicates the first Classification feature map of each sample, The square of the L2 norm is used to measure the difference between two gradient maps. This represents the summation over all samples, used for batch training.

[0041] That is, the gradient directions of polarization and classification pathway features are aligned and constrained in the feature space to guide the network to maintain semantic discriminability of polarization gradient boundaries.

[0042] This mechanism creates a feedback loop between the decoding of polarization physical quantities and the downstream classification and discrimination of targets, thereby achieving end-to-end network collaborative optimization.

[0043] It should be noted that the end-to-end implementation process for material differentiation is as follows: The input image is acquired by a hardware array to obtain a four-channel polarization map. and synchronously collected Subsequently, they are processed through their respective network encoding paths and fused into joint features. Finally, the material type label is output through the category header. The entire process can be represented by the following function combination: ; The predicted material classification result, i.e., the category label or probability distribution output by the model; The classification head, typically one or more fully connected layers, is used to map the fused features to the class space. Feature fusion module, which includes spatial attention (SA) and channel attention (CFA) mechanisms, for fusing polarization and RGB features; Polarization image coding network for processing four-channel polarization images Extracting polarization features ; RGB image encoding network for encoding three-channel RGB images Extracting RGB features ; Four-channel polarized light intensity images, acquired by a four-way polarizing filter array, include... The intensity of polarized light in four directions; Three-channel RGB images were acquired synchronously by the same imaging system.

[0044] This process achieves a complete closed loop from hardware sampling to neural network optimization, effectively improving the accuracy and generalization ability of object discrimination in multiple scenarios (such as strong reflection and low contrast materials).

[0045] According to embodiments of the present invention, the method further includes constructing a high-precision separation model of metal-plastic materials: Because metals and plastics often have similar colors and textures in the RGB color space, traditional methods are prone to misidentification. This invention utilizes the principle of polarization reflection: the reflection intensity of polarized light from metal surfaces typically exhibits strong directionality, while plastic materials, due to their lower refractive index, display high-frequency noise dispersion as their polarization characteristic. Therefore, this invention distinguishes materials by constructing the following polarization frequency response function: , in: Polarization frequency response value: This value measures the energy intensity of a material in high-frequency polarization information; metals typically have higher energy intensity. value; Fourier transform, transforming the polarization image Transformation from the spatial domain to the frequency domain; Polarization degree image, calculated from a four-channel polarization image, represents the degree of polarization of each pixel; : Coordinates in the frequency domain, corresponding to the position of the image in the frequency domain; The energy spectrum in the frequency domain represents the energy intensity of that frequency component. Indicator function, when the frequency radius is greater than a threshold The value is 1 if the condition is met, and 0 otherwise. Used to filter high-frequency components. Frequency threshold, used to distinguish between high-frequency and low-frequency regions, is usually set based on experiments or prior knowledge; Fourier transform; For high-frequency region mask; This is the frequency threshold.

[0046] The experiment revealed that the high-frequency polarization response of metallic materials was significantly higher than that of plastics. Based on this, a binary classification supervision signal was constructed and introduced into the network training process as an auxiliary supervision to further improve the model's discrimination ability.

[0047] According to embodiments of the present invention, experimental verification and system deployment include: To verify the effectiveness of the proposed method for co-optimizing polarization imaging lenses and algorithms, we conducted system experiments on multiple datasets and actual sampling systems, including the following aspects: Polarization information reconstruction accuracy; Accuracy of material identification tasks; Accuracy of metal-plastic classification task; Inference efficiency and deployment performance.

[0048] The dataset and hardware platform include: a dedicated acquisition system equipped with the FPFA array proposed in this invention, using a Sony IMX series sensor combined with a custom polarizing filter array, achieving a resolution of 1920×1080 and supporting real-time 60FPS shooting. A total of 20,000 images were acquired, encompassing various materials including metal, plastic, ceramic, and glass, and each image was labeled with the following tags: RGB image (3 channels); Polarization image (4 channels: ); Material category label; Region segmentation mask.

[0049] As a baseline for comparison, the following publicly available dataset is introduced: POL-MV (Polarization Material Recognition Dataset); FMD (Flickr Material Database); RGB-Only Baseline.

[0050] All models were trained on an NVIDIA RTX 3090 GPU platform using the PyTorch framework and the Adam optimizer, with a learning rate set to [value missing]. The batch size is 32, and the training rounds are 100.

[0051] Polarization decoding accuracy assessment includes: Firstly, in the polarization reconstruction task, this application compares with the traditional Stokes inversion method, using mean squared error (MSE) and structural similarity (SSIM) as evaluation metrics, as shown in Table 1. Table 1 .

[0052] The results show that the jointly trained polarization decoding network can significantly improve the decoding accuracy, especially in the boundary transition region where the error is significantly reduced. This is due to its use of spatial attention mechanism to focus on regions with drastic polarization changes.

[0053] Material recognition accuracy comparison includes: In the material classification task, the results were compared with several mainstream models, including ResNet50 (RGB-only), PolarNet (polarization + RGB early fusion), and DeepLabV3+. The results are shown in Table 2. Table 2 .

[0054] It is evident that the polarization-RGB fusion mechanism proposed in this application significantly improves the model's discrimination ability in fine-grained recognition of multiple materials, especially in the blurred area of ​​the metal-plastic boundary.

[0055] The metal-plastic binary classification experiment includes: By focusing on the misclassification of metals and plastics, and using precision, recall, and F1 score as indicators, the analysis results are shown in Table 3. Table 3 .

[0056] Analysis revealed that the high-frequency region in the polarization diagram greatly contributes to the brightening of metal edges, thereby enhancing the classifier's response to differences in optical properties, thanks to the polarization spectral response function. Guided high-frequency feature fusion.

[0057] From hardware, encoding, decoding, feature fusion, target discrimination to final deployment, a complete polarization-sensing vision system was constructed. All core variables, structures, and formulas form the following logical closed loop: Input signal chain: , , , , These represent light intensity images acquired through polarizing filters at four directions: 0°, 45°, 90°, and 135°. These are the raw data directly acquired at the hardware level. , , : This is the Stokes parameter, used to describe the polarization state of light; Total light intensity ( ); The difference between the horizontal and vertical polarization components ( ); The difference between the 45° and 135° polarization components ( ); Polarization states (DoP, AoP): DoP (Degree of Polarization): Degree of polarization, indicating the degree to which light is polarized; AoP (Angle of Polarization): Polarization angle, indicating the polarization direction.

[0058] Encoder-decoder fusion chain: , Stokes parameter reconstruction loss is used to supervise the accuracy of physical decoding of polarization information. The mean square error (MSE) between the predicted Stokes parameters and the true values ​​is calculated. Classification loss, typically cross-entropy loss, is used to supervise the accuracy of material classification tasks. Consistency regularization term, used to constrain the consistency between polarization features and classification features in gradient space, enhancing the alignment between semantic and physical information.

[0059] The specific form is as follows: , That is, for each sample Calculate the L2 distance between the polarization feature gradient and the classification feature gradient; , , Hyperparameter weights are used to balance the importance of the three losses.

[0060] Physical + Semantic Collaborative Mechanism: Gradient direction regularization constraint This enables the polarization edge enhancement to transmit feature weights to the classification head, making the polarization structure the dominant element in semantic decoding.

[0061] In summary, this invention is the first in the field of polarization imaging to propose a system-level collaborative optimization method from hardware structure to algorithm architecture, and has made several technological breakthroughs, especially in FPFA structure design, Stokes parameter estimation network construction, polarization-RGB feature collaborative mechanism and physical-semantic multi-task learning.

[0062] Through integrated design, the system has the following core advantages: High real-time performance: Hardware-level polarization coding does not depend on any time-division or multi-frame structure; High robustness: The fusion attention mechanism effectively bridges intermodal differences; Strong ability to differentiate materials: especially suitable for materials with significant differences in reflectance, such as metals and plastics; Highly deployable: The model can run in real time on mobile devices and embedded platforms.

[0063] Secondly, embodiments of this application provide an imaging system based on the coordinated optimization of polarization imaging lens and vision. The system includes a memory and a processor. The memory includes a program for an imaging method based on the coordinated optimization of polarization imaging lens and vision. When the program for the imaging method based on the coordinated optimization of polarization imaging lens and vision is executed by the processor, it performs the following steps: A four-way polarization filter array is integrated onto the surface of an imaging sensor, and a four-channel polarized light intensity image is simultaneously acquired based on the imaging sensor during a single-frame exposure. Acquire a three-channel RGB image, construct a polarization-RGB fusion neural network, and input a four-channel polarization intensity image and a three-channel RGB image into the polarization-RGB fusion neural network; The polarization features of a four-channel polarized light intensity image and the RGB features of a three-channel RGB image are extracted based on a polarization-RGB fusion neural network. Based on spatial guided attention mechanism and channel fusion attention mechanism, heterogeneous information coupling optimization of polarization feature and RGB feature is performed to obtain fused feature; A polarization-color joint decoding network is constructed. Based on the polarization-color joint decoding network, polarization and RGB information are jointly decoded for fused features. Based on an end-to-end joint optimization strategy, the polarization decoding process and the material classification task are trained together to output the material classification result.

[0064] According to an embodiment of the present invention, the modeling method for a four-way polarizing filter array is as follows: Parallel sensing of polarization angle based on Malus's law, the intensity of polarized light after passing through a linear polarizer. Represented as: ; in: Total intensity of incident light; : Polarizing plate direction; The direction of light polarization; To obtain omnidirectional polarization information in a single exposure, a four-directional polarization array sampling function is constructed, as shown in the following formula: ; The trigonometric identity is obtained using the following formula: ; The sampling function of the four-directional polarization array is uniformly represented by the following formula: ; Define two polarization descriptors: degree of polarization and polarization angle. The formula for degree of polarization is as follows: ; The formula for polarization angle is as follows: ; According to the definition of the Stokes parameter: ; Based on the Stokes parameters, four polarizer orientations are arranged on the surface of the image sensor with a period of 4 pixels to obtain a four-directional polarization filter array: .

[0065] According to an embodiment of the present invention, the polarization-RGB fusion neural network design method is as follows: The network structure is constructed as follows: The feature extraction module extracts the geometric reflection features of the polarization image and the texture and color distribution features of the RGB image. The fusion module, based on the attention mechanism, introduces a spatially guided attention module and a channel fusion attention module to optimize the coupling of geometric reflection features with texture and color distribution features. The classification and discrimination module performs material identification and target detection based on the coupled and optimized features; Define the fusion function as follows: , in: Polarization characteristics; RGB characteristics; Feature splicing operation; SA: Spatial Guided Attention Module; CFA: Channel Fusion Attention Module.

[0066] A third aspect of the present invention provides a computer-readable storage medium including an imaging method program based on polarization imaging lens and vision co-optimization, wherein when the imaging method program based on polarization imaging lens and vision co-optimization is executed by a processor, it implements the steps of the imaging method based on polarization imaging lens and vision co-optimization as described above.

[0067] This invention discloses an imaging method, system, and medium based on polarization imaging lens and visual collaborative optimization. It integrates a four-way polarization filter array onto the surface of an imaging sensor, simultaneously acquiring four-channel polarization intensity images during single-frame exposure. A three-channel RGB image is then acquired, and the four-channel polarization intensity image and the three-channel RGB image are input into a polarization-RGB fusion neural network. Polarization and RGB features are extracted and heterogeneous information coupling optimization is performed to obtain fused features. A polarization-color joint decoding network is constructed to collaboratively decode the polarization and RGB information of the fused features. Based on an end-to-end joint optimization strategy, the polarization decoding process is trained together with a material classification task, outputting material classification results. The polarization filter array ensures efficient acquisition of original polarization information at the hardware level. The end-to-end joint training strategy achieves optimal decoupling and fusion of perceived information.

[0068] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0069] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0070] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0071] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0072] Alternatively, if the integrated units of the present invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

Claims

1. An imaging method based on polarization imaging lens and visual collaborative optimization, characterized in that, The application comprises the following steps: Integrating a four-direction polarization filter array on the surface of an imaging sensor to synchronously collect four-channel polarization intensity images based on single-frame exposure of the imaging sensor; Obtaining a three-channel RGB image, constructing a polarization-RGB fusion neural network, and inputting the four-channel polarization intensity image and the three-channel RGB image into the polarization-RGB fusion neural network; Extracting polarization features of the four-channel polarization intensity image and RGB features of the three-channel RGB image based on the polarization-RGB fusion neural network, Optimizing the polarization features and the RGB features through heterogeneous information coupling based on a spatial guidance attention mechanism and a channel fusion attention mechanism to obtain fusion features; Constructing a polarization-color joint decoding network, decoding the fusion features through polarization and RGB information based on the polarization-color joint decoding network, and outputting a material classification result through joint training of the polarization decoding process and the material classification task based on an end-to-end joint optimization strategy.

2. The imaging method based on polarization imaging lens and visual collaborative optimization according to claim 1, characterized in that, The modeling method of the four-direction polarization filter array is as follows: Based on Malus law, the polarization angle is perceived in parallel, and the light intensity after the polarizer is represented as: ； wherein: : total intensity of the incident light; : polarizing plate direction; : polarization direction of light; Obtaining omnidirectional polarization information under single exposure, constructing a four-direction polarization array sampling function, and the formula is as follows: ； Obtaining a triangular identity, and the formula is as follows: ； Uniformly representing the four-direction polarization array sampling function, and the formula is as follows: ； Defining two polarization descriptors of the degree of polarization and the polarization angle, and the formula of the degree of polarization is as follows: ； The formula of the polarization angle is as follows: ； Wherein, the Stokes parameter is defined as follows: ； I total represents the total light intensity, I0 represents the light intensity of the 0° polarization direction, I45 represents the light intensity of the 45° polarization direction, I90 represents the light intensity of the 90° polarization direction, I135 represents the light intensity of the 135° polarization direction, and U represents the linear polarization component, P represents the degree of polarization, which measures the degree of polarization of light, P represents the polarization angle; Based on the Stokes parameters, four kinds of polarizing plate directions are arranged on the image sensor surface in a period of 4 pixels to obtain a four-way polarizing filter array: .

3. The imaging method based on polarization imaging lens and visual collaborative optimization according to claim 2, characterized in that, The design method of the polarization-RGB fusion neural network is as follows: Constructing a network result, and the network structure comprises: A feature extraction module, which extracts geometric reflection features of a polarization image and texture and color distribution features of an RGB image based on the feature extraction module; A fusion module, which introduces a spatial guidance attention module and a channel fusion attention module based on an attention mechanism to couple and optimize the geometric reflection features and the texture and color distribution features; A classification and discrimination module, which performs material recognition and target detection on the coupled and optimized features based on the classification and discrimination module; Defining a fusion function, and the formula is as follows: ， Wherein: : polarization characteristics; : RGB features; : feature stitching operation; SA: spatial guidance attention module; CFA: channel fusion attention module.

4. The imaging method based on polarization imaging lens and visual collaborative optimization according to claim 3, characterized in that, The four-direction polarization filter array construction method further comprises: The polarization angles of the four-way polarization filter include 0°, 45°, 90°, and 135°, and the resolution of the image sensor is obtained Each four pixel units are set as a group of sampling units: ； Wherein: : by the sampling pixel with polarizer direction of 0°; : direction is 45°; : direction is 90°; : direction is 135°; The sampling unit is periodically tiled on the entire photosensitive chip every 2*2 pixels to realize spatial multiplexing coding; and the Stokes vector is constructed as follows: 。 5. The imaging method based on polarization imaging lens and visual collaborative optimization according to claim 4, characterized in that, The core function of the polarization-RGB fusion neural network is represented as follows: ， Wherein: : polarized image encoding network; : RGB image encoding network; : a feature fusion module containing spatial attention and channel attention mechanisms; The following channel attention mapping mechanism is adopted: ； Wherein: denotes a channel attention weight vector, which is a vector with the same number of channels as the input feature map, and each channel corresponds to a weight value, which is used to emphasize or suppress the feature responses of different channels; denotes a sigmoid activation function that compresses the output between (0, 1) and represents the importance weight of each channel; , are weight matrices of two fully connected layers, used to linearly transform and reduce / dimension the features after global average pooling to learn the nonlinear relationship between channels; denotes a ReLU activation function, which is used to introduce nonlinearity and enhance the expressiveness of the model; GAP represents global average pooling, which compresses the feature map of each channel into a scalar representing the global response intensity of the channel; F represents the input feature map, which is usually a three-dimensional tensor with shape where C is the number of channels, H and W are the height and width; GAP represents global average pooling; denotes a ReLU activation; Sigmoid; , : is a weight matrix; The final fusion features are as follows: ； In combination with the material classification task, the final discrimination loss is defined as follows: ； The total loss function is in a joint optimization form: ； wherein , are hyperparameter weights.

6. The imaging method based on polarization imaging lens and visual collaborative optimization according to claim 5, characterized in that, Based on the end-to-end joint optimization strategy, an end-to-end physical-semantic network is constructed, the physical polarization decoding task and the material recognition task are jointly constructed into a multi-task network framework, the joint loss function of the end physical-semantic network is obtained, and the formula is as follows: ， Wherein: : physical decoding supervisor; : categorical cross-entropy loss; : polarization-semantic consistency regularizer; The consistency term is constructed as follows: 。 7. An imaging system based on polarization imaging lens and visual collaborative optimization, characterized in that, The system comprises a memory and a processor, the memory comprising a program of a polarization imaging lens and visual collaborative optimization based imaging method, the program of the polarization imaging lens and visual collaborative optimization based imaging method being executed by the processor to implement the following steps: Integrating a four-way polarization filter array on the surface of an imaging sensor, synchronously acquiring four-channel polarization light intensity images based on the imaging sensor in single-frame exposure; Obtaining a three-channel RGB image, constructing a polarization-RGB fusion neural network, and inputting the four-channel polarization light intensity images and the three-channel RGB image into the polarization-RGB fusion neural network; Extracting polarization features of the four-channel polarization light intensity images and RGB features of the three-channel RGB image based on the polarization-RGB fusion neural network, Optimizing the polarization features and the RGB features based on a spatially guided attention mechanism and a channel fusion attention mechanism to obtain fusion features; Constructing a polarization-color joint decoding network, collaboratively decoding the fusion features based on the polarization-color joint decoding network, training the polarization decoding process and the material classification task based on an end-to-end joint optimization strategy, and outputting a material classification result.

8. The imaging system based on polarization imaging lens and visual co-optimization according to claim 7, characterized in that, The modeling method of the four-way polarization filter array is as follows: Based on Malus law, the polarization angle is perceived in parallel, and the light intensity after the polarizer is expressed as: ； wherein: : total intensity of the incident light; : polarizing plate direction; : polarization direction of light; Obtaining omnidirectional polarization information in single exposure, constructing a four-way polarization array sampling function, and the formula is as follows: ； Obtaining a triangular identity, and the formula is as follows: ； Uniformly representing the four-way polarization array sampling function, and the formula is as follows: ； Defining two polarization descriptors of the degree of polarization and the polarization angle, and the formula of the degree of polarization is as follows: ； The formula of the polarization angle is as follows: ； Wherein, the Stokes parameters are defined as follows: ； represents the total light intensity, represents the light intensity of the 0° polarization direction, represents the light intensity of the 45° polarization direction, represents the light intensity of the 90° polarization direction, represents the light intensity of the 135° polarization direction, U represents the linear polarization component, represents the degree of polarization, which measures the degree of polarization of light, represents the polarization angle; Arranging four polarization plate directions according to the Stokes parameters on the image sensor surface with a period of 4 pixels to obtain a four-way polarization filter array: 。 9. The imaging system based on polarization imaging lens and visual co-optimization according to claim 8, characterized in that, The design method of the polarization-RGB fusion neural network is as follows: Constructing a network result, and the network structure comprises: A feature extraction module, which extracts geometric reflection features of a polarization image and texture and color distribution features of an RGB image based on the feature extraction module; A fusion module, which introduces a spatially guided attention module and a channel fusion attention module based on an attention mechanism to optimize the geometric reflection features and the texture and color distribution features; A classification and discrimination module, which performs material recognition and target detection on the optimized features based on the classification and discrimination module; Defining a fusion function, and the formula is as follows: ； Wherein: : polarization characteristics; : RGB features; : feature stitching operation; SA: spatially guided attention module; CFA: channel fusion attention module.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium comprises a program of a polarization imaging lens and visual collaborative optimization based imaging method, and the program of the polarization imaging lens and visual collaborative optimization based imaging method is executed by the processor to implement the steps of the polarization imaging lens and visual collaborative optimization based imaging method according to any one of claims 1 to 6.

Images