Full-polarization image restoration method based on improved sub-pixel segmentation and related device

Abstract

The application relates to the technical field of image processing, in particular to a full-polarization image recovery method based on improved sub-pixel segmentation and related equipment, which comprises the following steps: acquiring image data collected by a polarization camera, the image data containing multiple sub-pixels; determining a known region and three prediction regions of the sub-pixels; selecting one known region from two sub-pixels adjacent to the prediction regions and recording the same kind of polarization information; calculating the absolute value of the difference between the gray mean values of the two known regions to obtain the absolute difference value corresponding to the prediction region; determining a judgment threshold based on the direction of the prediction region in the sub-pixel, and then determining the polarization information; obtaining the polarization image corresponding to the prediction region based on the polarization information; combining the polarization images corresponding to each region in the sub-pixel to obtain the polarization image corresponding to the sub-pixel; and combining the polarization images corresponding to multiple sub-pixels to obtain the polarization image corresponding to the image data; and the application effectively improves the accuracy and stability of full-polarization image recovery.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method and related equipment for full polarization image restoration based on improved subpixel segmentation. Background Technology

[0002] Although existing technologies can restore polarization information acquired by focal plane polarization imaging systems to some extent, they still have many shortcomings. For example, traditional image polarization information restoration is based on sub-pixel recovery (the average gray level of a sub-pixel is defined as the gray level of the entire sub-pixel, so only one polarization information can be read from each sub-pixel), which greatly reduces image resolution and loses much polarization information. Furthermore, traditional restoration methods predict missing polarization information through simple interpolation, which not only blurs image boundaries but may also produce abnormal polarization information. Summary of the Invention

[0003] In view of this, the purpose of this invention is to provide a fully polarized image restoration method and related equipment based on improved subpixel segmentation, so as to solve one or more technical problems existing in the prior art, and at least provide a beneficial option or create conditions.

[0004] On one hand, embodiments of the present invention provide a method for restoring fully polarized images based on improved sub-pixel segmentation, the method comprising the following steps:

[0005] Image data acquired by a polarization camera is obtained. The image data contains multiple sub-pixels. Each sub-pixel is divided into four orthogonal regions. Each region of the sub-pixel is adjacent to a sub-pixel that records different polarization information.

[0006] The known region and the predicted region of the sub-pixel are determined. One known region is selected from each of the two sub-pixels adjacent to the predicted region that record the same polarization information, thus obtaining two known regions corresponding to the predicted region. The known region is the region that records known polarization information, and the predicted region is the region whose polarization information needs to be predicted. The distance between the predicted region and the known region in the two sub-pixels is different.

[0007] The grayscale mean values ​​of the two known regions are obtained respectively, and the absolute value of the difference between the grayscale mean values ​​of the two known regions is calculated to obtain the absolute difference value corresponding to the predicted region.

[0008] Determine the orientation of the predicted region in the sub-pixel, and determine the judgment threshold corresponding to the predicted region based on the orientation of the predicted region in the sub-pixel;

[0009] Polarization information is determined from two selected known regions based on the absolute mean and judgment threshold corresponding to the prediction region, and the prediction region is predicted based on the polarization information.

[0010] The polarization image corresponding to the known region in the sub-pixel is combined with the polarization images corresponding to the three predicted regions to obtain the polarization image corresponding to the sub-pixel. The polarization images corresponding to multiple sub-pixels are combined to obtain the polarization image corresponding to the image data.

[0011] Optionally, the average grayscale value is obtained in the following way:

[0012] Read the grayscale values ​​of multiple n×n regions within the region, and take the average of the grayscale values ​​of the multiple n×n regions as the grayscale mean of the region.

[0013] Optionally, the determination threshold corresponding to the predicted region includes a first threshold in the horizontal or vertical direction, and a second threshold in the ±45° direction. Determining the determination threshold corresponding to the predicted region based on its orientation within the sub-pixel includes:

[0014] If the direction of the predicted region in the sub-pixel is horizontal or vertical, then the determination threshold is a first threshold.

[0015] If the predicted region is oriented at ±45° within the sub-pixel, then the determination threshold is a second threshold; wherein the formula for calculating the second threshold is: JT1 is the first threshold, and JT2 is the second threshold.

[0016] Optionally, determining polarization information from two selected known regions based on the absolute mean and a decision threshold corresponding to the prediction region, and predicting the prediction region based on the polarization information, includes:

[0017] If the absolute difference is determined to be less than the determination threshold, then the polarization information of the two selected known regions is used for fitting, and the polarization information of the predicted region is predicted based on the fitting curve obtained by fitting.

[0018] If the absolute difference is greater than the threshold, the polarization information of the known region closer to the prediction region is used to predict the prediction region.

[0019] Optionally, the step of using polarization information from a known region closer to the prediction region to predict the prediction region includes:

[0020] Select a first sub-region and a second sub-region from the known regions that are closer to the prediction region; wherein, the first sub-region is the region closest to the prediction region among the known regions that are closer to the prediction region, and the second sub-region is the region farthest from the prediction region among the known regions that are closer to the prediction region.

[0021] Read the average gray value of the first sub-region and the average gray value of the second sub-region respectively, calculate the absolute value of the difference between the average gray value of the first sub-region and the average gray value of the second sub-region, and obtain the absolute difference of the known region.

[0022] Based on the orientation of the predicted region in the sub-pixel, a decision threshold corresponding to the known region is determined. Based on the absolute difference of the known region and the corresponding decision threshold, polarization information is determined from the first sub-region and the second sub-region. Based on the polarization information, the predicted region is predicted.

[0023] Optionally, the determination threshold corresponding to the known region includes a third threshold in the horizontal or vertical direction, and a fourth threshold in the ±45° direction. Determining the determination threshold corresponding to the known region based on the orientation of the predicted region within the sub-pixel includes:

[0024] If the direction of the predicted region in the sub-pixel is horizontal or vertical, then the judgment threshold corresponding to the known region is determined as the third threshold; wherein, the calculation formula of the third threshold is: JF1=JT1×EAGE / (4×D-3×EAGE): where JF1 is the third threshold, JT1 is the first threshold, D is the side length of the sub-pixel, and EAGE is the maximum value among the distances of each pixel in the sub-pixel from the center point of the sub-pixel;

[0025] If the predicted region is oriented at ±45° within the sub-pixel, then the determination threshold corresponding to the known region is determined as the fourth threshold; wherein, the formula for calculating the fourth threshold is: .

[0026] Optionally, determining polarization information from the first sub-region and the second sub-region based on the absolute difference of the known region and the corresponding decision threshold, and predicting the prediction region based on the polarization information, includes:

[0027] If the absolute difference does not exceed the judgment threshold corresponding to the known region, then the polarization information of the first sub-region and the polarization information of the second sub-region are used to perform linear fitting, and the polarization information of the predicted region is predicted based on the fitting curve obtained by fitting.

[0028] If the absolute difference is greater than the judgment threshold corresponding to the known region, then the polarization information of the first sub-region is assigned to the predicted region.

[0029] On the other hand, embodiments of the present invention provide a fully polarized image restoration apparatus based on improved sub-pixel segmentation, comprising:

[0030] The first module is used to acquire image data collected by a polarization camera. The image data contains multiple sub-pixels, each of which is divided into four orthogonal regions. Each region of the sub-pixel is adjacent to a sub-pixel that records different polarization information.

[0031] The second module is used to determine the known region and the predicted region of the sub-pixel. It selects one known region from two sub-pixels adjacent to the predicted region that record the same polarization information, thus obtaining two known regions corresponding to the predicted region. The known region is the region that records known polarization information, and the predicted region is the region whose polarization information needs to be predicted. The distance between the predicted region and the known regions in the two sub-pixels is different.

[0032] The third module is used to obtain the grayscale mean values ​​of the two known regions respectively, calculate the absolute value of the difference between the grayscale mean values ​​of the two known regions, and obtain the absolute difference value corresponding to the predicted region.

[0033] The fourth module is used to determine the direction of the predicted region in the sub-pixel, and to determine the judgment threshold corresponding to the predicted region based on the direction of the predicted region in the sub-pixel;

[0034] The fifth module is used to determine polarization information from two selected known regions based on the absolute mean and a judgment threshold corresponding to the prediction region, and to predict the prediction region based on the polarization information.

[0035] The sixth module is used to combine the polarization image corresponding to the known region in the sub-pixel and the polarization images corresponding to the three predicted regions to obtain the polarization image corresponding to the sub-pixel, and to combine multiple polarization images corresponding to the sub-pixel to obtain the polarization image corresponding to the image data.

[0036] On the other hand, embodiments of the present invention provide an electronic device, including:

[0037] At least one processor;

[0038] At least one memory for storing at least one program;

[0039] When the at least one program is executed by the at least one processor, the at least one processor performs the method described above.

[0040] On the other hand, embodiments of the present invention provide a computer-readable storage medium storing a processor-executable program, which, when executed by a processor, is used to perform the above-described method.

[0041] The embodiments of this invention include the following beneficial effects: This embodiment determines the known region and the predicted region of the sub-pixel. From two sub-pixels adjacent to the predicted region that record the same polarization information, a known region is selected to obtain two known regions corresponding to the predicted region. Then, the absolute value of the difference between the grayscale mean of the known region and the grayscale mean of the predicted region is calculated. Based on the comparison result of this absolute difference with a judgment threshold, not only is the utilization rate of polarization information improved, but the resolution of the polarized image is also increased, making the final image closer to the actual value. Accurate determination of the polarization information of the predicted region effectively improves the accuracy and stability of fully polarized image restoration. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a schematic flowchart of a fully polarized image restoration method based on improved sub-pixel segmentation provided in an embodiment of the present invention;

[0044] Figure 2 This is a schematic diagram of a known region selected in related technologies;

[0045] Figure 3 This is a schematic diagram of selecting a known region in an embodiment of the present invention;

[0046] Figure 4a This is an example diagram of selecting a known region for polarization information 2 when the absolute difference does not exceed the judgment threshold in an embodiment of the present invention;

[0047] Figure 4b This is an example diagram of selecting a known region for polarization information 2 when the absolute difference is greater than the judgment threshold in an embodiment of the present invention;

[0048] Figure 5a This is an example diagram of selecting a known region for polarization information 3 when the absolute difference does not exceed the judgment threshold in an embodiment of the present invention;

[0049] Figure 5b This is an example diagram of selecting a known region for polarization information 3 when the absolute difference is greater than the judgment threshold in an embodiment of the present invention;

[0050] Figure 6a This is an example diagram of selecting a known region for polarization information 4 when the absolute difference does not exceed the judgment threshold in an embodiment of the present invention;

[0051] Figure 6b This is an example diagram of selecting a known region for polarization information 4 when the absolute difference is greater than the judgment threshold in an embodiment of the present invention;

[0052] Figure 7 This is an example diagram of predicting the prediction region when the absolute difference does not exceed the judgment threshold in an embodiment of the present invention;

[0053] Figure 8 This is an example diagram of predicting the prediction region when the absolute difference is greater than the judgment threshold in an embodiment of the present invention;

[0054] Figure 9 This is a schematic diagram of each sub-pixel sampling point in an embodiment of the present invention;

[0055] Figure 10 This is a schematic diagram of the final combination in an embodiment of the present invention;

[0056] Figure 11 This is an imaging image from an embodiment of the present invention;

[0057] Figure 12 It is an image obtained by using bilinear interpolation in related technologies;

[0058] Figure 13 This is an architectural diagram of a fully polarized image restoration device based on improved sub-pixel segmentation provided in an embodiment of the present invention;

[0059] Figure 14 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0061] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0062] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0063] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0064] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0065] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0066] Polarization imaging is an imaging technique that uses the polarization characteristics of light waves to acquire image information. It can decompose various information such as polarization degree, polarization purity, and dichroism through Stokes vector decomposition. This can simplify complex image information, visualize invisible information, and enhance the contrast of similar information, thereby analyzing important information such as the material, texture, and anisotropic structure of the photographed object.

[0067] A focal plane polarization imaging system, also known as a polarization camera, is fabricated by directly integrating micro- and nano-polarization elements onto the target surface of an image sensor (such as a CCD or CMOS sensor). A representative example is the commercially available Sony industrial polarization camera, which integrates a 2×2 periodic nano-metal wire grating array one-to-one onto a CMOS chip (where each square in the 2×2 array is called a subpixel, and the entirety of 2×2 subpixels is called a superpixel). By capturing an image in this way, four types of linear polarization can be detected on the target. Subsequent interpolation processing can reconstruct four images, which, after further calculation, yield the Stokes vector, decompose the linear polarization degree, and polarization angle information. However, this product can only record the linear polarization information of the target, making it difficult to handle applications with rich circular polarization information, such as underwater scenes and biological samples. Furthermore, since the position of each polarization subpixel inevitably lacks the other three types of polarization information, it results in a loss of imaging resolution and a reduction in polarization solution accuracy.

[0068] Fully polarized cameras can acquire not only linear polarization angles and degrees of linear polarization, but also circular polarization, providing richer and more accurate image data. A representative product is the SALSA polarization imaging system manufactured by the U.S. Army Laboratory. It uses a liquid crystal spatial modulator (LC-SLM) as an electrically controlled adjustable phase delay film, applying different voltages to the modulator at different times and combining this with a fixed polarizer to acquire images in different polarization states. However, this product requires multiple shots to image the same scene, making it unsuitable for applications involving polarization imaging of fast-moving targets.

[0069] Therefore, integrating metasurface micro-polarization arrays or liquid crystal micro-polarization arrays into image sensors to form an integrated fully polarized camera can solve the above problems. However, metasurface micro-polarization arrays require nanometer-level processing precision, making them highly susceptible to performance degradation due to processing misalignment. Liquid crystal micro-polarization arrays, on the other hand, suffer from misalignment between different polarization sub-pixels due to elastic forces between liquid crystal molecules. This misalignment, especially when the sub-pixel size is below 5 μm, severely impacts performance. Therefore, fully polarized cameras based on these micro-polarization arrays currently remain in the laboratory stage and cannot be mass-produced.

[0070] Fabrication of liquid crystal micro-polarization arrays with side lengths in the range of 10 to 30 μm is very simple and performs well, making them highly suitable for the integration and mass production of fully polarized cameras. However, in existing polarization information reading and processing techniques, common methods involve directly calculating the polarization information by averaging the gray levels within sub-pixels, or using methods similar to bilinear interpolation and bicubic interpolation to process the polarization information on a sub-pixel basis before imaging. In this case, due to the large size of the sub-pixels, one sub-pixel corresponds to multiple image sensor pixels (with side lengths typically around 2 to 6 μm), which significantly reduces the imaging resolution and polarization solution accuracy. If gray level prediction could be performed within the sub-pixels to recover the fully polarized image, the promotion cost of fully polarized cameras and their imaging technology could be greatly reduced.

[0071] To address the problem of image restoration, this invention improves the method of reading known polarization information, thereby increasing the utilization rate of known polarization information. This not only improves the quality and accuracy of imaging but also significantly reduces reliance on high-precision optical components, thus reducing costs. For the problem of predicting unknown polarization information, this invention optimizes data processing to fit the known polarization information in a more scientific way, solves for unknowns, and handles outliers to obtain more accurate imaging results. Regarding the overall image restoration quality, this invention also proposes using an intelligent optimization algorithm to optimize image restoration parameters using PSNR (Peak Signal-to-Noise Ratio), a commonly used metric for measuring image or video quality, ultimately obtaining an image with optimal imaging performance.

[0072] like Figure 1 As shown, Figure 1 A fully polarized image restoration method based on improved sub-pixel segmentation is provided in this embodiment of the invention. The method includes the following steps:

[0073] S100, acquire image data collected by a polarization camera, the image data containing multiple sub-pixels, each sub-pixel being divided into four orthogonal regions, each region of the sub-pixel being adjacent to a sub-pixel recording different polarization information;

[0074] Among them, one region of the sub-pixel records known polarization information, while the other three regions are prediction regions where polarization information needs to be predicted.

[0075] S200, determine the known region and the predicted region of the sub-pixel, and select one known region from two sub-pixels that record the same polarization information adjacent to the predicted region to obtain two known regions corresponding to the predicted region; wherein, the known region is the region that records known polarization information, and the predicted region is the region where polarization information needs to be predicted; the distance between the predicted region and the known region in the two sub-pixels is different;

[0076] S300, obtain the grayscale mean values ​​of the two known regions respectively, calculate the absolute value of the difference between the grayscale mean values ​​of the two known regions, and obtain the absolute difference value corresponding to the predicted region;

[0077] S400, determine the direction of the predicted region in the sub-pixel, and determine the judgment threshold corresponding to the predicted region based on the direction of the predicted region in the sub-pixel;

[0078] S500, polarization information is determined from two selected known regions based on the absolute mean and judgment threshold corresponding to the prediction region, and the prediction region is predicted based on the polarization information;

[0079] S600, combine the polarization image corresponding to the known region in the sub-pixel and the polarization images corresponding to the three predicted regions to obtain the polarization image corresponding to the sub-pixel, and combine multiple polarization images corresponding to the sub-pixels to obtain the polarization image corresponding to the image data.

[0080] This invention requires first calibrating the polarization camera and then initially setting the judgment criteria. After reading image data, it predicts unknown polarization information under different conditions based on the judgment criteria. Finally, it processes abnormal data values ​​and generates a polarization map. The calibration algorithm of this invention uses measurement results from standard devices for system calibration, effectively solving measurement errors caused by modulation problems and ambient background light, ensuring high data accuracy. This calibration method provides an important technical guarantee for complex polarization measurements.

[0081] By using sub-pixel multi-sampling and intelligent prediction of unknown polarization information, this invention not only improves the utilization rate of polarization information but also enhances the resolution of the polarization image, making the final image closer to the actual value. Furthermore, this invention employs an intelligent optimization algorithm to automatically modify parameters to obtain the optimal polarization image. These features make this invention significantly superior to traditional methods in both image resolution and accuracy.

[0082] Compared to existing polarization imaging techniques, this invention achieves high-resolution image restoration even using larger sub-pixels. This significantly reduces the size requirements of the polarization array pixels, resulting in substantial reductions in manufacturing processes and costs. Furthermore, this invention makes a significant contribution to polarization restoration imaging with fully polarization cameras, overcoming hardware limitations imposed on such imaging.

[0083] Point selection methods in related technologies, such as Figure 2 As shown, Figure 2The diagram illustrates subpixels recording four types of polarization information, distinguished by four colors. For simplicity, the polarization information recorded by the blue, red, yellow, and green subpixels is referred to as polarization information 1, polarization information 2, polarization information 3, and polarization information 4, respectively. Traditional point-sampling methods simply average the grayscale values ​​within a subpixel and directly calculate the Stokes vector, then assign the information to the position corresponding to one of the subpixels.

[0084] This invention employs a compact point-taking, multi-set data reading method, such as... Figure 3 As shown.

[0085] Each sub-pixel is divided into four regions, each of which can contact sub-pixels recording different polarization information. Regions close to the predicted location are selected to predict unknown polarization information.

[0086] The compactness of the sampling point is reflected in the fact that this invention divides each sub-pixel into four regions, and each region can contact the region that records the corresponding polarization information. For the region used to predict unknown polarization information, it is also selected from the region of the adjacent sub-pixel that is close to the predicted position.

[0087] Reading multiple sets of data means that this data compresses multiple sets of grayscale values ​​from an n×n region within the area, and takes the average grayscale value as the grayscale mean of that region. For example... Figure 3 The points shown are taken in a 4×4 area. In actual use, the size of n can be set as needed to increase or decrease the image resolution.

[0088] In some embodiments, the average grayscale value is obtained in the following manner:

[0089] Read the grayscale values ​​of multiple n×n regions within the region, and take the average of the grayscale values ​​of the multiple n×n regions as the grayscale mean of the region.

[0090] Specifically, the formulas for calculating the average grayscale value of region 1 and region 2 are as follows:

[0091] ;

[0092] ;

[0093] ΔAverage=|Average1-Average2|;

[0094] Where i represents the index of a pixel in the region, X1i represents the grayscale value of the i-th pixel in region 1, N1 represents the total number of pixels in region 1, X2i represents the grayscale value of the i-th pixel in region 2, and N2 represents the total number of pixels in region 2. This represents the average gray level of region 1. ΔAverage represents the average gray value of region 2, Average1 represents the average gray value of region 1, Average2 represents the average gray value of region 2, and ΔAverage represents the absolute value of the difference between Average1 and Average2, i.e., the absolute difference.

[0095] In some embodiments, the determination threshold corresponding to the predicted region includes a first threshold in the horizontal or vertical direction and a second threshold in the ±45° direction. Determining the determination threshold corresponding to the predicted region based on its orientation within the sub-pixel includes:

[0096] If the direction of the predicted region in the sub-pixel is horizontal or vertical, then the determination threshold is a first threshold.

[0097] If the direction of the predicted region in the sub-pixel is ±45°, then the determination threshold is the second threshold; wherein, the calculation formula for the second threshold is: JT1 is the first threshold, and JT2 is the second threshold.

[0098] Specifically, JT1 represents the first threshold in the horizontal or vertical direction, used to determine polarization information 2 and polarization information 3, and the value range of JT1 is 0 to 180; JT2 represents the second threshold in the ±45° direction, the value of which is derived from geometric relationships, and is used to determine polarization information 4.

[0099] In some embodiments, determining polarization information from two selected known regions based on the absolute mean and a decision threshold corresponding to the prediction region, and predicting the prediction region based on the polarization information, includes:

[0100] S510, if the absolute difference is determined to be less than the determination threshold, then the polarization information of the two selected known regions is used for fitting, and the polarization information of the predicted region is predicted based on the fitting curve obtained by fitting.

[0101] S520, if the absolute difference is greater than the judgment threshold, then the polarization information of the known region that is closer to the prediction region is used to predict the prediction region.

[0102] The polarization information reading point selection method of the present invention will be described next:

[0103] like Figures 4a to 6bAs shown, four types of polarization information in the prediction region of image data are read and predicted. The prediction region records known polarization information 1, which can be directly read from sub-pixels. Polarization information 2, 3, and 4 need to be predicted based on the surrounding known polarization information. Regarding polarization information 2, 3, and 4, this invention selects to read the polarization information of the illustrated region (marked with boxes) respectively.

[0104] For each type of polarization information to be predicted (taking polarization information 2 as an example), this invention selects the polarization information of one region from two sub-pixels that record the same polarization information in adjacent prediction regions. One region is closer to the prediction region (region 1.1), and the other region is farther away from the prediction region (region 1.2).

[0105] In some embodiments, predicting the prediction region using polarization information from a known region closer to the prediction region includes:

[0106] S521, Select a first sub-region and a second sub-region from the known regions that are closer to the prediction region; wherein, the first sub-region is the region closest to the prediction region among the known regions that are closer to the prediction region, and the second sub-region is the region farthest from the prediction region among the known regions that are closer to the prediction region.

[0107] S522, read the average gray value of the first sub-region and the average gray value of the second sub-region respectively, calculate the absolute value of the difference between the average gray value of the first sub-region and the average gray value of the second sub-region, and obtain the absolute difference of the known region;

[0108] S523, determine the judgment threshold corresponding to the known region based on the direction of the predicted region in the sub-pixel, determine polarization information from the first sub-region and the second sub-region based on the absolute difference of the known region and the corresponding judgment threshold, and predict the predicted region based on the polarization information.

[0109] The specific explanation of the judgment is as follows:

[0110] Case 1: such as Figure 4a , 5a As shown in Figure 6a, if the absolute difference ΔAverage does not exceed the judgment threshold, then the polarization characteristics of regions 1.1 and 1.2 are determined to be insignificant. Therefore, the known polarization information of regions 1.1 and 1.2 is used to predict the prediction region, as follows: Figure 7 As shown.

[0111] Scenario 2: such as Figure 4b , 5bAs shown in Figure 6b, if the absolute difference ΔAverage is greater than the judgment threshold, then the polarization characteristics of these two regions are determined to change significantly. Therefore, the polarization information of the known region (region 1.1) closer to the prediction region is used to predict the prediction region, such as... Figure 8 As shown.

[0112] In some embodiments, the determination threshold corresponding to the known region includes a third threshold in the horizontal or vertical direction and a fourth threshold in the ±45° direction. Determining the determination threshold corresponding to the known region based on the orientation of the predicted region within the sub-pixel includes:

[0113] If the direction of the predicted region in the sub-pixel is horizontal or vertical, then the judgment threshold corresponding to the known region is determined as the third threshold; wherein, the calculation formula of the third threshold is: JF1=JT1×EAGE / (4×D-3×EAGE): where JF1 is the third threshold, JT1 is the first threshold, D is the side length of the sub-pixel, and EAGE is the maximum value among the distances of each pixel in the sub-pixel from the center point of the sub-pixel;

[0114] If the predicted region is oriented at ±45° within the sub-pixel, then the determination threshold corresponding to the known region is determined as the fourth threshold; wherein, the formula for calculating the fourth threshold is: .

[0115] Specifically, JF1 represents the third threshold in the horizontal or vertical direction after determining that the absolute difference is greater than the first threshold JT1, used to determine polarization information 2 and polarization information 3. The value of the third threshold is derived from geometric relationships and is approximately: JF1 = JT1 × EAGE / (4 × D - 3 × EAGE), where D is the side length of the sub-pixel, a known constant; EAGE is the maximum value among the distances of each pixel in the sub-pixel from the center point of the sub-pixel. Setting EAGE can avoid the problem of inaccurate grayscale values ​​due to diffraction limit caused by the data reading position being too close to the edge of the sub-pixel.

[0116] JF2 represents the fourth threshold in the ±45° direction after the absolute difference is determined to be greater than the second threshold JT2, used to determine polarization information 4; the value of the fourth threshold is derived from geometric relationships. .

[0117] In some embodiments, determining a decision threshold corresponding to the known region based on the orientation of the predicted region in the sub-pixel, determining polarization information from the first sub-region and the second sub-region based on the absolute difference of the known region and the corresponding decision threshold, and predicting the predicted region based on the polarization information includes:

[0118] If the absolute difference does not exceed the judgment threshold corresponding to the known region, then the polarization information of the first sub-region and the polarization information of the second sub-region are used to perform linear fitting, and the polarization information of the predicted region is predicted based on the fitting curve obtained by fitting.

[0119] If the absolute difference is greater than the judgment threshold corresponding to the known region, then the polarization information of the first sub-region is assigned to the predicted region.

[0120] For case 2: If the absolute difference is greater than the judgment threshold, the present invention uses the polarization information of the region (region 1.1) closer to the prediction region to predict the prediction region, and reads the polarization information of the regions (sub-region 2.1 and sub-region 2.2) shown in the figure to make predictions respectively.

[0121] Case 2.1: If ΔData is less than JF1 (JF2), the distance is short and can be regarded as a linear relationship. Then, the polarization information of the first sub-region and the polarization information of the second sub-region are linearly fitted to predict the polarization information of the prediction region.

[0122] Case 2.2: If ΔData is greater than JF1 (JF2), then Data1 is assigned to the prediction region. The polarization information recorded for each sub-pixel is represented by grayscale values.

[0123] ;

[0124] ;

[0125] Where Data1 represents the mean gray value of the location closest to the prediction area in region 1.1 (sub-region 2.1). Data2 represents the mean gray value of the location farthest from the prediction area in region 1.1 (sub-region 2.2). ΔData represents the absolute value of the difference between Data1 and Data2, ΔData = |Data1 - Data2|.

[0126] Furthermore, when reading image data, this invention records the maximum and minimum grayscale values ​​of the image. If the predicted grayscale value is subsequently found to exceed the maximum grayscale value, the outlier is assigned the maximum grayscale value; if the predicted grayscale value is subsequently found to be lower than the minimum grayscale value, the outlier is assigned the minimum grayscale value.

[0127] The above describes only the method for reading and predicting polarization information at one location, but the methods for reading and predicting polarization information at other locations are similar.

[0128] like Figure 9 This corresponds to the point selection method of four sub-pixels within a superpixel. Then, the polarization images corresponding to the sub-pixels are combined to obtain... Figure 10 The complete polarization image is shown.

[0129] like Figure 11 and Figure 12 The figure illustrates a comparison between the present invention and traditional image restoration techniques employing bilinear interpolation. In the figure, S0 represents the Stokes parameter, DoP (Degree of Polarization) represents the degree of polarization, and AoP (Angle of Polarization) represents the polarization angle.

[0130] Compared with the prior art, the advantages of the present invention are as follows:

[0131] Firstly, regarding the polarization information reading method: This invention proposes a more compact information reading method compared to traditional methods. This effectively avoids the problem of inaccurate polarization information prediction caused by the large distance between the predicted polarization information position and the existing data position. It also further subdivides the polarization information calculation area, which is beneficial for improving imaging resolution. Reading multiple sets of data at the sub-pixel level greatly improves information utilization, helps to reveal image details, and simultaneously enhances imaging resolution.

[0132] Secondly, the method for predicting polarization information: This invention proposes to predict the value of missing unknown information by fitting the changing trend of known polarization information, thereby improving the rationality and accuracy of missing information prediction. By using different prediction schemes based on index determination, the rationality and accuracy of missing information prediction are further improved, thus enhancing the final imaging quality.

[0133] Thirdly, the invention utilizes intelligent optimization algorithms to correct and optimize the data. Since outliers inevitably occur in polarization information prediction, this invention proposes setting outlier judgment indicators and corresponding repair schemes to improve the rationality of the data and the quality of the generated images. Intelligent optimization algorithms are used to optimize the preset judgment indicators. Specifically, the image data acquired by the polarization camera is simply processed to eliminate polarization differences between sub-pixels, generating a pseudo-original image. The intelligent optimization algorithm then restores the original image and the pseudo-original image, using peak signal-to-noise ratio (PSNR) as the indicator to iteratively optimize the judgment threshold multiple times. This significantly improves the imaging effect of objects with low polarization characteristics without negatively impacting the imaging of objects with high polarization characteristics. Peak signal-to-noise ratio (PSNR) is a commonly used quality evaluation indicator in image and video processing, typically used to measure the quality of image or video restoration.

[0134] It avoids the subjectivity and uncertainty of data caused by human intervention, and also avoids the cumbersome operation, thus realizing the automation and absolute objectivity of the imaging system.

[0135] See Figure 13This invention provides a fully polarized image restoration device based on improved sub-pixel segmentation, comprising:

[0136] The first module is used to acquire image data collected by a polarization camera. The image data contains multiple sub-pixels, each of which is divided into four orthogonal regions. Each region of the sub-pixel is adjacent to a sub-pixel that records different polarization information.

[0137] The second module is used to determine the known region and the predicted region of the sub-pixel. It selects one known region from two sub-pixels adjacent to the predicted region that record the same polarization information, thus obtaining two known regions corresponding to the predicted region. The known region is the region that records known polarization information, and the predicted region is the region whose polarization information needs to be predicted. The distance between the predicted region and the known regions in the two sub-pixels is different.

[0138] The third module is used to obtain the grayscale mean values ​​of the two known regions respectively, calculate the absolute value of the difference between the grayscale mean values ​​of the two known regions, and obtain the absolute difference value corresponding to the predicted region.

[0139] The fourth module is used to determine the direction of the predicted region in the sub-pixel, and to determine the judgment threshold corresponding to the predicted region based on the direction of the predicted region in the sub-pixel;

[0140] The fifth module is used to determine polarization information from two selected known regions based on the absolute mean and a judgment threshold corresponding to the prediction region, and to predict the prediction region based on the polarization information.

[0141] The sixth module is used to combine the polarization image corresponding to the known region in the sub-pixel and the polarization images corresponding to the three predicted regions to obtain the polarization image corresponding to the sub-pixel, and to combine multiple polarization images corresponding to the sub-pixel to obtain the polarization image corresponding to the image data.

[0142] It is evident that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented in this device embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0143] See Figure 14 This invention provides an electronic device, comprising:

[0144] At least one processor;

[0145] At least one memory for storing at least one program;

[0146] When the at least one program is executed by the at least one processor, the at least one processor performs the method described above.

[0147] It is evident that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented in this device embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0148] Furthermore, this application also discloses a computer program product or computer program stored in a computer-readable storage medium. A processor of a computer device can read the computer program from the computer-readable storage medium, and the processor executes the computer program, causing the computer device to perform the described method. Similarly, the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0149] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0150] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0151] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0152] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0153] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0154] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0155] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0156] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0157] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for restoring fully polarized images based on improved sub-pixel segmentation, characterized in that, The method includes the following steps: Image data acquired by a polarization camera is obtained. The image data contains multiple sub-pixels. Each sub-pixel is divided into four orthogonal regions. Each region of the sub-pixel is adjacent to a sub-pixel that records different polarization information. Obtain the known region and three prediction regions of the sub-pixel. Select one known region from two sub-pixels that record the same polarization information adjacent to the prediction region to obtain two known regions corresponding to the prediction region. The known region is the region that records known polarization information, and the prediction region is the region where polarization information needs to be predicted. The distance between the prediction region and the known region in the two sub-pixels is different. The grayscale mean values ​​of the two known regions are obtained respectively, and the absolute value of the difference between the grayscale mean values ​​of the two known regions is calculated to obtain the absolute difference value corresponding to the predicted region. Determine the orientation of the predicted region in the sub-pixel, and determine the judgment threshold corresponding to the predicted region based on the orientation of the predicted region in the sub-pixel; Polarization information is determined from two selected known regions based on the absolute mean and decision threshold corresponding to the predicted region. The predicted region is then predicted based on the polarization information to obtain the polarization image corresponding to the predicted region. The polarization image corresponding to the known region in the sub-pixel is combined with the polarization images corresponding to the three predicted regions to obtain the polarization image corresponding to the sub-pixel. The polarization images corresponding to multiple sub-pixels are combined to obtain the polarization image corresponding to the image data.

2. The method according to claim 1, characterized in that, The average grayscale value is obtained in the following way: Read the grayscale values ​​of multiple n×n regions within the region, and take the average of the grayscale values ​​of the multiple n×n regions as the grayscale mean of the region.

3. The method according to claim 1, characterized in that, The determination threshold corresponding to the predicted region includes a first threshold in the horizontal or vertical direction, and a second threshold in the ±45° direction. Determining the determination threshold corresponding to the predicted region based on its orientation within the sub-pixel includes: If the direction of the predicted region in the sub-pixel is horizontal or vertical, then the determination threshold is a first threshold. If the predicted region is oriented at ±45° within the sub-pixel, then the determination threshold is a second threshold; wherein the formula for calculating the second threshold is: JT1 is the first threshold, and JT2 is the second threshold.

4. The method according to claim 1, characterized in that, The step of determining polarization information from two selected known regions based on the absolute mean and a decision threshold corresponding to the prediction region, and predicting the prediction region based on the polarization information, includes: If the absolute difference is determined to be less than the determination threshold, then the polarization information of the two selected known regions is used for fitting, and the polarization information of the predicted region is predicted based on the fitting curve obtained by fitting. If the absolute difference is greater than the threshold, the polarization information of the known region closer to the prediction region is used to predict the prediction region.

5. The method according to claim 4, characterized in that, The step of using polarization information from a known region closer to the prediction region to predict the prediction region includes: Select a first sub-region and a second sub-region from the known regions that are closer to the prediction region; wherein, the first sub-region is the region closest to the prediction region among the known regions that are closer to the prediction region, and the second sub-region is the region farthest from the prediction region among the known regions that are closer to the prediction region. Read the average gray value of the first sub-region and the average gray value of the second sub-region respectively, calculate the absolute value of the difference between the average gray value of the first sub-region and the average gray value of the second sub-region, and obtain the absolute difference of the known region. Based on the orientation of the predicted region in the sub-pixel, a decision threshold corresponding to the known region is determined. Based on the absolute difference of the known region and the corresponding decision threshold, polarization information is determined from the first sub-region and the second sub-region. Based on the polarization information, the predicted region is predicted.

6. The method according to claim 5, characterized in that, The determination threshold corresponding to the known region includes a third threshold in the horizontal or vertical direction, and a fourth threshold in the ±45° direction. Determining the determination threshold corresponding to the known region based on the orientation of the predicted region within the sub-pixel includes: If the direction of the predicted region in the sub-pixel is horizontal or vertical, then the judgment threshold corresponding to the known region is determined as the third threshold; wherein, the calculation formula of the third threshold is: JF1=JT1×EAGE / (4×D-3×EAGE): where JF1 is the third threshold, JT1 is the first threshold, D is the side length of the sub-pixel, and EAGE is the maximum value among the distances of each pixel in the sub-pixel from the center point of the sub-pixel; If the predicted region is oriented at ±45° within the sub-pixel, then the determination threshold corresponding to the known region is determined as the fourth threshold; wherein, the formula for calculating the fourth threshold is: .

7. The method according to claim 5, characterized in that, The step of determining polarization information from the first sub-region and the second sub-region based on the absolute difference of the known region and the corresponding decision threshold, and predicting the prediction region based on the polarization information, includes: If the absolute difference does not exceed the judgment threshold corresponding to the known region, then the polarization information of the first sub-region and the polarization information of the second sub-region are used to perform linear fitting, and the polarization information of the predicted region is predicted based on the fitting curve obtained by fitting. If the absolute difference is greater than the judgment threshold corresponding to the known region, then the polarization information of the first sub-region is assigned to the predicted region.

8. A fully polarized image restoration device based on improved sub-pixel segmentation, characterized in that, The device includes: The first module is used to acquire image data collected by a polarization camera. The image data contains multiple sub-pixels, each of which is divided into four orthogonal regions. Each region of the sub-pixel is adjacent to a sub-pixel that records different polarization information. The second module is used to determine the known region and the predicted region of the sub-pixel. It selects one known region from two sub-pixels adjacent to the predicted region that record the same polarization information, thus obtaining two known regions corresponding to the predicted region. The known region is the region that records known polarization information, and the predicted region is the region whose polarization information needs to be predicted. The distance between the predicted region and the known regions in the two sub-pixels is different. The third module is used to obtain the grayscale mean values ​​of the two known regions respectively, calculate the absolute value of the difference between the grayscale mean values ​​of the two known regions, and obtain the absolute difference value corresponding to the predicted region. The fourth module is used to determine the direction of the predicted region in the sub-pixel, and to determine the judgment threshold corresponding to the predicted region based on the direction of the predicted region in the sub-pixel; The fifth module is used to determine polarization information from two selected known regions based on the absolute mean and a judgment threshold corresponding to the prediction region, and to predict the prediction region based on the polarization information. The sixth module is used to combine the polarization image corresponding to the known region in the sub-pixel and the polarization images corresponding to the three predicted regions to obtain the polarization image corresponding to the sub-pixel, and to combine multiple polarization images corresponding to the sub-pixel to obtain the polarization image corresponding to the image data.

9. An electronic device, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor performs the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a processor-executable program, characterized in that, The processor-executable program, when executed by the processor, is used to perform the method as described in any one of claims 1 to 7.

Images