Full-polarization image restoration method based on improved sub-pixel segmentation and related equipment

Abstract

The invention relates to the technical field of image processing, in particular to a full-polarization image recovery method and related equipment based on improved sub-pixel segmentation, and the method comprises the steps: obtaining image data collected by a polarization camera, determining a known region and three prediction regions of the sub-pixels, a known area is selected from each of two sub-pixels which are close to the prediction area and record the same kind of polarization information, the absolute value of the difference between the gray average values of the two known areas is calculated, and the absolute difference value corresponding to the prediction area is obtained; determining a judgment threshold based on the direction of the prediction region in the sub-pixels, further determining polarization information, and obtaining a polarization image corresponding to the prediction region based on the polarization information; combining the polarization images corresponding to the areas in the sub-pixels to obtain a polarization image corresponding to the sub-pixels, and combining the polarization images corresponding to the plurality of sub-pixels to obtain a polarization image corresponding to the image data; according to the invention, the accuracy and stability of full-polarization image restoration are effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular, to a full-polarization image restoration method based on improved sub-pixel segmentation and related devices. Background Art

[0002] Although existing technologies can restore the polarization information collected by a split focal plane polarization imaging system to a certain extent, there are still many defects. For example, traditional image polarization information restoration is based on sub-pixels (the gray value of a sub-pixel is averaged to define the gray value of the entire sub-pixel, so only one polarization information can be read from each sub-pixel), which greatly reduces the resolution of the image and loses a lot of polarization information. In addition, traditional restoration methods predict the missing polarization information through simple interpolation methods, which not only blurs the boundaries of the image but also may generate abnormal polarization information. Summary of the Invention

[0003] In view of this, the purpose of the embodiments of the present invention is to provide a full-polarization image restoration method based on improved sub-pixel segmentation and related devices to solve one or more technical problems existing in the prior art, and at least provide a beneficial choice or create conditions.

[0004] On the one hand, the embodiments of the present invention provide a full-polarization image restoration method based on improved sub-pixel segmentation, and the method includes the following steps: Obtain image data collected by a polarization camera. The image data includes a plurality of sub-pixels, and each sub-pixel is divided into four orthogonal regions. Each region in the sub-pixel is adjacent to a sub-pixel recording different polarization information; Determine the known region and the prediction region of the sub-pixel. Select one known region from each of the two sub-pixels recording the same polarization information adjacent to the prediction region to obtain two known regions corresponding to the prediction region. Among them, the known region is the region recording known polarization information, and the prediction region is the region where the polarization information needs to be predicted. The distances between the prediction region and the known regions in the two sub-pixels are different; Obtain the gray value means of the two known regions respectively, calculate the absolute value of the difference between the gray value means of the two known regions to obtain the absolute difference value corresponding to the prediction region; Determine the direction of the prediction region in the sub-pixel, and determine the decision threshold corresponding to the prediction region based on the direction of the prediction region in the sub-pixel; Determine the polarization information from the selected two known regions based on the absolute mean value and the decision threshold corresponding to the prediction region, and predict the prediction region based on the polarization information; Combine the polarization images corresponding to the known regions in the sub-pixels and the polarization images corresponding to the three prediction regions to obtain the polarization image corresponding to the sub-pixel, and combine the polarization images corresponding to multiple sub-pixels to obtain the polarization image corresponding to the image data.

[0005] Optionally, the gray mean value is obtained in the following manner: Read the gray values of multiple n×n regions within the region, and take the average of the gray values of multiple n×n regions as the gray mean value of this region.

[0006] Optionally, the determination threshold corresponding to the prediction region includes a first threshold in the horizontal or vertical direction, and a second threshold in the ±45° direction. Determining the determination threshold corresponding to the prediction region based on the direction of the prediction region in the sub-pixel includes: If the direction of the prediction region in the sub-pixel is the horizontal or vertical direction, the determination threshold is the first threshold; If the direction of the prediction region in the sub-pixel is the ±45° direction, the determination threshold is the second threshold; where the calculation formula for the second threshold is: , JT1 is the first threshold, and JT2 is the second threshold.

[0007] Optionally, determining the polarization information from the two selected known regions based on the absolute mean value and the determination threshold corresponding to the prediction region, and predicting the prediction region based on the polarization information includes: If it is determined that the absolute difference does not exceed the determination threshold, use the polarization information of the two selected known regions for fitting, and predict the polarization information of the prediction region based on the fitting curve obtained by fitting; If it is determined that the absolute difference is greater than the determination threshold, use the polarization information of the known region closer to the prediction region to predict the prediction region.

[0008] Optionally, using the polarization information of the known region closer to the prediction region to predict the prediction region includes: Select a first sub-region and a second sub-region from the known region closer to the prediction region; where the first sub-region is the region closest to the position of the prediction region in the known region closer to the prediction region, and the second sub-region is the region farthest from the position of the prediction region in the known region closer to the prediction region; Read the gray mean value of the first sub-region and the gray mean value of the second sub-region respectively, and calculate the absolute value of the difference between the gray mean value of the first sub-region and the gray mean value of the second sub-region to obtain the absolute difference of the known region; Determine the determination threshold corresponding to the known region based on the direction of the prediction region in the sub-pixel, determine the polarization information from the first sub-region and the second sub-region based on the absolute difference of the known region and the corresponding determination threshold, and predict the prediction region based on the polarization information.

[0009] Optionally, the determination threshold corresponding to the known region includes a third threshold in the horizontal direction or the vertical direction, and a fourth threshold in the ±45° direction. The determining the determination threshold corresponding to the known region based on the direction of the prediction region in the sub-pixel includes: If the direction of the prediction region in the sub-pixel is the horizontal direction or the vertical direction, determine the determination threshold corresponding to the known region as the third threshold; wherein, the calculation formula of the third threshold is: JF1 = JT1×EAGE / (4×D - 3×EAGE): where JF1 is the third threshold, JT1 is the first threshold, D is the side length of the sub-pixel, and EAGE is the maximum value of the distances from each pixel point in the sub-pixel to the center point of the sub-pixel; If the direction of the prediction region in the sub-pixel is the ±45° direction, determine the determination threshold corresponding to the known region as the fourth threshold; wherein, the calculation formula of the fourth threshold is: .

[0010] Optionally, the determining the polarization information from the first sub-region and the second sub-region based on the absolute difference of the known region and the corresponding determination threshold, and predicting the prediction region based on the polarization information includes: If the absolute difference does not exceed the determination threshold corresponding to the known region, perform linear fitting using the polarization information of the first sub-region and the polarization information of the second sub-region, and predict the polarization information of the prediction region based on the obtained fitting curve; If the absolute difference is greater than the determination threshold corresponding to the known region, assign the polarization information of the first sub-region to the prediction region.

[0011] On the other hand, an embodiment of the present invention provides a full polarization image restoration device based on improved sub-pixel segmentation, including: A first module for acquiring image data collected by a polarization camera, the image data including a plurality of sub-pixels, each of the sub-pixels being divided into four orthogonal regions, and each region in the sub-pixel being adjacent to a sub-pixel recording different polarization information; A second module, configured to determine a known region and a prediction region of the sub-pixel, respectively select a known region from two sub-pixels adjacent to the prediction region and recording the same polarization information, so as to obtain two known regions corresponding to the prediction region; wherein, the known region is a region recording known polarization information, and the prediction region is a region for which polarization information needs to be predicted; the distances between the prediction region and the known regions in the two sub-pixels are different; A third module, configured to respectively obtain the gray-scale means of the two known regions, calculate the absolute value of the difference between the gray-scale means of the two known regions, so as to obtain the absolute difference value corresponding to the prediction region; A fourth module, configured to determine the direction of the prediction region in the sub-pixel, and determine a determination threshold corresponding to the prediction region based on the direction of the prediction region in the sub-pixel; A fifth module, configured to determine polarization information from the two selected known regions based on the absolute mean value and the determination threshold corresponding to the prediction region, and predict the prediction region based on the polarization information; A sixth module, configured to combine the polarization images corresponding to the known regions in the sub-pixel and the polarization images corresponding to 3 prediction regions, so as to obtain the polarization image corresponding to the sub-pixel, and combine the polarization images corresponding to multiple sub-pixels to obtain the polarization image corresponding to the image data.

[0012] On the other hand, an embodiment of the present invention provides an electronic device, including: At least one processor; At least one memory, configured to store at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the above method.

[0013] On the other hand, an embodiment of the present invention provides a computer-readable storage medium, in which a program executable by a processor is stored, and the program executable by the processor is used to execute the above method when executed by the processor.

[0014] The embodiment of the present invention has the following beneficial effects: By determining the known region and the prediction region of the sub-pixel, respectively selecting a known region from two sub-pixels adjacent to the prediction region and recording the same polarization information, two known regions corresponding to the prediction region are obtained; furthermore, the absolute value of the difference between the gray-scale means of the known region and the prediction region is calculated, and according to the comparison result between the absolute difference value and the determination threshold, not only the utilization rate of polarization information is improved, but also the resolution of the polarization image is improved, making the finally formed image closer to the actual value. Accurately determine the polarization information of the prediction region, and effectively improve the accuracy and stability of full polarization image restoration. Description of the Drawings

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0016] Figure 1 It is a schematic flowchart of the steps of a full-polarization image restoration method based on improved sub-pixel segmentation provided by an embodiment of the present invention; Figure 2 It is a schematic diagram of selecting a known area in the related art; Figure 3 It is a schematic diagram of selecting a known area in an embodiment of the present invention; Figure 4a It is an example diagram of selecting a known area for polarization information 2 when the absolute difference does not exceed the determination threshold in an embodiment of the present invention; Figure 4b It is an example diagram of selecting a known area for polarization information 2 when the absolute difference is greater than the determination threshold in an embodiment of the present invention; Figure 5a It is an example diagram of selecting a known area for polarization information 3 when the absolute difference does not exceed the determination threshold in an embodiment of the present invention; Figure 5b It is an example diagram of selecting a known area for polarization information 3 when the absolute difference is greater than the determination threshold in an embodiment of the present invention; Figure 6a It is an example diagram of selecting a known area for polarization information 4 when the absolute difference does not exceed the determination threshold in an embodiment of the present invention; Figure 6b It is an example diagram of selecting a known area for polarization information 4 when the absolute difference is greater than the determination threshold in an embodiment of the present invention; Figure 7 It is an example diagram of predicting a prediction area when the absolute difference does not exceed the determination threshold in an embodiment of the present invention; Figure 8 It is an example diagram of predicting a prediction area when the absolute difference is greater than the determination threshold in an embodiment of the present invention; Figure 9 It is a schematic diagram of taking points for each sub-pixel in an embodiment of the present invention; Figure 10 It is a final combination schematic diagram in an embodiment of the present invention; Figure 11 It is an imaging diagram in an embodiment of the present invention; Figure 12 It is an imaging diagram obtained by the image restoration technology using bilinear interpolation in the related art; Figure 13 It is the architecture diagram of a full polarization image restoration device based on improved sub-pixel segmentation provided by an embodiment of the present invention; Figure 14 It is the structural block diagram of an electronic device provided by an embodiment of the present invention. Detailed implementation manners

[0017] In order to make the objectives, technical solutions and advantages of the present application more clear and understandable, the present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0018] It should be noted that although functional module division is performed in the device schematic diagram and the logical sequence is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order from the module division in the device or the sequence in the flowchart. Terms such as "first" and "second" in the specification, claims and the above-mentioned drawings are used to distinguish similar objects and do not necessarily need to be used to describe a specific order or sequence.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0020] In addition, the described features, structures or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, many specific details are provided to give a full understanding of the embodiments of the present application. However, those skilled in the art will realize that the technical solutions of the present application can be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, well-known methods, devices, implementations or operations are not shown or described in detail to avoid obscuring various aspects of the present application.

[0021] The block diagrams shown in the drawings are only functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0022] The flowcharts shown in the drawings are only illustrative and do not necessarily include all the content and operations / steps, nor do they necessarily need to be executed in the described order. For example, some operations / steps can be decomposed, while some operations / steps can be combined or partially combined, so the actual execution order may change according to the actual situation.

[0023] Polarization imaging is an imaging technique that uses the polarization characteristics of light waves to obtain image information. Multiple information such as degree of polarization, polarization purity, and dichroism can be decomposed from the Stokes vector. Thus, complex picture information can be made single, invisible information can be visualized, and the contrast of similar information can be enhanced, so as to analyze important information such as the material, texture, and anisotropic structure of the photographed object.

[0024] The split focal plane polarization imaging system, also known as a polarization camera, is directly made by integrating micro-nano polarization elements on the target surface of an image sensor (such as CCD, CMOS, etc.). A representative example is the currently commercially available Sony industrial polarization camera, which integrates a 2×2 periodic nano metal wire grating array one-to-one on a CMOS chip (where each square in the 2×2 is called a sub-pixel, and the whole formed by 2×2 sub-pixels is called a super-pixel). In this way, four linear polarization detections can be performed on the target by taking one image. Subsequent interpolation processing can be used to recover 4 images, and information such as the Stokes vector, degree of linear polarization, and polarization angle can be obtained through calculation. However, this product can only record the linear polarization information of the target and is difficult to handle application scenarios rich in circular polarization information such as underwater scenes and biological samples; in addition, since the position of each polarization sub-pixel will necessarily lack the other three polarization information, it will cause a loss of imaging resolution and a reduction in polarization solution accuracy.

[0025] The full polarization camera can not only obtain attributes such as the linear polarization angle and degree of linear polarization of an image, but also has the attribute of circular polarization, thus providing richer and more accurate image data. A representative product is the SALSA polarization imaging system manufactured by the US Army Laboratory, which uses a liquid crystal spatial light modulator (LC-SLM) as an electrically controlled tunable phase retarder, applies different voltages to the liquid crystal modulator at different times, and combines a fixed polarizer to collect images of different polarization states. However, this product requires multiple shots to image the same scene and is difficult to handle application scenarios of polarizing imaging of fast-moving target objects.

[0026] Based on this, integrating a metasurface micro-polarization array or a liquid crystal micro-polarization array into an image sensor to form an integrated full polarization camera can solve the above problems. However, the metasurface micro-polarization array must adopt nano-scale processing accuracy, and it is extremely easy to cause performance degradation due to processing misalignment; and due to the elastic force between liquid crystal molecules in the liquid crystal micro-polarization array, there will be disclinations between different polarization sub-pixels. Especially when the sub-pixel size is less than 5μm, the disorder of liquid crystal arrangement will seriously affect its performance. Therefore, the full polarization camera based on the above micro-polarization array currently only stays in the laboratory stage and cannot be mass-produced.

[0027] The preparation of a liquid crystal micro-polarization array with a side length in the order of 10 to 30 μm is very simple, and its performance is also good, which is very suitable for the integration and large-scale production of a full-polarization camera. However, in the existing polarization information reading and processing technologies, the common method is to directly calculate after averaging the gray levels within a sub-pixel, or use methods such as bilinear interpolation and bicubic interpolation based on sub-pixels to process the polarization information for imaging. In this case, due to the too large size of the sub-pixel, one sub-pixel will correspond to multiple image sensor pixels (the side length is usually about 2 to 6 μm), so that the imaging resolution and the polarization solving accuracy will be greatly reduced. If the gray level within the sub-pixel can be predicted to recover the full-polarization image, the popularization cost of the full-polarization camera and its imaging technology can be greatly reduced.

[0028] Regarding the problem of restored imaging, the present invention improves the method for reading known polarization information, enhances the utilization rate of the known polarization information, can not only improve the quality and accuracy of imaging, but also significantly reduce the dependence on high-precision optical components and reduce costs. Regarding the prediction problem of unknown polarization information, the present invention optimizes data processing, fits based on the known polarization information in a more scientific way, solves the unknown quantity, and processes the outliers that appear to obtain a more accurate imaging result. For the overall restoration degree of the image, the present invention also proposes to use an intelligent optimization algorithm to optimize the image restoration parameters with the PSNR (Peak Signal-to-Noise Ratio), a common index for measuring the quality of an image or video, as the index, and finally obtain an image with the optimal imaging effect.

[0029] As Figure 1 shown, Figure 1 The present invention provides a method for full-polarization image restoration based on improved sub-pixel segmentation according to an embodiment of the present invention. The method includes the following steps: S100, obtaining image data collected by a polarization camera, the image data including a plurality of sub-pixels, each sub-pixel being divided into four orthogonal regions, and each region in the sub-pixel being adjacent to a sub-pixel recording different polarization information; wherein, one region in the sub-pixel records known polarization information, and the other three regions are prediction regions for which polarization information needs to be predicted, S200, determining the known region and the prediction region of the sub-pixel, and respectively selecting a known region from two sub-pixels recording the same polarization information adjacent to the prediction region to obtain two known regions corresponding to the prediction region; wherein, the known region is the region recording known polarization information, and the prediction region is the region for which polarization information needs to be predicted; the distances between the prediction region and the known regions in the two sub-pixels are different; S300. Obtain the grayscale means of the two known regions respectively, calculate the absolute value of the difference between the grayscale means of the two known regions, and obtain the absolute difference corresponding to the prediction region. S400. Determine the direction of the prediction region in the sub-pixel, and determine the decision threshold corresponding to the prediction region based on the direction of the prediction region in the sub-pixel. S500. Determine the polarization information from the two selected known regions based on the absolute mean and decision threshold corresponding to the prediction region, and predict the prediction region based on the polarization information. S600. Combine the polarization images corresponding to the known regions in the sub-pixel and the polarization images corresponding to the 3 prediction regions to obtain the polarization image corresponding to the sub-pixel, and combine the polarization images corresponding to multiple sub-pixels to obtain the polarization image corresponding to the image data.

[0030] The present invention needs to calibrate the polarization camera first and then initially set the judgment index. After reading the image data, it will predict the unknown polarization information in different situations according to the judgment index. Finally, it will process the abnormal data values and generate a polarization map. The calibration algorithm of the present invention effectively solves the measurement errors caused by modulation problems and environmental background light through system calibration using the measurement results of standard devices, ensuring high data accuracy. This calibration method provides an important technical guarantee for complex polarization measurements.

[0031] By using multi-sampling points within the sub-pixel and intelligently predicting unknown polarization information, not only the utilization rate of polarization information is improved but also the resolution of the polarization image is increased, making the finally formed image closer to the actual value. In addition, the present invention also adopts an intelligent optimization algorithm to automatically modify parameters to obtain the optimal polarization image. These features make the present invention significantly superior to traditional methods in terms of image resolution and accuracy.

[0032] Compared with the existing polarization imaging technology, the present invention can also achieve high-resolution image restoration using larger-sized sub-pixels. This greatly reduces the requirements for the size of polarization array pixels, and the process and cost are also greatly reduced. In addition, the present invention makes a great contribution to the polarization restoration imaging of full-polarization cameras, breaking through the hardware conditions' limitations on the restoration imaging of full-polarization cameras.

[0033] The point-taking method in the related technology is as Figure 2 shown. Figure 2The sub-pixels recording four polarization information are shown, distinguished by four colors. Briefly, the polarization information recorded by the sub-pixels in blue, red, yellow, and green are referred to as polarization information 1, polarization information 2, polarization information 3, and polarization information 4, respectively. The traditional point-taking method only directly calculates the Stokes vector by averaging the grayscale within the sub-pixel and assigns the information to the position corresponding to one of the sub-pixels.

[0034] The present invention adopts a point-taking compact and multi-group data reading method, as Figure 3 shown.

[0035] Each sub-pixel is divided into four regions, and each region can contact the sub-pixels recording different polarization information. The region close to the predicted position is selected to predict the unknown polarization information.

[0036] The point-taking compactness is reflected in that the present invention divides each sub-pixel into four regions, and each region can contact the region recording the corresponding polarization information. For the region used to predict the unknown polarization information, it is also selected from the region close to the predicted position in the adjacent sub-pixels.

[0037] The reading of multi-group data is reflected in that this data compactly reads the grayscale values of multiple n×n regions within the region and takes the average grayscale value as the grayscale mean value of this region. As Figure 3 shown, the point-taking region is 4×4, and the size of n can be set as required in actual use to increase or decrease the picture resolution.

[0038] In some embodiments, the grayscale mean value is obtained in the following manner: Read the grayscale values of multiple n×n regions within the region, and take the average value of the grayscale values of multiple n×n regions as the grayscale mean value of this region.

[0039] Specifically, the calculation formulas for the grayscale mean values of region 1 and region 2 are as follows: ; ; ΔAverage = |Average1 - Average2|; where i represents the index of the pixel in the region, X1i represents the grayscale value of the i-th pixel in region 1, N1 represents the total number of pixels in region 1, X2i represents the grayscale value of the i-th pixel in region 2, N2 represents the total number of pixels in region 2, represents the grayscale mean value of region 1, Average0 represents the average grayscale value of region 2, Average1 represents the average grayscale value of region 1, Average2 represents the average grayscale value of region 2, and ΔAverage represents the absolute value of the difference between Average1 and Average2, that is, the absolute difference.

[0040] In some embodiments, the determination threshold corresponding to the prediction region includes a first threshold in the horizontal or vertical direction and a second threshold in the ±45° direction. Determining the determination threshold corresponding to the prediction region based on the direction of the prediction region in the sub-pixels includes: If the direction of the prediction region in the sub-pixels is the horizontal or vertical direction, the determination threshold is the first threshold; If the direction of the prediction region in the sub-pixels is the ±45° direction, the determination threshold is the second threshold; where the calculation formula for the second threshold is; , JT1 is the first threshold and JT2 is the second threshold.

[0041] Specifically, JT1 represents the first threshold in the horizontal or vertical direction, which is used to determine polarization information 2 and polarization information 3, and the value range of JT1 is from 0 to 180; JT2 represents the second threshold in the ±45° direction, and its numerical value is obtained from geometric relationships and is used to determine polarization information 4.

[0042] In some embodiments, determining the polarization information from the two selected known regions based on the absolute mean value and the determination threshold corresponding to the prediction region, and predicting the prediction region based on the polarization information includes: S510, if it is determined that the absolute difference does not exceed the determination threshold, then use the polarization information of the two selected known regions for fitting, and predict the polarization information of the prediction region based on the fitting curve obtained from the fitting; S520, if it is determined that the absolute difference is greater than the determination threshold, then use the polarization information of the known region closer to the prediction region to predict the prediction region.

[0043] Next, the method for selecting points for reading polarization information of the present invention will be described: As Figures 4a to 6b shown, read and predict the four polarization information of the prediction region in the image data. The prediction region records the known polarization information 1, and the polarization information 1 can be directly read from the sub-pixels. The polarization information 2, polarization information 3, and polarization information 4 need to be predicted based on the surrounding known polarization information. Regarding the polarization information 2, polarization information 3, and polarization information 4, the present invention selects to read the polarization information of the illustrated regions (marked with boxes) respectively.

[0044] For each polarization information to be predicted (taking polarization information 2 as an example), the present invention will select the polarization information of a region from two sub-pixels with the same polarization information near the prediction region. One region is closer to the prediction region (region 1.1), and one region is farther from the prediction region (region 1.2).

[0045] In some embodiments, predicting the prediction region using the polarization information of the known region closer to the prediction region includes: S521, selecting a first sub-region and a second sub-region from the known region closer to the prediction region; wherein, the first sub-region is the region closest to the prediction region in the known region closer to the prediction region, and the second sub-region is the region farthest from the prediction region in the known region closer to the prediction region; S522, respectively reading the grayscale mean value of the first sub-region and the grayscale mean value of the second sub-region, calculating the absolute value of the difference between the grayscale mean value of the first sub-region and the grayscale mean value of the second sub-region, to obtain the absolute difference value of the known region; S523, determining the determination threshold corresponding to the known region based on the direction of the prediction region in the sub-pixel, determining the polarization information from the first sub-region and the second sub-region based on the absolute difference value of the known region and the corresponding determination threshold, and predicting the prediction region based on the polarization information.

[0046] The specific description of the determination is as follows: Case 1: As Figure 4a 、 5a 、shown in 6a, if the absolute difference value ΔAverage does not exceed the determination threshold, then it is determined that the change in the polarization characteristics of region 1.1 and region 1.2 is not significant. Therefore, the polarization information of the known region 1.1 and the known region 1.2 is used to predict the prediction region, as Figure 7 shown.

[0047] Case 2: As Figure 4b 、 5b 、shown in 6b, if the absolute difference value ΔAverage is greater than the determination threshold, it is determined that the change in the polarization characteristics of these two regions is significant. Therefore, the polarization information of the known region (region 1.1) closer to the prediction region is used to predict the prediction region, as Figure 8 shown.

[0048] In some embodiments, the determination threshold corresponding to the known region includes a third threshold in the horizontal direction or the vertical direction, and a fourth threshold in the ±45° direction. Determining the determination threshold corresponding to the known region based on the direction of the prediction region in the sub-pixel includes: If the direction of the prediction region in the sub-pixel is horizontal or vertical, determine that the determination threshold corresponding to the known region is the third threshold; wherein, the calculation formula of the third threshold is: JF1 = JT1 × EAGE / (4 × D - 3 × EAGE): where JF1 is the third threshold, JT1 is the first threshold, D is the side length of the sub-pixel, and EAGE is the maximum value of the distances from each pixel point in the sub-pixel to the center point of the sub-pixel; If the direction of the prediction region in the sub-pixel is the ±45° direction, determine that the determination threshold corresponding to the known region is the fourth threshold; wherein, the calculation formula of the fourth threshold is: 。

[0049] Specifically, JF1 represents the third threshold in the horizontal or vertical direction after determining that the absolute difference is greater than the first threshold JT1, and is used to determine polarization information 2 and polarization information 3. The value of the third threshold is obtained from geometric relationships and is approximately: JF1 = JT1 × EAGE / (4 × D - 3 × EAGE), where D is the side length of the sub-pixel and D is a known constant; EAGE is the maximum value of the distances from each pixel point in the sub-pixel to the center point of the sub-pixel. Setting EAGE can avoid the problem that the gray value read is inaccurate due to the diffraction limit when the data reading position is too close to the edge of the sub-pixel.

[0050] JF2 represents the fourth threshold in the ±45° direction after determining that the absolute difference is greater than the second threshold JT2, and is used to determine polarization information 4; the value of the fourth threshold is obtained from geometric relationships, 。

[0051] In some embodiments, determining the determination threshold corresponding to the known region based on the direction of the prediction region in the sub-pixel, determining polarization information from the first sub-region and the second sub-region based on the absolute difference of the known region and the corresponding determination threshold, and predicting the prediction region based on the polarization information includes: If the absolute difference does not exceed the determination threshold corresponding to the known region, linearly fit the polarization information of the first sub-region and the polarization information of the second sub-region, and predict the polarization information of the prediction region based on the obtained fitting curve; If the absolute difference is greater than the determination threshold corresponding to the known region, assign the polarization information of the first sub-region to the prediction region.

[0052] For case 2: If it is determined that the absolute difference is greater than the determination threshold, the present invention uses the polarization information of the region closer to the prediction region (region 1.1) to predict the prediction region, and reads the polarization information of the illustrated regions (sub-region 2.1 and sub-region 2.2) respectively for prediction.

[0053] Case 2.1: If ΔData is less than JF1 (JF2) and the distance is short, it can be regarded as a linear relationship. Then, linearly fit the polarization information of the first sub-region and the polarization information of the second sub-region to predict the polarization information of the prediction region.

[0054] Case 2.2: If ΔData is greater than JF1 (JF2), then assign Data1 to the prediction region. The polarization information recorded by each sub-pixel is reflected by the gray value.

[0055] ; ; Among them, Data1 represents the gray mean value closest to the prediction region in Region 1.1 (Sub-region 2.1). Data2 represents the gray mean value farthest from the prediction region in Region 1.1 (Sub-region 2.2). ΔData represents the absolute value of the difference between Data1 and Data2, and ΔData = |Data1 - Data2|.

[0056] In addition, when reading the picture data, the present invention records the maximum gray value and the minimum gray value that appear in the image. If it is subsequently found that the predicted gray value exceeds the maximum gray value, the outlier is assigned the maximum gray value; if it is subsequently found that the predicted gray value is lower than the minimum gray value, the outlier is assigned the minimum gray value.

[0057] The above only illustrates the reading and prediction methods of the polarization information at one position, but the reading and prediction methods at other positions are similar to it.

[0058] Such as Figure 9 , this is the way of taking points corresponding to four sub-pixels within a super-pixel. Subsequently, combine the polarization images corresponding to the sub-pixels to obtain a complete polarization image as shown in Figure 10 .

[0059] Such as Figure 11 and Figure 12 show the comparison between the present invention and the traditional image restoration technology using bilinear interpolation. In the figure, S0 represents the Stokes parameter, DoP (Degree of Polarization) represents the degree of polarization, and AoP (Angle of Polarization) represents the angle of polarization.

[0060] Compared with the prior art, the advantages of the present invention are: One is the method for reading polarization information: The present invention proposes a more compact information reading method different from traditional methods, effectively avoiding the problem of inaccurate prediction of polarization information caused by the excessive distance between the predicted position of polarization information and the position of existing data. At the same time, the calculation area of polarization information is further subdivided, which is beneficial to improving the imaging resolution. The method of reading multiple groups of data in sub-pixels greatly improves the utilization rate of information, is beneficial to presenting image details, and also improves the imaging resolution.

[0061] The second is the prediction method of polarization information: The present invention proposes to predict the value of missing unknown information by fitting the known change trend of polarization information, improving the rationality and accuracy of the prediction of missing information. By using different prediction schemes through index determination, the rationality and accuracy of the prediction of missing information are further improved, and the final imaging quality is improved.

[0062] The third is the correction and optimization of data through intelligent optimization algorithms: Abnormal values are inevitable in the prediction of polarization information. The present invention proposes to set abnormal value determination indicators and corresponding repair schemes, improving the rationality of data and the imaging quality. The intelligent optimization algorithm is used to optimize the preset determination indicators. Specifically, the image data collected by the polarization camera is simply processed to eliminate the polarization difference between each sub-pixel, generating a pseudo-original image. Then, the intelligent optimization algorithm is used to restore the image and the pseudo-original image, and the determination threshold is iteratively optimized multiple times with the peak signal-to-noise ratio as the index. The imaging effect of objects with low polarization characteristics is significantly improved, and there is no negative effect on the imaging of objects with high polarization characteristics. The peak signal-to-noise ratio (PSNR) is a commonly used quality evaluation index in image and video processing, usually used to measure the restoration quality of images or videos.

[0063] It avoids the subjectivity and uncertainty of data caused by human intervention, and also avoids the complexity of operation, realizing the automation and absolute objectivity of the imaging system.

[0064] Refer to Figure 13 , the embodiment of the present invention provides a full polarization image restoration device based on improved sub-pixel segmentation, including: The first module is used to obtain the image data collected by the polarization camera. The image data includes multiple sub-pixels, each sub-pixel is divided into four orthogonal regions, and each region in the sub-pixel is adjacent to a sub-pixel recording different polarization information; A second module, configured to determine a known region and a prediction region of the sub-pixel. One known region is respectively selected from two sub-pixels that record the same polarization information and are adjacent to the prediction region, so as to obtain two known regions corresponding to the prediction region. Wherein, the known region is a region that records known polarization information, and the prediction region is a region for which polarization information needs to be predicted. The distances between the prediction region and the known regions in the two sub-pixels are different. A third module, configured to respectively obtain the gray-scale means of the two known regions, and calculate the absolute value of the difference between the gray-scale means of the two known regions, so as to obtain the absolute difference corresponding to the prediction region. A fourth module, configured to determine the direction of the prediction region in the sub-pixel, and determine a determination threshold corresponding to the prediction region based on the direction of the prediction region in the sub-pixel. A fifth module, configured to determine polarization information from the two selected known regions based on the absolute mean and the determination threshold corresponding to the prediction region, and predict the prediction region based on the polarization information. A sixth module, configured to combine the polarization images corresponding to the known regions in the sub-pixel and the polarization images corresponding to 3 prediction regions to obtain the polarization image corresponding to the sub-pixel, and combine the polarization images corresponding to multiple sub-pixels to obtain the polarization image corresponding to the image data.

[0065] It can be seen that the content in the above method embodiments is applicable to the device embodiments of the present invention. The functions specifically implemented by the device embodiments of the present invention are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those of the above method embodiments.

[0066] Referring to Figure 14 , an embodiment of the present invention provides an electronic device, including: At least one processor; At least one memory, configured to store at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the above method.

[0067] It can be seen that the content in the above method embodiments is applicable to the device embodiments of the present invention. The functions specifically implemented by the device embodiments of the present invention are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those of the above method embodiments.

[0068] In addition, an embodiment of the present application also discloses a computer program product or a computer program, which is stored in a computer-readable storage medium. The processor of the computer device can read the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the above method. Similarly, the content in the above method embodiments is applicable to this storage medium embodiment. The functions specifically implemented by this storage medium embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those of the above method embodiments.

[0069] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0070] Those of ordinary skill in the art can understand that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, and appropriate combinations thereof.

[0071] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily need to be used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances, so that the embodiments of the present application described here can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products, or devices.

[0072] It should be understood that in this application, "at least one (item)" means one or more, and "a plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that there can be three relationships. For example, "A and / or B" can mean: only A exists, only B exists, and both A and B exist at the same time. Among them, A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after. "At least one (item) of the following" or its similar expression refers to any combination of these items, including any combination of single items (items) or plural items (items). For example, at least one (item) of a, b, or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0073] In several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division. In actual implementation, there can be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some interfaces. The indirect coupling or communication connection of the devices or units can be in electrical, mechanical or other forms.

[0074] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place, or they can be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0075] In addition, the functional units in each embodiment of this application can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units.

[0076] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of this application. The aforementioned storage medium includes: various media that can store programs such as USB flash drives, mobile hard disks, read-only memories (ROM for short), random access memories (RAM for short), magnetic disks, or optical discs.

[0077] The preferred embodiments of the embodiments of this application have been described above with reference to the accompanying drawings, which does not limit the scope of the rights of the embodiments of this application. Any modifications, equivalent replacements, and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of this application shall fall within the scope of the rights of the embodiments of this application.

Claims

1. A full polarization image restoration method based on improved sub-pixel segmentation, characterized in that: The method comprises the following steps: Acquire image data collected by a polarization camera, wherein the image data includes a plurality of sub-pixels, each of the sub-pixels is divided into four orthogonal regions, and each region in the sub-pixel is adjacent to a sub-pixel recording different polarization information; Acquire the known area and three predicted areas of the sub-pixel, select one known area from each of two sub-pixels adjacent to the predicted area and recording the same polarization information, and obtain two known areas corresponding to the predicted area; wherein the known area is an area recording known polarization information, and the predicted area is an area for which polarization information needs to be predicted; and the predicted area is at a different distance from the known areas in the two sub-pixels; respectively obtaining grayscale means of the two known regions, calculating the absolute value of the difference between the grayscale means of the two known regions, and obtaining the absolute difference corresponding to the predicted region; Determine the direction of the prediction area in the sub-pixel, and determine a determination threshold corresponding to the prediction area based on the direction of the prediction area in the sub-pixel; Determine polarization information from two selected known areas based on an absolute mean value and a determination threshold corresponding to the predicted area, predict the predicted area based on the polarization information, and obtain a polarization image corresponding to the predicted area; The polarization image corresponding to the known area in the sub-pixel and the polarization images corresponding to the three predicted areas are combined to obtain the polarization image corresponding to the sub-pixel, and the polarization images corresponding to multiple sub-pixels are combined to obtain the polarization image corresponding to the image data.

2. The method according to claim 1, characterized in that The grayscale mean is obtained by: Grayscale values ​​of multiple groups of n×n regions are read in the region, and the average value of the grayscale values ​​of the multiple groups of n×n regions is taken as the grayscale mean value of the region.

3. The method according to claim 1, characterized in that The determination threshold corresponding to the prediction area includes a first threshold in the horizontal direction or the vertical direction, and a second threshold in the ±45° direction, and the determination of the determination threshold corresponding to the prediction area based on the direction of the prediction area in the sub-pixel includes: If the direction of the prediction area in the sub-pixel is a horizontal direction or a vertical direction, the determination threshold is a first threshold; If the direction of the prediction area in the sub-pixel is ±45°, the determination threshold is the second threshold; wherein the calculation formula of the second threshold is: , JT1 is the first threshold, and JT2 is the second threshold.

4. The method according to claim 1, characterized in that: The determining polarization information from the selected two known areas based on the absolute mean value and the determination threshold corresponding to the predicted area, and predicting the predicted area based on the polarization information, comprises: If it is determined that the absolute difference does not exceed the determination threshold, fitting is performed using the polarization information of the two selected known regions, and the polarization information of the predicted region is predicted based on a fitting curve obtained by fitting; If it is determined that the absolute difference is greater than the determination threshold, the prediction area is predicted using polarization information of a known area closer to the prediction area.

5. The method according to claim 4, characterized in that The using polarization information of a known area closer to the prediction area to predict the prediction area includes: Selecting a first sub-region and a second sub-region from a known region closer to the predicted region; wherein the first sub-region is a region closest to the predicted region in the known region closer to the predicted region, and the second sub-region is a region farthest from the predicted region in the known region closer to the predicted region; Respectively reading the grayscale mean of the first sub-region and the grayscale mean of the second sub-region, calculating the absolute value of the difference between the grayscale mean of the first sub-region and the grayscale mean of the second sub-region, and obtaining the absolute difference of the known region; Based on the direction of the predicted area in the sub-pixel, a determination threshold corresponding to the known area is determined; based on the absolute difference of the known area and the corresponding determination threshold, polarization information is determined from the first sub-area and the second sub-area; and the predicted area is predicted based on the polarization information.

6. The method according to claim 5, characterized in that The determination threshold corresponding to the known area includes a third threshold in the horizontal direction or the vertical direction, and a fourth threshold in the ±45° direction, and the determining the determination threshold corresponding to the known area based on the direction of the predicted area in the sub-pixel includes: If the direction of the predicted area in the sub-pixel is a horizontal direction or a vertical direction, the determination threshold corresponding to the known area is determined to be a third threshold; wherein the calculation formula of the third threshold is: JF1=JT1×EAGE / (4×D-3×EAGE): wherein JF1 is the third threshold, JT1 is the first threshold, D is the side length of the sub-pixel, and EAGE is the maximum value of the distances between each pixel point in the sub-pixel and the center point of the sub-pixel; If the direction of the predicted area in the sub-pixel is ±45°, the determination threshold corresponding to the known area is determined to be a fourth threshold; wherein the calculation formula of the fourth threshold is: .

7. The method according to claim 5, characterized in that The determining polarization information from the first sub-region and the second sub-region based on the absolute difference value of the known region and the corresponding determination threshold, and predicting the predicted region based on the polarization information, comprises: If the absolute difference does not exceed the determination threshold corresponding to the known area, linear fitting is performed using the polarization information of the first sub-area and the polarization information of the second sub-area, and the polarization information of the predicted area is predicted based on a fitting curve obtained by fitting; If the absolute difference is greater than the determination threshold corresponding to the known area, the polarization information of the first sub-area is assigned to the predicted area.

8. A full polarization image restoration device based on improved sub-pixel segmentation, characterized in that: The device comprises: A first module is used to obtain image data collected by a polarization camera, wherein the image data includes a plurality of sub-pixels, each of which is divided into four orthogonal regions, and each region in the sub-pixel is adjacent to a sub-pixel recording different polarization information; The second module is used to determine the known area and the predicted area of ​​the sub-pixel, and select one known area from each of the two sub-pixels adjacent to the predicted area and recording the same polarization information, to obtain two known areas corresponding to the predicted area; wherein the known area is an area recording known polarization information, and the predicted area is an area for which polarization information needs to be predicted; and the predicted area is at different distances from the known areas in the two sub-pixels; The third module is used to obtain the grayscale means of the two known areas respectively, calculate the absolute value of the difference between the grayscale means of the two known areas, and obtain the absolute difference corresponding to the predicted area; A fourth module is used to determine the direction of the prediction area in the sub-pixel, and determine a determination threshold corresponding to the prediction area based on the direction of the prediction area in the sub-pixel; A fifth module is used to determine polarization information from two selected known areas based on an absolute mean value and a determination threshold corresponding to the prediction area, and predict the prediction area based on the polarization information; The sixth module is used to combine the polarization image corresponding to the known area in the sub-pixel and the polarization images corresponding to the three predicted areas to obtain the polarization image corresponding to the sub-pixel, and combine the polarization images corresponding to multiple sub-pixels to obtain the polarization image corresponding to the image data.

9. An electronic device, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a program executable by a processor, characterized in that: The processor-executable program is used to perform the method according to any one of claims 1 to 7 when executed by the processor.

Images