Composite passive three-dimensional imaging method, device, equipment and storage medium
Through the composite passive three-dimensional imaging method, combined with polarization characteristic solution and depth estimation, high-precision three-dimensional imaging is achieved, solving the accuracy and cost problems in the prior art, and is suitable for long-distance detection and detail recovery.
Patent Information
- Application Number
- CN202411033164.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-30
- Publication Date
- 2024-11-15
AI Technical Summary
Existing three-dimensional imaging methods cannot meet the needs of high-precision, low-cost and long-distance detection at the same time, especially the passive method has shortcomings in accuracy and detail recovery.
By using the composite passive three-dimensional imaging method, by acquiring the camera input image, extracting the polarization sub-map and synthesizing the whole-optical image, calculating the surface normal, combining monocular depth estimation and scene semantic segmentation, three-dimensional reconstruction and depth correction are performed, and pseudo-radar point matching is used. and weight matrix update to achieve the generation of high-precision depth maps.
Passive, high-precision, low-cost, long-distance pixel-level three-dimensional imaging is realized, with strong real-time, simple structure and strong concealment. It can effectively detect at different distances and scenarios and restore the three-dimensional detailed information of the target of interest.
Smart Images

Figure CN118967910A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing technology, and in particular to a composite passive three-dimensional imaging method, device, equipment and storage medium. Background Art
[0002] With the rapid development of science and technology, 3D imaging technology is increasingly used in many fields such as education, entertainment, autonomous driving, security monitoring, etc. However, traditional 3D imaging methods often have many limitations.
[0003] The three-dimensional imaging methods in the related art are mainly divided into two types: active and passive.
[0004] Active methods include laser radar, APD array, structured light, etc. Among them, laser radar only obtains distance information, and it is a discrete distance measurement and cannot reach the pixel level; APD array has a small array size, low resolution and high cost; structured light has poor effect outdoors.
[0005] Passive methods mainly include multi-eye vision, ranging methods based on light fields, three-dimensional reconstruction based on polarization characteristics, and depth estimation based on deep learning. Among them, multi-eye vision is greatly affected by lighting and object texture, and the working distance is relatively short; the ranging method based on light fields has a short working distance and a complex system; three-dimensional reconstruction based on polarization characteristics can better retain the detailed information of the object surface, but it can only be used on continuous object surfaces, resulting in limited application scenarios, and the azimuth angle of the target surface normal during the solution process is uncertain, resulting in reduced accuracy; the depth estimation method based on deep learning can achieve pixel-level three-dimensional imaging, but the result lacks detailed information of the scene, resulting in reduced accuracy.
[0006] Based on this, the three-dimensional imaging methods in related technologies cannot meet the growing demand for high-precision, low-cost, and long-distance detection. Summary of the invention
[0007] The present invention provides a composite passive three-dimensional imaging method, device, equipment and storage medium, which solves the problem that the three-dimensional imaging method in the related art cannot simultaneously meet the requirements of high-precision, low-cost and long-distance detection.
[0008] In order to achieve the above objectives, this application adopts the following technical solutions:
[0009] In a first aspect, a composite passive three-dimensional imaging method is provided, comprising:
[0010] Obtaining an input image from a camera, extracting a polarization sub-image based on the input image, and synthesizing a plenoptic image;
[0011] Based on the polarization sub-image and the plenoptic image, the second surface normals of all pixels are calculated; at the same time, based on the plenoptic image, an initial absolute depth map of the input image is obtained through a monocular depth estimation network, and a target region of interest is extracted through a scene semantic segmentation network and its corresponding mask is obtained;
[0012] Calculate a first surface normal of the target area of interest based on the initial absolute depth map, and correct the second surface normal corresponding to the mask area according to the first surface normal;
[0013] Based on the corrected second surface normal, a relative depth map of the target region of interest is obtained, and three-dimensional reconstruction is performed on each mask region;
[0014] In combination with the intrinsic parameters of the camera, according to the initial absolute depth map and the mask, a first pseudo radar point of each mask area is obtained, and according to the three-dimensional reconstruction result of the mask area, a second pseudo radar point of each mask area is obtained;
[0015] Uniformly selecting points from the first pseudo radar points as landmark points, and matching them with corresponding points in the second pseudo radar points to obtain landmark point matching point pairs;
[0016] Constructing a K-nearest-neighbor directed graph of the second pseudo radar point and calculating a weight matrix;
[0017] Based on the landmark point matching point pair, the second pseudo radar point is replaced by the first pseudo radar point in the weight matrix, the weight matrix is updated, the residual pixel depth of each mask area is corrected, and the high-precision depth of the mask area is obtained.
[0018] Correcting the remaining pixel depth of each mask area according to the depth information of the absolute distance point cloud to obtain a high-precision depth of the mask area;
[0019] The depth of the target area of interest in the initial absolute depth map is replaced by the high-precision depth of the mask area to obtain a final depth map, and three-dimensional imaging is performed based on the final depth map.
[0020] In a first possible implementation manner of the first aspect, extracting a polarization sub-image based on the input image and synthesizing a plenoptic image includes:
[0021] Extract polarization sub-images I0, I1 at 0, 45, 90, and 135 degrees from the input image 45 ,I 90 ,I 135 , and synthesize the plenoptic image I:
[0022]
[0023] Based on the first possible implementation manner of the first aspect, in a second possible implementation manner of the first aspect, calculating the second surface normals of all pixels based on the polarization sub-image and the plenoptic image includes:
[0024] Polarization characteristics are calculated based on the polarization sub-graph to obtain the Stokes vector S:
[0025]
[0026] Where, Q is the linear polarization component at 0 degrees, and U is the linear polarization component at 45 degrees;
[0027] The polarization degree P and the azimuth angle are calculated based on the Stokes vector and the plenoptic image.
[0028]
[0029] Calculate the zenith angle of each pixel on the object surface:
[0030]
[0031] j2=4P(n 2 +1)(j1-4P)
[0032] j3=j2 2 +16P 2 (16P 2 -j1 2 )(n 2 -1 )2
[0033]
[0034] Where j1, j2, and j3 represent process variables, and n is the target reflection refractive index;
[0035] Calculate the surface normals of all pixels in the input image based on polarization characteristics
[0036]
[0037] In a third possible implementation manner of the first aspect, calculating the first surface normal of the target area of interest based on the initial absolute depth map, and correcting the second surface normal corresponding to the mask area according to the first surface normal, includes:
[0038] The first surface normal of each pixel in the target area of interest is obtained by calculating the two-dimensional array gradient of the initial absolute depth map in the target area of interest.
[0039] If the first surface normal and the second surface normal If the angle between is less than or equal to 90°, If the first surface normal and the second surface normal If the angle between is greater than 90°, in, is the second surface normal after correction.
[0040] In a fourth possible implementation manner of the first aspect, the first pseudo radar point and the second pseudo radar point are three-dimensional point clouds obtained by projecting corresponding depth information into a three-dimensional space according to an intrinsic parameter of the camera.
[0041] In a second aspect, a composite passive three-dimensional imaging device is provided, comprising:
[0042] An image preprocessing module, used to obtain an input image of a camera, extract a polarization sub-image based on the input image, and synthesize a plenoptic image;
[0043] A polarization characteristic solving module, used for calculating the second surface normals of all pixels based on the polarization sub-image and the plenoptic image;
[0044] A monocular depth estimation module, configured to obtain an initial absolute depth map of the input image through a monocular depth estimation network based on the plenoptic image;
[0045] The scene semantic segmentation module is used to extract the target region of interest through the scene semantic segmentation network and obtain its corresponding mask;
[0046] A normal correction module, configured to calculate a first surface normal of the target area of interest based on the initial absolute depth map, and correct the second surface normal corresponding to the mask area according to the first surface normal;
[0047] A three-dimensional reconstruction module, used to obtain a relative depth map of the target area of interest based on the corrected second surface normal, and perform three-dimensional reconstruction on each of the mask areas;
[0048] A pseudo radar point conversion module, configured to obtain a first pseudo radar point of each mask area according to the initial absolute depth map and the mask in combination with the intrinsic parameters of the camera, and obtain a second pseudo radar point of each mask area according to a three-dimensional reconstruction result of the mask area;
[0049] A landmark point matching module, configured to uniformly select points from the first pseudo radar points as landmark points, and match them with corresponding points in the second pseudo radar points to obtain landmark point matching point pairs;
[0050] A depth correction and imaging module, used to construct a K-nearest neighbor directed graph of the second pseudo radar point and calculate a weight matrix;
[0051] Based on the landmark point matching point pair, replacing the second pseudo radar point with the first pseudo radar point in the weight matrix, updating the weight matrix, correcting the remaining pixel depth of each mask area, and obtaining a high-precision depth of the mask area;
[0052] Correcting the remaining pixel depth of each mask area according to the depth information of the absolute distance point cloud to obtain a high-precision depth of the mask area;
[0053] The depth of the target area of interest in the initial absolute depth map is replaced by the high-precision depth of the mask area to obtain a final depth map, and three-dimensional imaging is performed based on the final depth map.
[0054] In a first possible implementation manner of the second aspect, the polarization characteristic solving module is specifically used to:
[0055] Polarization characteristics are calculated based on the polarization sub-graph to obtain the Stokes vector S:
[0056]
[0057] In the formula, I0, I 45 ,I 90 ,I 135 They are polarization subgraphs in the directions of 0, 45, 90, and 135 degrees, respectively. Q is the linear polarization component in the direction of 0, and U is the linear polarization component in the direction of 45.
[0058] The polarization degree P and the azimuth angle are calculated based on the Stokes vector and the plenoptic image.
[0059]
[0060] Calculate the zenith angle of each pixel on the object surface:
[0061]
[0062] j2=4P(n 2 +1)(j1-4P)
[0063] j3=j2 2 +16P 2 (16P 2 -j1 2 )(n 2 -1) 2
[0064]
[0065] Where j1, j2, and j3 represent process variables, and n is the target reflection refractive index;
[0066] Calculate the surface normals of all pixels in the input image based on polarization characteristics
[0067]
[0068] In a second possible implementation manner of the second aspect, the normal correction module is specifically configured to:
[0069] The first surface normal of each pixel in the target area of interest is obtained by calculating the two-dimensional array gradient of the initial absolute depth map in the target area of interest.
[0070] If the first surface normal and the second surface normal If the angle between is less than or equal to 90°, If the first surface normal and the second surface normal If the angle between is greater than 90°, in, is the second surface normal after correction.
[0071] According to a third aspect, an electronic device is provided, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the composite passive three-dimensional imaging method according to the first aspect.
[0072] In a fourth aspect, a readable storage medium is provided, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the composite passive three-dimensional imaging method as described in the first aspect are implemented.
[0073] Beneficial effects:
[0074] This application achieves passive, high-precision, low-cost, simple system, long detection distance, and pixel-level three-dimensional imaging. This application obtains the depth value of the entire scene based on a single-frame image, does not require active lighting such as laser and infrared, and has the characteristics of strong real-time performance, simple structure, easy integration, strong concealment, low power consumption, and eye safety; it only calculates the depth information of the entire scene through image data, and can be applied to different scenes at different distances by adjusting the focal length of the optical system, with a wide range of detection distances; it uses algorithms such as monocular depth estimation, scene semantic segmentation, polarization characteristic solution, and graph-based depth correction, and combines the advantages of monocular depth estimation and polarization characteristic solution to achieve pixel-level three-dimensional imaging while accurately restoring the three-dimensional detail information of the target of interest, with high three-dimensional imaging accuracy and rich detail textures in the area of interest. BRIEF DESCRIPTION OF THE DRAWINGS
[0075] Figure 1 A schematic flow chart of a composite passive three-dimensional imaging method provided in an embodiment of the present application;
[0076] Figure 2 A schematic flow chart of another composite passive three-dimensional imaging method provided in an embodiment of the present application;
[0077] Figure 3 A schematic diagram of four polarization states of an input polarization image provided in an embodiment of the present application;
[0078] Figure 4 A schematic flow chart of a step of calculating the second surface normal of all pixels provided in an embodiment of the present application;
[0079] Figure 5 A structural block diagram of a composite passive three-dimensional imaging device provided in an embodiment of the present application;
[0080] Figure 6 A structural block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0081] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined purpose, the technical solutions in the embodiments of the present application are clearly described. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present application.
[0082] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.
[0083] The description of the method flow in the specification of the present application and the steps of the flowchart in the drawings of the present specification do not have to be strictly executed according to the step numbers, and the method steps can be executed in a different order. Moreover, some steps can be omitted, multiple steps can be combined into one step, and / or one step can be decomposed into multiple steps.
[0084] The following is a detailed description of a composite passive three-dimensional imaging method, device, equipment and storage medium provided in the embodiments of the present application in combination with the accompanying drawings and preferred embodiments.
[0085] First, the application scenarios of the composite passive three-dimensional imaging method, device, equipment and storage medium of the embodiments of the present application are described in detail.
[0086] During the camera imaging process, only the two-dimensional projection of the three-dimensional scene is collected, and the important distance information of the three-dimensional scene is lost. If the distance data can be obtained, it will effectively improve the ability of computer vision tasks such as environmental perception and understanding.
[0087] Among the existing passive 3D imaging methods, the method based on polarization characteristic solution can retain the detailed information of the object surface, but this method realizes 3D reconstruction by integrating the normal vector of the object surface, so it can only process continuous object surfaces. For discontinuous surfaces or objects with complex geometric features, the reconstruction effect may be poor. In addition, the azimuth angle of the target surface normal is uncertain during the solution process. This uncertainty will directly affect the reconstruction accuracy of the 3D surface and cause distortion of the reconstructed surface. The uncertainty of the azimuth angle is further converted into the uncertainty of the direction of the normal gradient field. Fine surface features may be smoothed or lost during the reconstruction process, resulting in a decrease in the accuracy of the reconstruction result. Moreover, the method ultimately obtains the relative distance, so it needs further calibration and optimization, which increases the complexity of data processing and the computational cost.
[0088] Depth estimation methods based on deep learning can achieve pixel-level three-dimensional imaging, but the results lack detailed information about the scene. This is due to the limitations of the algorithm itself, which makes it difficult to capture and restore the fine details of the scene; therefore, its imaging accuracy is difficult to meet the needs of some application fields that require high-precision three-dimensional reconstruction.
[0089] Based on this, an embodiment of the present application provides a composite passive three-dimensional imaging method, which uses monocular depth estimation, scene semantic segmentation, polarization characteristic solution, and graph-based depth correction algorithms, and combines the advantages of monocular depth estimation and polarization characteristic solution to achieve pixel-level three-dimensional imaging while accurately restoring the three-dimensional detail information of the target of interest; thereby achieving passive, high-precision, low-cost, simple system, long detection distance, and pixel-level three-dimensional imaging.
[0090] See also Figure 1-2 , the present application embodiment provides a composite passive three-dimensional imaging method, such as Figure 1-2 As shown, the three-dimensional imaging method of the embodiment of the present application includes:
[0091] Step S1, obtaining an input image of a camera, extracting a polarization sub-image based on the input image, and synthesizing a plenoptic image.
[0092] See also Figure 3 This step is image preprocessing. Specifically, the input polarization image is preprocessed to obtain four polarization sub-images of 0 degree, 45 degree, 90 degree, and 135 degree and a full optical image.
[0093] In the specific implementation, the image preprocessing extracts the polarization sub-images I0, I1 at 0, 45, 90, and 135 degrees from the input image. 45 ,I 90 ,I 135 , and synthesize the plenoptic image I:
[0094]
[0095] Step S2, based on the polarization sub-image and the plenoptic image, calculate the second surface normals of all pixels; at the same time, based on the plenoptic image, obtain the initial absolute depth map of the input image through a monocular depth estimation network, extract the target area of interest through a scene semantic segmentation network and obtain its corresponding mask.
[0096] See also Figure 4 Specifically, in step S2, the surface normals of all pixels are calculated based on the polarization sub-image and the plenoptic image; specifically, the step includes:
[0097] Step S201, performing polarization characteristic calculation based on the polarization sub-graph to obtain the Stokes vector.
[0098] Polarization characteristics are solved by calculating the linear polarization component Q of the Stokes vector in the 0-degree direction and the linear polarization component U in the 45-degree direction through the four directional polarization sub-graphs:
[0099]
[0100] Step S202 , obtaining the polarization degree and azimuth angle by calculation based on the Stokes vector and the plenoptic image.
[0101] Calculate the polarization degree P and azimuth angle based on the Stokes vector Q, U and plenoptic image I
[0102]
[0103] Step S203, calculating the zenith angle of each pixel on the surface of the object.
[0104] Assuming the target reflection refractive index n = 1.4, calculate the zenith angle θ of each pixel on the object surface.
[0105]
[0106] j2=4P(n 2 +1)(j1-4P)
[0107] j3=j2 2 +16P 2 (16P 2 -j1 2 )(n 2 -1) 2
[0108]
[0109] In the formula, j1, j2, and j3 represent process variables and have no practical meaning.
[0110] Step S204, calculating the surface normals of all pixels of the input image based on the polarization characteristics
[0111]
[0112] According to the four polarization sub-images and the full optical image, the Stokes vector, polarization degree, surface normal azimuth, and surface normal zenith angle are calculated in turn, and finally the surface normal of the entire scene of the input image is calculated. That is, the second surface normal.
[0113] In step S2, a monocular depth estimation is performed based on the plenoptic image to obtain an initial absolute depth map depth ini; By performing scene semantic segmentation based on the plenoptic image, the target region of interest in the input image can be extracted and the mask corresponding to the target region of interest can be obtained.
[0114] When the monocular depth estimation step is implemented, the plenoptic image is input into a trained monocular depth estimation network to calculate the initial absolute depth map of the entire scene; the monocular depth estimation network is a model based on deep learning, which is used to automatically learn and extract features for depth estimation from a single image. The network extracts and transforms the input image through structures such as convolutional layers, pooling layers, and activation functions, and finally outputs a depth map of the same size as the input image, which is an existing technology. The absolute depth map provides the three-dimensional spatial position information of objects in the scene.
[0115] When the semantic segmentation step is implemented, the plenoptic image is input into a trained semantic segmentation network to calculate the object of interest in the plenoptic image and its mask; the semantic segmentation network is a model based on deep learning, which is used to automatically learn image features and perform pixel-level classification. The network extracts and transforms the input image features through structures such as convolutional layers, pooling layers, and upsampling layers, and finally outputs a segmentation map of the same size as the input image, which is a prior art. Through the processing of the semantic segmentation network, the precise position and shape information of the object of interest in the image can be obtained and represented in the form of a mask.
[0116] The mask is an image of the same size as the input image, which is used to specify a specific area in the image. It can be a binary or Boolean image of the same size as the original image, in which the selected area is marked as 1 (or True), and the remaining areas are marked as 0 (or False). The marking process is actually a "blocking" or "selecting" operation on the image, so as to perform a specific image processing operation on the selected area. In this embodiment, the specific area in the mask is the location and range of the target area of interest. Through the mask, the non-target area on the image is shielded so that it does not participate in the calculation.
[0117] In the specific implementation of the above step S2, since the processing process of calculating each pixel point of all pixels is faster, while the monocular depth estimation and semantic segmentation steps are relatively time-consuming, at the beginning of the system, the pixel normal calculation of each pixel point and the extraction process of the region of interest are performed synchronously. When the pixel normal calculation is completed, the purpose of extracting the region of interest of the input image can be quickly achieved.
[0118] Step S3, calculating a first surface normal of the target area of interest based on the initial absolute depth map, and correcting the second surface normal corresponding to the mask area according to the first surface normal.
[0119] Specifically, by calculating the two-dimensional array gradient of the initial absolute depth map in the target area of interest, the first surface normal of each pixel in the target area of interest can be obtained. That is, the gradient of the initial absolute depth map can be used to correct the second surface normal corresponding to the target area of interest. Direction of the second surface normal It is the second surface normal of all pixels calculated in step S2.
[0120] Further, correcting the second surface normal corresponding to the mask area according to the first surface normal includes:
[0121] If the first surface normal and the second surface normal If the angle between is ≤90°, If the first surface normal and the second surface normal If the angle between them is > 90°, in, is the second surface normal after correction.
[0122] Step S4, based on the corrected second surface normal, a relative depth map of the target area of interest is obtained, and three-dimensional reconstruction is performed on each mask area.
[0123] Specifically, the relative depth is the relative distance between pixels in the image, which is the relative distance between pixels in the image. Continuous integration can obtain the relative distance depth of each target area of interest rel , and then perform three-dimensional reconstruction.
[0124] Step S5, combining the intrinsic parameters of the camera, obtaining a first pseudo radar point of each mask area according to the initial absolute depth map and the mask, and obtaining a second pseudo radar point of each mask area according to the three-dimensional reconstruction result of the mask area;
[0125] Pseudo radar points refer to points similar to lidar point cloud data that are calculated through depth maps and camera intrinsic parameters. Based on the acquired depth information, each pixel point on the two-dimensional image is mapped to specific coordinates in three-dimensional space.
[0126] In the specific implementation, according to the camera internal parameters (such as focal length, optical center, etc.), the initial absolute depth map depth of the area of interest is ini Project into three-dimensional space to get three-dimensional point cloud point ini , that is, the first pseudo radar point; according to the camera intrinsic parameters, the relative distance depth of the area of interest is rel Project into three-dimensional space to get three-dimensional point cloud pointrel , which is the second pseudo radar point.
[0127] Step S6, uniformly selecting points from the first pseudo radar points as landmark points, and matching them with corresponding points in the second pseudo radar points to obtain landmark point matching point pairs;
[0128] The matching point pairs of two sets of point clouds are uniformly obtained in three-dimensional space, the landmark points are matched, and the corresponding relationship between them is established.
[0129] Step S7: construct a K-nearest-neighbor directed graph of the second pseudo radar point and calculate a weight matrix.
[0130] Construct the second pseudo radar point through K nearest neighbors rel The local graph of and the weight matrix are calculated.
[0131] Step S8: Based on the landmark point matching point pair, the second pseudo radar point is replaced with the first pseudo radar point in the weight matrix, the weight matrix is updated, the residual pixel depth of each mask area is corrected, and the high-precision depth of the mask area is obtained.
[0132] Replace the matching point in the local image with the first pseudo radar point point ini The landmark point that matches it in the update weight is obtained as the second pseudo radar point point rel The absolute distance point cloud point abs , update the depth of the landmark point; according to the camera internal parameters, the absolute distance point cloud point of the second pseudo radar point abs Project back to the camera plane to get the high-precision depth of the mask area cor .
[0133] Step S9: Replace the depth of the target area of interest in the initial absolute depth map with the high-precision depth of the mask area to obtain a final depth map, and perform three-dimensional imaging based on the final depth map.
[0134] The initial absolute depth map depth ini The depth of the target area of interest is replaced by the obtained high-precision depth depth cor , and obtain the final depth map, which is a high-precision full-scene depth map with clear details of the target area of interest.
[0135] Steps S7-S9 are graph-based depth correction, which projects the initial depth map and relative distance into three-dimensional space to obtain two sets of initial and relative point clouds, evenly obtain matching point pairs for the two sets of point clouds, construct K nearest neighbors for the relative point cloud and calculate the weight matrix, and calculate the high-precision three-dimensional point cloud after replacing the matching points in the weight matrix, so as to correct the area of interest of the initial depth and obtain a high-precision depth map with rich details.
[0136] For example: let n be the number of landmark points selected from the first pseudo radar point, and G be the distance, G∈R n , the number of the second pseudo radar points is n+m, the distance is Z, Z∈R n+m , where the first n are the distances between the landmark points and the matching points, and the last m are the distances between the remaining unmatched point pairs. m ∈R m It is expressed as the distance of m unmatched point pairs after the second pseudo radar point.
[0137] Construct a K-nearest-neighbor directed graph and use W ij Represents the weights of points i and j, and solves:
[0138]
[0139] satisfy:
[0140] Z-WZ=0;
[0141] W ij = 0 if j does not belong to N i ,
[0142] ∑ j W ij =1 pair
[0143] When the second pseudo radar point is replaced by the first pseudo radar point: Z is replaced by [G; Z m ], solve for Z' m Make
[0144]
[0145] Among them, Z' m The distance of the remaining points for correction.
[0146] In summary, the three-dimensional imaging method of the embodiment of the present application has the following advantages:
[0147] 1. This application adopts a passive three-dimensional imaging method to obtain full-scene depth information, and does not require active lighting such as laser and infrared. Therefore, it has the advantages of strong concealment, low power consumption and eye safety.
[0148] 2. This application only calculates the depth information of the entire scene through image data. Therefore, it can be applied to different scenes at different distances by adjusting the focal length of the optical system. It has the advantages of a wide detection distance range and multiple application scenarios.
[0149] 3. This application uses polarization characteristic solution and graph-based depth correction algorithm to correct the depth information of the region of interest, which has the advantages of high three-dimensional imaging accuracy and rich detail texture in the region of interest.
[0150] 4. This application obtains the depth value of the entire scene based on a single-frame image, so it has the characteristics of strong real-time performance, simple structure and easy integration.
[0151] See also Figure 5 , corresponding to the above-mentioned composite passive three-dimensional imaging method embodiment, the embodiment of the present application provides a composite passive three-dimensional imaging device, the composite passive three-dimensional imaging device comprising:
[0152] An image preprocessing module 1001 is used to obtain an input image of a camera, extract a polarization sub-image based on the input image, and synthesize a plenoptic image;
[0153] A polarization characteristic solving module 1002, configured to calculate the second surface normals of all pixels based on the polarization sub-image and the plenoptic image;
[0154] A monocular depth estimation module 1003, configured to obtain an initial absolute depth map of the input image through a monocular depth estimation network based on the plenoptic image;
[0155] The scene semantic segmentation module 1004 is used to extract the target region of interest through the scene semantic segmentation network and obtain its corresponding mask;
[0156] A normal correction module 1005 is used to calculate a first surface normal of the target area of interest based on the initial absolute depth map, and correct the second surface normal corresponding to the mask area according to the first surface normal;
[0157] A three-dimensional reconstruction module 1006 is used to obtain a relative depth map of the target region of interest based on the corrected second surface normal, and perform three-dimensional reconstruction on each of the mask regions;
[0158] A pseudo radar point conversion module 1007 is used to obtain a first pseudo radar point of each mask area according to the initial absolute depth map and the mask in combination with the intrinsic parameters of the camera, and obtain a second pseudo radar point of each mask area according to the three-dimensional reconstruction result of the mask area;
[0159] A landmark point matching module 1008 is used to uniformly select points from the first pseudo radar points as landmark points, and match them with corresponding points in the second pseudo radar points to obtain landmark point matching point pairs;
[0160] A depth correction and imaging module 1009 is used to construct a K-nearest neighbor directed graph of the second pseudo radar point and calculate a weight matrix;
[0161] Based on the landmark point matching point pair, replacing the second pseudo radar point with the first pseudo radar point in the weight matrix, updating the weight matrix, correcting the remaining pixel depth of each mask area, and obtaining a high-precision depth of the mask area;
[0162] Correcting the remaining pixel depth of each mask area according to the depth information of the absolute distance point cloud to obtain a high-precision depth of the mask area;
[0163] The depth of the target area of interest in the initial absolute depth map is replaced by the high-precision depth of the mask area to obtain a final depth map, and three-dimensional imaging is performed based on the final depth map.
[0164] Furthermore, the polarization characteristic solving module is specifically used for:
[0165] Polarization characteristics are calculated based on the polarization sub-graph to obtain the Stokes vector S:
[0166]
[0167] In the formula, I0, I 45 ,I 90 ,I 135 They are polarization subgraphs in the directions of 0, 45, 90, and 135 degrees, respectively. Q is the linear polarization component in the direction of 0, and U is the linear polarization component in the direction of 45.
[0168] The polarization degree P and the azimuth angle are calculated based on the Stokes vector and the plenoptic image.
[0169]
[0170] Calculate the zenith angle of each pixel on the object surface:
[0171]
[0172] j2=4P(n 2 +1)(j1-4P)
[0173] j3=j2 2 +16P 2 (16P 2 -j1 2 )(n 2 -1) 2
[0174]
[0175] Where j1, j2, and j3 represent process variables, and n is the target reflection refractive index;
[0176] Calculate the surface normals of all pixels in the input image based on polarization characteristics
[0177]
[0178] Furthermore, the normal correction module is specifically used for:
[0179] The first surface normal of each pixel in the target area of interest is obtained by calculating the two-dimensional array gradient of the initial absolute depth map in the target area of interest.
[0180] If the first surface normal and the second surface normal If the angle between is less than or equal to 90°, If the first surface normal and the second surface normal If the angle between is greater than 90°, in, is the second surface normal after correction.
[0181] The above-mentioned composite passive 3D imaging device implements the steps and various processes of the above-mentioned composite passive 3D imaging method embodiment and can achieve the same technical effect. To avoid repetition, they are not described again here.
[0182] See also Figure 6 Corresponding to the above-mentioned composite passive three-dimensional imaging method embodiment, an embodiment of the present application provides an electronic device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the steps of the above-mentioned composite passive three-dimensional imaging method embodiment and the various processes of the embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.
[0183] The memory 1010 can be used to store software programs and various data. The memory 1010 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instruction required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 1010 may include a volatile memory or a non-volatile memory, or the memory 1010 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 1010 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.
[0184] The processor 1011 may include one or more processing units; optionally, the processor 1011 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 1011.
[0185] Corresponding to the above-mentioned embodiment of the method for automatically generating preference data for secure alignment of a large language model, the embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by the processor, the steps and various processes of the above-mentioned embodiment of the method for automatically generating preference data for secure alignment of a large language model are implemented, and the same technical effect can be achieved. To avoid repetition, they will not be repeated here.
[0186] The processor is the processor in the electronic device described in the above embodiment of the present application. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.
[0187] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.
[0188] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.
[0189] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.
Claims
1. A composite passive three-dimensional imaging method, It is characterized in that include: Obtaining an input image from a camera, extracting a polarization sub-image based on the input image, and synthesizing a plenoptic image; Based on the polarization sub-image and the plenoptic image, the second surface normals of all pixels are calculated; at the same time, based on the plenoptic image, an initial absolute depth map of the input image is obtained through a monocular depth estimation network, and a target region of interest is extracted through a scene semantic segmentation network and its corresponding mask is obtained; Calculate a first surface normal of the target area of interest based on the initial absolute depth map, and correct the second surface normal corresponding to the mask area according to the first surface normal; Based on the corrected second surface normal, a relative depth map of the target region of interest is obtained, and three-dimensional reconstruction is performed on each mask region; In combination with the intrinsic parameters of the camera, according to the initial absolute depth map and the mask, a first pseudo radar point of each mask area is obtained, and according to the three-dimensional reconstruction result of the mask area, a second pseudo radar point of each mask area is obtained; Uniformly selecting points from the first pseudo radar points as landmark points, and matching them with corresponding points in the second pseudo radar points to obtain landmark point matching point pairs; Constructing a K-nearest-neighbor directed graph of the second pseudo radar point and calculating a weight matrix; Based on the landmark point matching point pair, replacing the second pseudo radar point with the first pseudo radar point in the weight matrix, updating the weight matrix, correcting the remaining pixel depth of each mask area, and obtaining a high-precision depth of the mask area; The depth of the target area of interest in the initial absolute depth map is replaced by the high-precision depth of the mask area to obtain a final depth map, and three-dimensional imaging is performed based on the final depth map.
2. The composite passive three-dimensional imaging method according to claim 1, It is characterized in that The step of extracting the polarization sub-image based on the input image and synthesizing the plenoptic image comprises: Extract polarization sub-images I at 0, 45, 90, and 135 degrees from the input image 0 ,I 45 ,I 90 ,I 135 , and synthesize the plenoptic image I:
3. The composite passive three-dimensional imaging method according to claim 2, It is characterized in that Calculating second surface normals of all pixels based on the polarization sub-image and the plenoptic image, comprising: Polarization characteristics are calculated based on the polarization sub-graph to obtain the Stokes vector S: Where, Q is the linear polarization component at 0 degrees, and U is the linear polarization component at 45 degrees; The polarization degree P and the azimuth angle are calculated based on the Stokes vector and the plenoptic image. Calculate the zenith angle of each pixel on the object surface: j 2 =4P(n 2 +1)(j 1 -4P) j 3 =j 2 2 +16P 2 (16P 2 -j 1 2 )(n 2 -1) 2 In the formula, j 1 、j 2 、j 3 represents the process variable, n is the target reflection refractive index; Calculate the surface normals of all pixels in the input image based on polarization characteristics 4. The composite passive three-dimensional imaging method according to claim 1, It is characterized in that The step of calculating the first surface normal of the target area of interest based on the initial absolute depth map, and correcting the second surface normal corresponding to the mask area according to the first surface normal, includes: The first surface normal of each pixel in the target area of interest is obtained by calculating the two-dimensional array gradient of the initial absolute depth map in the target area of interest. If the first surface normal and the second surface normal If the angle between is less than or equal to 90°, If the first surface normal and the second surface normal If the angle between is greater than 90°, in, is the second surface normal after correction.
5. The composite passive three-dimensional imaging method according to claim 1, It is characterized in that The first pseudo radar point and the second pseudo radar point are three-dimensional point clouds obtained by projecting corresponding depth information into three-dimensional space according to the intrinsic parameters of the camera.
6. A composite passive three-dimensional imaging device, It is characterized in that include: An image preprocessing module, used to obtain an input image of a camera, extract a polarization sub-image based on the input image, and synthesize a plenoptic image; A polarization characteristic solving module, used for calculating the second surface normals of all pixels based on the polarization sub-image and the plenoptic image; A monocular depth estimation module, configured to obtain an initial absolute depth map of the input image through a monocular depth estimation network based on the plenoptic image; The scene semantic segmentation module is used to extract the target region of interest through the scene semantic segmentation network and obtain its corresponding mask; A normal correction module, configured to calculate a first surface normal of the target area of interest based on the initial absolute depth map, and correct the second surface normal corresponding to the mask area according to the first surface normal; A three-dimensional reconstruction module, used to obtain a relative depth map of the target area of interest based on the corrected second surface normal, and perform three-dimensional reconstruction on each of the mask areas; a pseudo radar point conversion module, configured to obtain a first pseudo radar point of the target area of interest according to the initial absolute depth map in combination with the intrinsic parameters of the camera, and obtain a second pseudo radar point of each mask area according to the relative depth map of the target area of interest; A landmark point matching module, configured to uniformly select points from the first pseudo radar points as landmark points, and match them with corresponding points in the second pseudo radar points to obtain landmark point matching point pairs; A depth correction and imaging module, used to construct a K-nearest neighbor directed graph of the second pseudo radar point and calculate a weight matrix; Based on the landmark point matching point pair, replacing the second pseudo radar point with the first pseudo radar point in the weight matrix, updating the weight matrix, correcting the remaining pixel depth of each mask area, and obtaining a high-precision depth of the mask area; The depth of the target area of interest in the initial absolute depth map is replaced by the high-precision depth of the mask area to obtain a final depth map, and three-dimensional imaging is performed based on the final depth map.
7. The composite passive three-dimensional imaging method according to claim 6, It is characterized in that The polarization characteristic solving module is specifically used for: Polarization characteristics are calculated based on the polarization sub-graph to obtain the Stokes vector S: In the formula, I 0 ,I 45 ,I 90 ,I 135 They are polarization subgraphs in the directions of 0, 45, 90, and 135 degrees, respectively. Q is the linear polarization component in the direction of 0, and U is the linear polarization component in the direction of 45. The polarization degree P and the azimuth angle are calculated based on the Stokes vector and the plenoptic image. Calculate the zenith angle of each pixel on the object surface: j 2 =4P(n 2 +1)(j 1 -4P) j 3 =j 2 2 +16P 2 (16P 2 -j 1 2 )(n 2 -1) 2 In the formula, j 1 、j 2 、j 3 represents the process variable, n is the target reflection refractive index; Calculate the surface normals of all pixels in the input image based on polarization characteristics 8. The composite passive three-dimensional imaging method according to claim 6, It is characterized in that The normal correction module is specifically used for: The first surface normal of each pixel in the target area of interest is obtained by calculating the two-dimensional array gradient of the initial absolute depth map in the target area of interest. If the first surface normal and the second surface normal If the angle between is less than or equal to 90°, If the first surface normal and the second surface normal If the angle between is greater than 90°, in, is the second surface normal after correction.
9. An electronic device, It is characterized in that The electronic device comprises: a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the composite passive three-dimensional imaging method as claimed in any one of claims 1 to 5.
10. A readable storage medium, It is characterized in that The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the composite passive three-dimensional imaging method according to any one of claims 1 to 5 are implemented.
Citation Information
Cited By
Monocular human body distance measurement method and system based on AI depth and segmentation model
CN120339296A
