A multi-target scene polarization 3D imaging method based on deep learning
By building a clear and fuzzy distance estimation network module, the problem of low three-dimensional reconstruction accuracy in multi-objective scenarios is solved, and high-precision multi-objective polarization three-dimensional imaging is achieved. It is suitable for a simple hardware platform and provides spatial position information for multi-objective scenarios. .
Patent Information
- Application Number
- CN202210135322.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-02-14
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2042-02-14
AI Technical Summary
The prior art is difficult to accurately obtain the spatial relationships of multiple targets in multi-objective scenarios, resulting in distortion in three-dimensional reconstruction, and the voxel-based neural network three-dimensional reconstruction method has the problem of low model accuracy.
Using a multi-objective scene polarization three-dimensional imaging method based on deep learning, by constructing a clear network module and a fuzzy distance estimation network module, a polarization three-dimensional imaging system is used to obtain images of different polarization angles in multiple-objective scenes, and perform threshold segmentation and training to achieve multi-dimensional Clear reconstruction and depth estimation of the target scene, correct normal vector information, and reconstruct the three-dimensional contour of the object surface.
It realizes high-precision polarization three-dimensional imaging in multi-objective scenarios, provides the target's spatial position prior information, improves the accuracy and frequency details of three-dimensional reconstruction, and is suitable for hardware platforms with simple structures and fast computing speed.
Smart Images

Figure CN114663578B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of optical imaging, and in particular relates to a multi-target scene polarization three-dimensional imaging method based on deep learning. Background Art
[0002] As one of the methods for three-dimensional reconstruction of targets, polarization three-dimensional imaging has the characteristics of simple equipment, no need to modulate the light source, and rich high-frequency information of target three-dimensional reconstruction. It has become one of the most promising methods in the field of three-dimensional imaging technology. At the same time, in the field of computer vision, deep neural networks can obtain depth information of objects through training and show good performance. However, when performing polarization three-dimensional reconstruction of multi-target scenes, since there are two or more targets to be reconstructed in the scene, the optical detection system of a single camera and the method of inverting three-dimensional contour information by normal vector cannot accurately obtain the spatial relationship of multiple targets in the scene, resulting in distortion in the reconstruction at the edge positions of different targets. Therefore, it is necessary to further improve the reconstruction effect of three-dimensional reconstruction, reduce the complexity of the structure, and make a wider range of applications possible.
[0003] At present, a voxel-based neural network 3D reconstruction method is proposed. This method designs a 3D recursive reconstruction neural network (3D-R2N2) and uses deep learning to map from 2D images to their corresponding 3D voxel models. The model structure is Encoder + 3D LSTM (Long short-term memory) neural network + Decoder. Through the training of the data set, a single or multiple RGB images are input to the network, and the 3D expression of the target in the voxel space can be output. This method is applicable to both single view and multiple views. If it is multiple views, the multiple views are regarded as a sequence, input into the LSTM neural network, and multiple results are output. The advantage of this method is that the network does not require any image annotations or classification labels for training or testing, and only a single color picture is needed to generate a 3D image. The model can perform 3D reconstruction when there are too few feature points in the picture or the number of picture angles is insufficient. However, the disadvantage of the voxel-based neural network 3D reconstruction method is that the accuracy of the reconstructed model is low, and it cannot complete the fine 3D reconstruction task. Summary of the invention
[0004] In order to solve the above problems existing in the prior art, the present invention provides a multi-target scene polarization three-dimensional imaging method based on deep learning. The technical problem to be solved by the present invention is achieved by the following technical solutions:
[0005] The present invention provides a multi-target scene polarization three-dimensional imaging method based on deep learning, comprising:
[0006] S1: Use polarization 3D imaging system to obtain scene images of multi-target scenes at different polarization angles;
[0007] S2: constructing a neural network, the neural network comprising a sharpening network module and a fuzzy distance estimation network module, wherein the sharpening network module is used to sharpen and reconstruct targets at different object distances in the input threshold segmented polarization image to obtain a sharp image, and the fuzzy distance estimation network module is used to use the mapping relationship between the fuzzy image and the sharp image as prior information in a multi-target scene to estimate the target distance and obtain the estimated value of the distance between different targets and the camera;
[0008] S3: training the clarity network module and the fuzzy distance estimation network module in the neural network respectively to obtain a trained neural network model;
[0009] S4: using the trained neural network model to obtain depth estimation results and clear target images of different targets in scene images of multi-target scenes at different polarization angles, and identifying different target objects;
[0010] S5: Obtaining the polarization degrees of the surfaces of the different target objects, and the azimuth angles and incident angles of the incident light on the surfaces of the different target objects;
[0011] S6: correcting the normal vector obtained by polarization solution in the multi-target polarization three-dimensional imaging scene to obtain corrected azimuth information;
[0012] S7: Using the azimuth and incident angle information of different targets, the three-dimensional contour of the object surface in multi-target scenes can be reconstructed.
[0013] In one embodiment of the present invention, the S1 includes:
[0014] S11: using the polarization 3D imaging system to obtain the reflected light from the surface of the target object in the multi-target complex scene, and obtaining the original polarization images I′0, I′ of the object scene at four angles of 0°, 45°, 90° and 135° respectively. 45 , I′ 90 , I′ 135 ;
[0015] S12: Using a threshold segmentation algorithm to segment the original polarization images I′0, I′ 45 , I′ 90 , I′ 135 The object and background in the image are segmented to obtain the polarization images I0 and I 45 ,I 90 ,I 135 .
[0016] In one embodiment of the present invention, the clearing network module includes an interconnected encoding unit and a decoding unit, which is used to perform an encoding-decoding process on the input blurred image by introducing a fuzzy edge attention mechanism, so as to achieve clear reconstruction of the blurred target area in the blurred image.
[0017] In one embodiment of the present invention, the loss function of the neural network model includes the loss function of the clarity network module and the loss function of the fuzzy distance estimation network module, wherein the loss function of the clarity network module is:
[0018]
[0019] Where N represents the number of real blurred images input, α represents the weight value for adjusting the L1 loss, and SSIM(M f , M) represents the input real blurred image M and the clear image M finally output by the clear network module f The structural similarity results of
[0020] The loss function of the fuzzy distance estimation network module is:
[0021]
[0022] in, represents the output result of the prior information provided in the clarification network module under the kth level condition, β is the coefficient, represents the final fused image after the fuzzy distance estimation network module, represents the output result of image I after the k-th level of the sharpening network module, Φ rd (I) represents the output result of image I after the fuzzy distance estimation network module, and ||||2 represents the calculation of the binary norm.
[0023] In one embodiment of the present invention, S3 includes:
[0024] S31: Acquire a first training data set, where the first training data set is used to train the sharpening network module, and the first training data set includes a plurality of target images with different blur levels;
[0025] S32: training the clarity network module using the first training data set;
[0026] S33: Obtain a second training data set, where the second training data set is used to train the fuzzy distance estimation network module, wherein the second training data set includes a plurality of target images with different object distances;
[0027] S34: Using the second training data set to train the fuzzy distance estimation network module, thereby obtaining a trained neural network model.
[0028] In one embodiment of the present invention, the S5 includes:
[0029] S51: calculating the polarization degree of each target object surface using intensity information in four polarization sub-images of different target objects in the scene;
[0030] S52: using the intensity variation information of different target objects in the scene and the polarization degree to obtain the azimuth angle and the incident angle of the incident light on the surfaces of different target objects.
[0031] In one embodiment of the present invention, the S6 includes:
[0032] By using the changing relationship between the target reflected light intensity gradient field and the contour, the normal vector information of the target object surface is converted into gradient field information. The correction process of the normal vector multi-value problem in the multi-target polarization 3D imaging scene is expressed as:
[0033]
[0034] in, represents the gradient field change reference information obtained by solving the target surface intensity information, G polar represents the gradient field parameter obtained by using the polarization information of the target reflected light;
[0035] like The microfacet normal vector information directly obtained by using the polarization characteristics of the target reflected light is accurate. The azimuth angle value is flipped 180°.
[0036] Another aspect of the present invention provides a storage medium, in which a computer program is stored, and the computer program is used to execute the steps of the multi-target scene polarization three-dimensional imaging method based on deep learning described in any one of the above embodiments.
[0037] Another aspect of the present invention provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the processor calls the computer program in the memory, it implements the steps of the multi-target scene polarization three-dimensional imaging method based on deep learning as described in any of the above embodiments.
[0038] Compared with the prior art, the present invention has the following beneficial effects:
[0039] 1. The polarization 3D imaging method for multi-target scenes based on deep learning of the present invention, based on a single-camera polarization 3D imaging model, constructs a mapping relationship between the target edge blur degree and the object distance at different depths, and uses a deep learning-based method to obtain the spatial position relationship information of multiple targets, thereby providing the spatial position prior information of the targets in the multi-target scene for polarization 3D imaging, and realizing polarization 3D imaging of diffuse reflection targets in the multi-target scene.
[0040] 2. The present invention designs a deep neural network to perform defocusing and sharpening of input multi-target images at different spatial positions, and estimates the spatial position, providing prior information for multi-target polarization 3D imaging, and realizing high-precision polarization 3D imaging of multi-target scenes. Combining deep learning with polarization 3D imaging avoids the need for too many data sets for training, and the data is relatively simple and the operation speed is fast. At the same time, it has the advantages of utilizing the strong statistical characteristics of deep learning and the high precision and high-frequency details of polarization 3D imaging, making it possible to apply it on hardware platforms with simple structure and low performance.
[0041] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] Figure 1 is a flow chart of a multi-target scene polarization three-dimensional imaging method based on deep learning provided by an embodiment of the present invention;
[0043] Figure 2 It is a structural schematic diagram of a multi-target complex scene polarization three-dimensional imaging system provided by an embodiment of the present invention;
[0044] Figure 3 is a structural schematic diagram of a neural network model provided by an embodiment of the present invention;
[0045] Figure 4 is a schematic structural diagram of a coding unit provided by an embodiment of the present invention;
[0046] Figure 5 is a structural schematic diagram of a decoding unit provided by an embodiment of the present invention;
[0047] Figure 6 It is a schematic diagram of the azimuth angle and incident angle of incident light on the surface of an object provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0048] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined purpose of the invention, the following is a detailed description of a multi-target scene polarization three-dimensional imaging method based on deep learning proposed by the present invention in combination with the accompanying drawings and specific implementation methods.
[0049] The above and other technical contents, features and effects of the present invention are clearly presented in the following detailed description of the specific implementation modes in conjunction with the accompanying drawings. Through the description of the specific implementation modes, the technical means and effects adopted by the present invention to achieve the predetermined purpose can be more deeply and specifically understood. However, the attached drawings are only for reference and explanation purposes and are not used to limit the technical solutions of the present invention.
[0050] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants are intended to cover non-exclusive inclusion, so that an article or device including a series of elements includes not only those elements, but also other elements that are not explicitly listed. In the absence of more restrictions, the elements defined by the statement "including one..." do not exclude the existence of other identical elements in the article or device including the elements.
[0051] See also Figure 1 , Figure 1 : is a flow chart of a multi-target scene polarization three-dimensional imaging method based on deep learning provided by an embodiment of the present invention. The imaging method includes:
[0052] S1: Use the polarization 3D imaging system to obtain scene images of multi-target scenes at different polarization angles.
[0053] Specifically, we first build a polarization 3D imaging system for multi-target complex scenes. Figure 2 , Figure 2 1 is a schematic diagram of the structure of a multi-target complex scene polarization three-dimensional imaging system provided by an embodiment of the present invention. The imaging system includes a polarization detector and a slide rail located in front of the polarization detector, and different targets are set on the slide rail and can slide back and forth along the slide rail. The imaging system is then used to obtain scene images of the multi-target scene at different angles.
[0054] Specifically, the S1 includes:
[0055] S11: using the polarization 3D imaging system to obtain the reflected light from the surface of the target object in the multi-target complex scene, and respectively obtaining polarization images I′0, I′ of the object scene at four angles of 0°, 45°, 90° and 135° 45 , I′ 90 , I′ 135 .
[0056] Specifically, in a natural light environment, the CMOS camera in the polarization detector of the polarization 3D imaging system is used to collect the reflected light from the surface of the object, so that the polarization images I′0, I′1 of the object scene at four angles of 0°, 45°, 90° and 135° are obtained. 45 , I′ 90 , I′ 135 .
[0057] S12: Using a threshold segmentation algorithm to segment the polarization images I′0, I′ 45 , I′ 90 , I′ 135 The object and background in the image are segmented to obtain the polarization images I0 and I 45 ,I 90 ,I 135 .
[0058] Specifically, in order to reduce the amount of calculation, it is necessary to segment the acquired polarization image of the object scene and the scene depth information respectively, separate the object from the background, and remove the background part. Therefore, this embodiment uses a threshold segmentation algorithm to separate the polarization image of the object scene from the background. The polarization images after threshold segmentation are I0, I 45 ,I 90 ,I 135 .
[0059] S2: Constructing a neural network and training the neural network to obtain a trained neural network model.
[0060] See also Figure 3 , Figure 3 It is a structural schematic diagram of a neural network model provided by an embodiment of the present invention. The neural network is mainly composed of a sharpening network module and a fuzzy distance estimation network module, wherein the sharpening network module is used to sharpen and reconstruct the input polarization image after threshold segmentation to obtain a sharp image; the fuzzy distance estimation network module is used to use the mapping relationship between the fuzzy image and the sharp image as prior information in a multi-target scene to estimate the target distance and obtain the estimated value of the distance between different targets and the camera.
[0061] This embodiment adopts a supervised learning method and utilizes the mapping relationship between the image blur kernel (edge blur radius) and the image object distance to achieve multi-target image clarity under different object distance conditions and estimate the distance between the target and the camera at different spatial positions.
[0062] Specifically, first, a blur-clear end-to-end network architecture, i.e., a clearing network module, is constructed. Specifically, the clearing network module 1 of this embodiment includes an encoding unit 11 and a decoding unit 12 connected to each other, which are used to perform an encoding-decoding process on the input blurred image by introducing a fuzzy edge attention mechanism to achieve clear reconstruction of the blurred target area. For further information, please refer to Figure 2 , the encoding unit 11 includes a plurality of downsampling units connected in sequence, and the original image input to the decoding unit 11 undergoes a series of dimensionality reduction feature learning of the plurality of downsampling units to obtain a fixed highly abstract but low-dimensional feature vector, such as Figure 3 As shown in , the input 1024×1024 image is downsampled to a 32×32 feature image. For further information, see Figure 4 , Figure 4 It is a structural diagram of a downsampling unit provided in an embodiment of the present invention. The structures of the multiple downsampling units are the same, and all include a convolution layer, a normalization layer, an activation layer, a convolution layer, a normalization layer, an activation layer and a pooling layer arranged in sequence from the data input end to perform dimensionality reduction processing on the input image to obtain a feature image after dimensionality reduction.
[0063] Furthermore, the decoding unit 12 of this embodiment includes a plurality of upsampling units connected in sequence, and the image input to the decoding unit 12 undergoes a series of dimension-upgraded feature learning of the plurality of upsampling units to obtain an output image with the same size as the original image, such as Figure 3 As shown, the 32×32 image output by the encoding unit 11 is restored to a clear image of 1024×1024. Figure 5 , Figure 5 The present invention provides a schematic diagram of an upsampling unit structure. The upsampling unit includes a transposed convolution layer, a normalization layer, an activation layer, a convolution layer, a normalization layer, an activation layer, a convolution layer, a normalization layer, and an activation layer arranged in sequence from a data input end.
[0064] After the clearing process of multiple blurred areas in the multi-target scene is reconstructed, the mapping relationship between the blurred image and the clear image is used as the prior information in the multi-target scene and input into the subsequent fuzzy distance estimation network module for target distance estimation. In this way, under the condition of using a relatively simple network architecture, accurate distance estimation can be performed on objects at different spatial positions in the multi-target scene, and the distance between different objects and the CCD camera can be obtained, providing reliable distance information for the three-dimensional reconstruction process of the multi-target scene.
[0065] Furthermore, the loss function of the neural network model is defined as the predicted combination of defocus-sharpening estimation and multi-target distance estimation. For defocus-sharpening estimation, the loss function of the blur image sharpening process is defined using structural similarity (SSIM), which compares the input real blur image M with the final sharp image M output by the sharpening network module. f For comparison, the process can be expressed as:
[0066]
[0067] Where N represents the number of real blurred images input, α represents the weight value for adjusting the L1 loss, and SSIM can be defined as:
[0068]
[0069] Among them, μ represents the average brightness of the image, that is, μ Mf Represents the average brightness of the output clear image, μ M Represents the average brightness of the input real blurred image M, C1=(K1L) 2 , C2=(K2L) 2 It is used to prevent the denominator from being zero. K1 and K2 are both constants. L represents the grayscale dynamic range of the image. σ represents the degree of brightness change of the image, that is, the standard deviation of the image pixel value. To clarify the final output image M of the network module f The variance of is the variance of the input real blurred image M, is the real blurred image M and the clear image M f Preferably, K1=0.01, K2=0.03.
[0070] In the process of scene multi-target distance estimation, it is assumed that the depth information obtained in the output of the clarity network module is Φ rd , the distance estimation result obtained after the image passes through the fuzzy distance estimation network module is Φ d In the clear network module, there is a mapping relationship between the edge step function of the clear target and the edge step function of the blurred target regarding the object distance. Therefore, the information output by the clear network module is used as the prior information for distance estimation in the fuzzy distance estimation network module. Through the deep distillation network cascade technology, the prior information is introduced into the fuzzy distance estimation network module. The k-th level distance estimation loss function in the fuzzy distance estimation network module can be expressed as:
[0071]
[0072] in, represents the output result of the prior information provided in the clarification network module under the kth level condition, β is a coefficient, which is usually 1, represents the final fused image after the fuzzy distance estimation network module, represents the output result of image I after the k-th level of the sharpening network module, Φ rd (I) represents the output result of image I directly passing through the fuzzy distance estimation network module, and ||||2 represents the calculation of the binary norm.
[0073] It should be noted that, as described above, the clear network module 1 of this embodiment includes an encoding unit 11 and a decoding unit 12 connected to each other, the encoding unit 11 includes a plurality of downsampling units connected in sequence, and the decoding unit 12 includes a plurality of upsampling units connected in sequence, each downsampling unit and each upsampling unit are a level of the clear network module 1, and the level of the clear network module 1 is the sum of the number of downsampling units and upsampling units. Figure 3 As shown, the clarity network module 1 of this embodiment includes 9 levels.
[0074] S3: training the clarity network module and the fuzzy distance estimation network module in the neural network respectively to obtain a trained neural network model.
[0075] Specifically, S3 includes:
[0076] S31: Acquire a first training data set, where the first training data set is used to train the sharpening network module, wherein the first training data set includes a plurality of target images with different blur levels.
[0077] Specifically, the slider on the slide rail of the imaging system is moved to drive the target to move from a position of 610mm to a position of 1010mm from the polarization detector. By adjusting the focal length of the camera, the target surface at an object distance of 610mm can be clearly imaged. As the slider moves away from the polarization detector, polarization image information is acquired for scenes with a distance interval of 10mm. As the detection distance increases, the blurriness at the edge of the target becomes higher and higher. This blur information will be utilized and input into the neural network model for training. In addition, for the blur image sharpening process, target information at different object distances is also collected at intervals of 10mm, and the camera lens is randomly adjusted at different positions so that target images with different blurriness are obtained while the focal length remains unchanged, so as to ensure strong robustness for target sharpening in actual application. A total of 2952 training images were collected in the process.
[0078] S32: Using the first training data set to train the clarity network module.
[0079] Specifically, the clearing network module and the fuzzy distance estimation network module of this embodiment are trained separately. First, the fuzzy distance estimation network module is fixed, and the clearing network module is trained. The images in the first training data set are preprocessed, adjusted to a resolution of 1024×1024, and then input into the clearing network module for training. After the training is completed, a clearing network module capable of clearing the input fuzzy image is obtained.
[0080] S33: Obtain a second training data set, where the second training data set is used to train the fuzzy distance estimation network module, wherein the second training data set includes a plurality of target images with different object distances.
[0081] Specifically, in the process of acquiring training images for the fuzzy distance estimation network module, the parameters of the polarization detector are kept unchanged, and images of targets in the scene at different object distances are acquired. The corresponding object distance can be expressed as D∈[610mm,10mm,1010mm], and 492 training images are acquired, which together constitute the second training data set.
[0082] S34: Using the second training data set to train the fuzzy distance estimation network module, thereby obtaining a trained neural network model.
[0083] Specifically, the parameters of the sharpening network module are fixed, and the 492 training images in the second training data set are preprocessed to a resolution of 1024×1024, and then input into the trained sharpening network module to obtain a sharpened image output, and the input original image and the image corresponding to the output of the sharpening network module are used as a mapping relationship, and output to the fuzzy distance estimation network module for training, and the output result is the depth estimation of the target object. The trained fuzzy distance estimation network module has the function of predicting the depth information of the target edge.
[0084] By training the clarity network module and the fuzzy distance estimation network module respectively, a trained neural network model is obtained.
[0085] S4: Using the trained neural network model, depth estimation results of different targets in scene images of a multi-target scene with different polarization angles are obtained to identify different target objects.
[0086] Specifically, by inputting the scene images of the multi-target scene with different polarization angles obtained in step S1 into the trained neural network model, the depth estimation results of different target objects in the image can be obtained, providing spatial prior information for the reconstruction process. If there is a sudden change in the contour depth, it indicates another target object here, so that different target objects in the image can be identified.
[0087] S5: Obtain the polarization degrees of the surfaces of the different target objects, and the azimuth angles and incident angles of the incident light on the surfaces of the different target objects.
[0088] Further, step S5 includes:
[0089] S51: Calculate the polarization degree of the surface of each target object using the intensity information in the four polarization sub-images of different target objects in the scene.
[0090] Specifically, the Stokes vector representation is a commonly used method for representing polarization characteristics, which means that the polarization state of a beam of light can be fully described by four fixed parameters, which are called Stokes vectors. Since each Stokes parameter is represented by light intensity, it can be directly measured by certain optoelectronic instruments. The Stokes vector can be expressed as:
[0091]
[0092] Among them, E x and E y Respectively represent the components of the electric field vector of the reflected light on the surface of the object on the x-axis and y-axis. The x-axis and y-axis represent two orthogonal directions, and the x-axis is based on the polarization detector. L -I R It represents the difference between the intensity of left-handed circularly polarized light and the intensity of right-handed circularly polarized light. Re<> means taking the real part, and Im<> means taking the imaginary part.
[0093] The calculation formula of the polarization degree P expressed by the Stokes vector is:
[0094]
[0095] In summary, the polarization degrees of the surfaces of different target objects in the scene image can be obtained according to the calculation formula of the polarization degree.
[0096] S52: Using the intensity variation information of different target objects in the scene and the polarization degree, obtaining the azimuth angle of the incident light on the surface of different target objects and the angle of incidence θ, which is the polar coordinate of the normal to the surface of the object and θ, such as Figure 6 shown.
[0097] In this embodiment, step S52 includes:
[0098] S521: Using polarization images I0, I 45 ,I 90 , and I 135 , calculate the azimuth of the incident light on the object surface The calculation formula is:
[0099]
[0100] It should be noted that, in the process of rotating the polarizer in the polarization detector, the light intensity of the polarization images obtained at two rotation angles of 180° is the same. Therefore, in the calculation result, there is an uncertainty of 180° between the incident azimuth of the incident light on the surface of the object to be reconstructed and the incident azimuth of the actual incident light, which leads to the uncertainty of the direction of the surface normal of the object obtained from the polarization information. Therefore, it is necessary to correct the surface normal n 1 .
[0101] S522: Using polarization images I0, I 45 ,I 90 , and I 135 , the incident angle θ of the incident light on the object surface is calculated.
[0102] As mentioned above, the trained neural network model is used to estimate the distance of different target contours, and the estimated value of the distance between the target and the camera at different spatial positions is obtained to provide spatial prior information for the reconstruction process. If there is a sudden change in the depth of the contour, it means that there is another target object here. The zenith angle θ of the target from the edge of the contour is T and azimuth To solve the problem, for scenarios with multiple purposes, the process can be expressed as:
[0103]
[0104] in, represents intensity images with different polarization azimuth angles, T represents different continuous surface targets at different spatial positions, X represents [0, 45, 90, 135], P T It represents the polarization degree of different target objects, and n is the refractive index of different target objects.
[0105]
[0106] Among them, Λ represents a binary operator with a value of {0,1}. The above formula can be used to solve the azimuth and zenith angle information of the surface of the object in the multi-target scene, and obtain the normal vector information of each pixel point of the target. However, in this process, there is still the multi-value problem of the normal vector, and it is necessary to determine the value of the binary operator Λ to achieve the correct solution of the normal vector direction of the target surface.
[0107] S6: Correct the normal vector in the multi-target polarization three-dimensional imaging scene to obtain corrected azimuth information.
[0108] Specifically, the relationship between the intensity gradient field of the target reflected light and its contour can provide a reference for the value of the binary operator Λ. By converting the normal vector information into gradient field information, the correction process of the normal vector multi-value problem in the multi-target polarized 3D imaging scene can be expressed as:
[0109]
[0110] in, represents the gradient field change reference information obtained by solving the target surface intensity information, G polar represents the gradient field parameter obtained by using the polarization information of the target reflected light. This indicates that the estimated azimuth is accurate, that is, the microfacet normal vector information directly obtained by using the polarization characteristics of the target reflected light is accurate; on the contrary, if It is necessary to correct the azimuth angle obtained by using the polarization characteristics to achieve a 180° flip of the azimuth angle value. The azimuth angle information correction of the micro-facet on the target surface can be expressed by the following formula:
[0111]
[0112] in, represents the azimuth information of the corrected target microfacet, represents the azimuth before correction. Finally, the normal vector information of the microfacet on the target surface can be uniquely solved.
[0113] S7: Reconstruct the three-dimensional contour of the target by using the azimuth and incident angle information of different targets.
[0114] Specifically, for target 3D contour reconstruction in multi-target complex scenes, the integration process can be expressed as:
[0115] Z(u)={Z1(U),Z2(U),...,Z T (U)}
[0116] Among them, Z T (U) represents the Tth target in a multi-target scenario and can be written as:
[0117]
[0118] Among them, M T and N T They represent the number of pixels occupied by the target at different spatial positions, and their relationship with the number of pixels of the entire image acquired by the detector is: {M T,N T}∈{M,N}, represents the distance information of different targets obtained by the network model, and ε represents the scale factor related to the real height information of the target. μ and ν represent the Fourier coefficients of the discrete differential operator in the x-direction and y-direction, respectively. Thus, polarization 3D imaging of multi-target complex scenes can be realized.
[0119] The embodiment of the present invention is a multi-target scene polarization three-dimensional imaging method based on deep learning. On the basis of a single-camera polarization three-dimensional imaging model, by constructing a mapping relationship between the target edge blur degree and the object distance at different depths, a method based on deep learning is used to obtain spatial position relationship information of multiple targets, and provide spatial position prior information of the target in the multi-target scene for polarization three-dimensional imaging, so as to realize polarization three-dimensional imaging of diffuse reflection targets in the multi-target scene. In addition, this embodiment designs a deep neural network to perform defocusing and sharpening of input multi-target images at different spatial positions, and performs spatial position depth estimation, so as to provide prior information for multi-target polarization three-dimensional imaging, and realize high-precision polarization three-dimensional imaging of multi-target scenes. Combining deep learning with polarization three-dimensional imaging avoids the need for too many data sets for training, and the data is relatively simple and the operation speed is fast. At the same time, it has the advantages of utilizing the strong statistical characteristics of deep learning and the high precision and rich high-frequency details of polarization three-dimensional imaging, making it possible to apply it on hardware platforms with simple structure and low performance.
[0120] Another embodiment of the present invention provides a storage medium, in which a computer program is stored, and the computer program is used to execute the steps of the multi-target scene polarization three-dimensional imaging method based on deep learning described in the above embodiment. Another aspect of the present invention provides an electronic device, including a memory and a processor, in which a computer program is stored, and when the processor calls the computer program in the memory, the steps of the multi-target scene polarization three-dimensional imaging method based on deep learning as described in the above embodiment are implemented. Specifically, the above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including several instructions for an electronic device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform some steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.
[0121] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.
Claims
1. A multi-target scene polarization 3D imaging method based on deep learning, It is characterized in that include: S1: Use polarization 3D imaging system to obtain scene images of multi-target scenes at different polarization angles; S2: constructing a neural network, the neural network comprising a sharpening network module and a fuzzy distance estimation network module, wherein the sharpening network module is used to sharpen and reconstruct targets at different object distances in the input threshold segmented polarization image to obtain a sharp image, and the fuzzy distance estimation network module is used to use the mapping relationship between the fuzzy image and the sharp image as prior information in a multi-target scene to estimate the target distance and obtain the estimated value of the distance between different targets and the camera; S3: training the clarity network module and the fuzzy distance estimation network module in the neural network respectively to obtain a trained neural network model; S4: using the trained neural network model to obtain depth estimation results and clear target images of different targets in scene images of multi-target scenes at different polarization angles, and identifying different target objects; S5: Obtaining the polarization degrees of the surfaces of the different target objects, and the azimuth angles and incident angles of the incident light on the surfaces of the different target objects; S6: Correcting the normal vector obtained by polarization solution in the multi-target polarization three-dimensional imaging scene to obtain corrected azimuth information; S7: Using the azimuth and incident angle information of different targets, the 3D contour of the object surface in multi-target scenes can be reconstructed; The sharpening network module includes an encoding unit and a decoding unit connected to each other, and is used to perform an encoding-decoding process on the inputted blurred image by introducing a fuzzy edge attention mechanism, so as to realize sharpening and reconstruction of the blurred target area in the blurred image; The loss function of the neural network model includes the loss function of the clarity network module and the loss function of the fuzzy distance estimation network module, wherein the loss function of the clarity network module is: Where N represents the number of real blurred images input, α represents the weight value for adjusting the L1 loss, and SSIM(M f , M) represents the input real blurred image M and the clear image M finally output by the clear network module f The structural similarity results of The loss function of the fuzzy distance estimation network module is: in, represents the output result of the prior information provided in the clarification network module under the kth level condition, β is the coefficient, represents the final fused image after the fuzzy distance estimation network module, represents the output result of image I after the k-th level of the sharpening network module, Φ rd (I) represents the output result of image I after the fuzzy distance estimation network module, ║ ║ 2 It means to find the second norm; The S6 includes: By using the changing relationship between the target reflected light intensity gradient field and the contour, the normal vector information of the target object surface is converted into gradient field information. The correction process of the normal vector multi-value problem in the multi-target polarization 3D imaging scene is expressed as: in, represents the gradient field change reference information obtained by solving the target surface intensity information, G polar represents the gradient field parameter obtained by using the polarization information of the target reflected light; like The microfacet normal vector information directly obtained by using the polarization characteristics of the target reflected light is accurate. The azimuth angle value is flipped 180°.
2. The multi-target scene polarization three-dimensional imaging method based on deep learning according to claim 1, It is characterized in that The S1 includes: S11: Using the polarization three-dimensional imaging system to obtain the reflected light from the surface of the target object in the multi-target complex scene, and respectively obtain the original polarization images I of the object scene at four angles of 0°, 45°, 90° and 135° 0 ′、I 4 ' 5 ,I 9 ' 0 ,I 1 ' 35 ; S12: Using a threshold segmentation algorithm to segment the original polarization image I 0 ′、I 4 ' 5 ,I 9 ' 0 ,I 1 ' 35 The object and background in the image are segmented to obtain the polarization image I after threshold segmentation. 0 ,I 45 ,I 90 ,I 135 .
3. The multi-target scene polarization three-dimensional imaging method based on deep learning according to claim 1, It is characterized in that The S3 includes: S31: Acquire a first training data set, where the first training data set is used to train the sharpening network module, and the first training data set includes a plurality of target images with different blur levels; S32: training the clarity network module using the first training data set; S33: Obtain a second training data set, where the second training data set is used to train the fuzzy distance estimation network module, wherein the second training data set includes a plurality of target images with different object distances; S34: Using the second training data set to train the fuzzy distance estimation network module, thereby obtaining a trained neural network model.
4. The multi-target scene polarization three-dimensional imaging method based on deep learning according to claim 3, It is characterized in that The S5 includes: S51: calculating the polarization degree of each target object surface using intensity information in four polarization sub-images of different target objects in the scene; S52: using the intensity variation information of different target objects in the scene and the polarization degree to obtain the azimuth angle and the incident angle of the incident light on the surfaces of different target objects.
5. A storage medium, It is characterized in that The storage medium stores a computer program, which is used to execute the steps of the multi-target scene polarization three-dimensional imaging method based on deep learning as described in any one of claims 1 to 4.
6. An electronic device, It is characterized in that It comprises a memory and a processor, wherein a computer program is stored in the memory, and when the processor calls the computer program in the memory, the steps of the multi-target scene polarization three-dimensional imaging method based on deep learning as claimed in any one of claims 1 to 4 are implemented.
