Target surface normal reconstruction method and apparatus

Abstract

The application provides a target surface normal reconstruction method and device, wherein the method comprises the following steps: obtaining an approximate normal map reconstructed based on polarization information of a target surface; splicing the approximate normal map and an original polarization image in a channel dimension to obtain a spliced image; inputting the spliced image into a neural network which is pre-trained and based on a mapping relationship between a polarization image and a surface normal through deep learning, and outputting a remapped target surface normal map. According to the scheme, the ambiguity problem of the surface normal zenith angle and the azimuth angle can be solved, and reasonable target three-dimensional topography reconstruction can be realized.

Description

Technical Field

[0001] The present invention relates to the field of target three-dimensional structure reconstruction technology, and in particular to a method and apparatus for reconstructing target surface normals. Background Technology

[0002] In recent years, by observing the polarization characteristics of reflected light from the surface of an object, the shape information of the object can be obtained. Furthermore, the polarization characteristic of "enhancing weak light and weakening strong light" means that even if the external light source is not ideal, relatively clear polarization information can still be obtained, thereby enabling the reconstruction of the three-dimensional surface of the object.

[0003] Current 3D reconstruction techniques based on target polarization information calculate the surface normal vector by reflecting the shape curvature of the target surface through the polarization characteristics of the reflected light. However, when reconstructing the surface normal using specularly reflected polarized light, the degree of polarization does not change monotonically with the incident angle, leading to ambiguity in the zenith angle of the normal. This results in discontinuities or deformations in the 3D structure of the target. Furthermore, the refractive index of the target surface must be considered, and the surface is easily affected by object texture and environmental noise, making it difficult to ensure the reliability of the target structure.

[0004] Therefore, a new method for reconstructing the surface normals of a target is needed to construct a reasonable three-dimensional structure of the target. Summary of the Invention

[0005] This invention provides a method and apparatus for reconstructing the surface normal of a target, which can solve the ambiguity problem between the zenith angle and azimuth angle of the surface normal and achieve reasonable reconstruction of the three-dimensional shape of the target.

[0006] In a first aspect, embodiments of the present invention provide a method for reconstructing the normals of a target surface, comprising:

[0007] Obtain the approximate normal map reconstructed based on the polarization information of the target surface;

[0008] The approximate normal map and the original polarization image are stitched together in the channel dimension to obtain the stitched image;

[0009] The stitched image is input into a pre-trained neural network, which outputs a remapped target surface normal map based on the mapping relationship between the polarization image and the surface normals learned through deep learning.

[0010] In one possible implementation, the training method for the neural network includes:

[0011] Multiple sample pairs of the standard object are obtained by using the standard object as the target; each sample pair includes a sample approximate normal map, a sample polarization image, and a sample surface normal map; the sample approximate normal map is reconstructed based on the polarization information of the standard object surface;

[0012] For each sample pair, the approximate normal map and polarization image of the sample in the sample pair are stitched together in the channel dimension to obtain a stitched image. The stitched image is used as input and the surface normal map of the sample in the sample pair is used as output to train the neural network; thus, a trained neural network is obtained using multiple sample pairs.

[0013] In one possible implementation, the neural network comprises, from input to output, a first convolutional layer, a lightweight convolutional module, an efficient attention module, and a second convolutional layer.

[0014] The neural network, based on the mapping relationship between polarization images and surface normals learned through deep learning, outputs a remapped target surface normal map, including:

[0015] The spatial features of the stitched image are extracted using the first convolutional layer;

[0016] The lightweight convolutional module is used to extract deep features based on the spatial features;

[0017] The efficient attention module is used to perform feature dimensionality reduction based on the deep features. Matrix parameters in the correlation model are generated based on the dimensionality-reduced features, and these matrix parameters are substituted into the correlation model to output the correlation features. The correlation model is generated by the efficient attention module after training.

[0018] The spatial features are remapped using the second convolutional layer based on the correlation features to obtain the target surface normal map.

[0019] In one possible implementation, the lightweight convolutional module includes: multiple residual feature blocks; the residual feature blocks include: two residual units connected in parallel and a 1×1 convolutional layer and a 3×3 convolutional layer connected in series after the two residual units connected in parallel;

[0020] The step of using the lightweight convolution module to extract deep features based on the spatial features includes:

[0021] Each of the residual feature blocks is used to extract depth features F using the following formula. R (X f ):

[0022] F R (X f )=f 3×3 (f 1×1 (cat c (RU(X f ),RU(RU(X f )))))+Xf

[0023] RU(X f )=f k×k (σ(f k×k (X f )))+X f

[0024] Among them, cat c (·) represents splicing on the channel dimension, RU(·) represents the residual unit, X f Characterizing the spatial features, f k×k (·) represents the convolution function with a kernel size of k×k, and σ(·) represents the activation function.

[0025] In one possible implementation, the correlation model includes:

[0026] Y = e(AV)

[0027] A = soft max(QK) T )

[0028] Where Y represents the relevance feature, e(·) represents the dimensionality increase operation, A represents the attention feature, V represents the value matrix, softmax represents the normalization function, Q represents the query matrix, and K represents the key matrix.

[0029] In one possible implementation, the approximate normal map reconstructed based on the polarization information of the target surface includes:

[0030] Infrared radiation images of the target in a set polarization direction are obtained based on a polarization imaging measurement device;

[0031] Stokes parametric images S0, S1, and S2 were calculated using the infrared radiation images;

[0032] Calculate the polarization degree image of the target surface based on the Stokes parametric image;

[0033] The polarization degree image is used to determine the correlation between polarization degree, zenith angle, and object refractive index, so as to roughly reconstruct the normal map of the target surface based on the correlation and obtain an approximate normal map.

[0034] In one possible implementation, the original polarization image is at least one of the Stokes parametric image and the polarization degree image.

[0035] Secondly, embodiments of the present invention also provide a target surface normal reconstruction apparatus, comprising:

[0036] The acquisition unit is used to acquire the approximate normal map reconstructed based on the polarization information of the target surface;

[0037] The stitching unit is used to stitch the approximate normal map and the original polarization image in the channel dimension to obtain a stitched image;

[0038] The reconstruction unit is used to input the stitched image into a pre-trained neural network, which outputs a remapped target surface normal map based on the mapping relationship between the polarization image and the surface normals learned through deep learning.

[0039] Thirdly, embodiments of the present invention also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the method described in any embodiment of this specification.

[0040] Fourthly, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the methods described in any embodiment of this specification.

[0041] This invention provides a method and apparatus for reconstructing the surface normal of a target. By using deep learning to capture the correlation between image space and channel dimensions, the mapping relationship between polarization images and surface normals is extracted. This mapping relationship is then used to perform feature mapping on a stitched image of an approximate normal map and the original polarization image to obtain a more accurate target surface normal map. This solves the problem of ambiguity between the zenith angle and azimuth angle in the surface normal map and achieves reasonable reconstruction of the three-dimensional shape of the target. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a flowchart of a target surface normal reconstruction method provided in an embodiment of the present invention;

[0044] Figure 2 This is a normal polarization coordinate representation diagram of a target surface micro-element provided in an embodiment of the present invention;

[0045] Figure 3 This is a neural network architecture diagram provided in one embodiment of the present invention;

[0046] Figure 4 This is a hardware architecture diagram of an electronic device provided in an embodiment of the present invention;

[0047] Figure 5 This is a structural diagram of a target surface normal reconstruction device provided in an embodiment of the present invention. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0049] Please refer to Figure 1 This invention provides a method for reconstructing the normals of a target surface, the method comprising:

[0050] Step 100: Obtain the approximate normal map reconstructed based on the polarization information of the target surface;

[0051] Step 102: The approximate normal map and the original polarization image are stitched together in the channel dimension to obtain a stitched image;

[0052] Step 104: Input the stitched image into a pre-trained neural network. The neural network outputs a remapped target surface normal map based on the mapping relationship between the polarization image and the surface normals obtained through deep learning.

[0053] In this embodiment of the invention, the ability of deep learning to capture correlations in image space and channel dimensions is used to mine the mapping relationship between polarization images and surface normals. This mapping relationship is then used to perform feature mapping on the stitched image of the approximate normal map and the original polarization image to obtain a more accurate target surface normal map. This can solve the problem of ambiguity between the zenith angle and the azimuth angle in the surface normal map and achieve reasonable reconstruction of the target's three-dimensional shape.

[0054] The following description Figure 1 The execution method for each step is shown.

[0055] First, for step 100, an approximate normal map reconstructed based on the polarization information of the target surface is obtained.

[0056] In this embodiment of the invention, the method of reconstructing an approximate normal map using the polarization information of the target surface may include the following steps S1-S4:

[0057] S1. Obtain an infrared radiation image of the target in a set polarization direction based on a polarization imaging measurement device.

[0058] The polarization direction needs to cover 0° to 180°. Preferably, it can be 0°, 45°, 90°, and 135°, or it can be 0°, 60°, and 180°. Taking 0°, 45°, 90°, and 135° as an example, an infrared radiation image I can be obtained. 0° I 45° I 90° I 135° .

[0059] S2. Calculate the Stokes parametric images S0, S1, and S2 using the infrared radiation images.

[0060] In this embodiment of the invention, the Stokes parametric image can be calculated according to the following formula:

[0061]

[0062] Among them, I RCP and I LCP These represent right-handed circular polarization and left-handed circular polarization, respectively. Since the circular polarization component of natural targets is small, it can be ignored.

[0063] S3. Calculate the polarization degree image of the target surface based on the Stokes parametric image.

[0064] In this embodiment, the polarization information image of the target surface is obtained by calculating the Stokes parametric image. The degree of polarization is a dimensionless number from 0 to 1, typically calculated using Stokes parameters. The polarization information of the target surface is calculated according to the following formula:

[0065]

[0066] Where p is the degree of polarization, and the degree of polarization can be used as a polarization image.

[0067] S4. Use the polarization degree image to determine the correlation between polarization degree and zenith angle and object refractive index, so as to roughly reconstruct the normal map of the target surface according to the correlation and obtain an approximate normal map.

[0068] Please refer to Figure 2 This is a representation of the polarization coordinates of the normal to a micro-element of the target surface; where θ is the zenith angle of the surface normal, representing the angle between the observation direction and the normal direction; α is the azimuth angle, representing the angle between the projection of the surface normal onto the image plane and the coordinate axis. Given a polarization angle φ... pol Then the light intensity at a point on the polarization image can be expressed as:

[0069]

[0070] Where φ represents the phase angle of the light, I max with Imin This represents the upper and lower limits of the light intensity that the system can receive. The φ obtained from the above formula has an ambiguity of π. Based on the phase angle φ, the azimuth angle α of the surface normal has an ambiguity of π / 2, that is, when diffuse reflection is the main component of the reflected light, α = φ; when specular reflection is the main component, α = φ + π / 2.

[0071] It is understandable that the zenith angle θ of the surface normal is related to the degree of polarization p. The physical definition of the degree of polarization p is as follows:

[0072]

[0073] When diffuse reflection is the dominant component, the degree of polarization p d The relationship between the zenith angle θ and the refractive index n of the object is as follows:

[0074]

[0075] When the refractive index is known, the zenith angle θ has a unique solution.

[0076] When specular reflection is the dominant component, the degree of polarization p s The relationship between the zenith angle θ and the refractive index n of the object is as follows:

[0077]

[0078] At this point, the zenith angle θ will have two possible solutions.

[0079] Once it is determined whether diffuse reflection or specular reflection is the dominant component, the degree of polarization calculated in S3 is substituted into the degree of polarization formula corresponding to S4 in this step.

[0080] Specifically, if diffuse reflection is the dominant component, then substitute the degree of polarization p into the formula p d If specular reflection is the dominant component, then the degree of polarization p is substituted into the formula p s In this process, the polarization degree image of the target surface and the correlation between the polarization degree and the zenith angle θ and the object's refractive index n are obtained. Based on the specular and diffuse reflection components of the polarization image and their correlation, the normal map of the target surface is roughly reconstructed to obtain an approximate normal map. Note that the zenith angle and azimuth angle may contain ambiguities.

[0081] Next, we will explain step 102, "sew the approximate normal map and the original polarization image together in the channel dimension to obtain a stitched image", and step 104, "input the stitched image into a pre-trained neural network, which outputs a re-mapped target surface normal map based on the mapping relationship between the polarization image and the surface normals in deep learning".

[0082] In this embodiment of the invention, the original polarization image may be at least one of the Stokes parametric image and the polarization degree image.

[0083] Stitching the approximate normal map and the original polarization image in the channel dimension can be achieved by summing the pixel values ​​of corresponding pixels to obtain the stitched image.

[0084] In order to obtain the mapping relationship between polarization images and surface normals using deep learning, it is necessary to train a neural network. In this embodiment of the invention, the training method of the neural network may include the following steps A1 to A2:

[0085] A1. Using a standard body as the target, obtain multiple sample pairs of the standard body; the sample pairs include a sample approximate normal map, a sample polarization image, and a sample surface normal map; the sample approximate normal map is reconstructed based on the polarization information of the standard body surface.

[0086] The standard body is an object with a standard shape, such as a cube, cuboid, or sphere, and whose parameters (such as size and angles) are known. Since the parameters of the standard body are known, an accurate surface normal diagram can be obtained.

[0087] In addition, the approximate normal map can be used as a physical prior to help the neural network learn the mapping relationship between the polarization image and the surface normal. Therefore, it is necessary to reconstruct the approximate normal map of the standard body based on the polarization information of the standard body surface (see S1 to S4 for the reconstruction method).

[0088] The sample polarization image is at least one of the Stokes parametric image and polarization degree image in the process of reconstructing the approximate normal map of the standard body.

[0089] It should be noted that the method for determining the original polarization image in step 102 must be the same as the method for determining the sample polarization image. For example, if the sample polarization image is the Stokes parameter image S1 in the approximate normal map reconstruction process of the standard body, then the original polarization image is the Stokes parameter image S1 in the approximate normal map reconstruction process of the target.

[0090] A2. For each sample pair, the approximate normal map and polarization image of the sample in the sample pair are stitched together in the channel dimension to obtain a stitched image. The stitched image is used as input and the surface normal map of the sample in the sample pair is used as output to train the neural network; so as to obtain a trained neural network using multiple sample pairs.

[0091] In this embodiment of the invention, the neural network needs to learn and output the sample surface normal map based on the input sample stitched image, and deeply learn the mapping relationship between the polarization image and the surface normal. Based on this, in one implementation, please refer to... Figure 3 The neural network consists of the following components from input to output: a first convolutional layer, a lightweight convolutional module, an efficient attention module, and a second convolutional layer.

[0092] The functions of each module in this neural network are as follows: the first convolutional layer is used to extract the spatial features of the stitched image; the lightweight convolutional module is used to extract the deep features based on the spatial features; the efficient attention module is used to perform feature dimensionality reduction based on the deep features, and to build a correlation model based on the dimensionality-reduced features so that the correlation model can output the correlation features; the second convolutional layer is used to remap the spatial features based on the correlation features to obtain the surface normal map.

[0093] Since the computationally complex high-efficiency attention module is relatively high, it is advisable to first use a lightweight convolutional module with low computational complexity to deeply extract and organize the features required for target surface reconstruction. This can effectively improve the performance of the subsequent high-efficiency attention module. Combining the lightweight convolutional module with a vision-based Transformer high-efficiency attention module enables the reconstruction of target surface normals.

[0094] Based on this, multiple sample pairs can be used to train the neural network, so as to adjust the parameters in the neural network to the optimal value, thereby obtaining a trained neural network.

[0095] Specifically, when using a neural network based on deep learning to map polarization images and surface normals to output a remapped target surface normal map, the following steps B1 to B4 may be included:

[0096] B1. Extract the spatial features of the stitched image using the first convolutional layer.

[0097] The first convolutional layer extracts spatial and channel-dimensional features, maps the image and normal information to the feature space, and obtains spatial features extracted from the stitched image, which facilitates subsequent deep feature extraction and intrinsic correlation learning.

[0098] B2. Utilize the lightweight convolution module to extract deep features based on the spatial features.

[0099] In this embodiment of the invention, the lightweight convolution module includes: multiple residual feature blocks; each residual feature block includes: two residual units connected in parallel, and a 1×1 convolutional layer and a 3×3 convolutional layer connected in series after the two parallel residual units. Further, each residual unit consists of a convolution and an activation function for feature extraction. The two residual units concatenate the extracted feature maps along the channel dimension to form dense connections, thereby more fully expressing the channel-dimensional features of the polarization image. Then, a 1×1 kernel convolutional layer is used to fuse and interact information between channels, and a 3×3 kernel convolutional layer is used to compress the number of channels back to the original dimension.

[0100] Specifically, each of the residual feature blocks extracts depth features F using the following formula. R (X f ):

[0101] F R (X f )=f 3×3 (f 1×1 (cat c (RU(X f ),RU(RU(X f )))))+X f

[0102] RU(X f )=f k×k (σ(f k×k (X f )))+X f

[0103] Among them, cat c (·) represents splicing on the channel dimension, RU(·) represents the residual unit, X f Characterizing the spatial features, f k×k (·) represents the convolution function with a kernel size of k×k, and σ(·) represents the activation function.

[0104] B3. The efficient attention module is used to perform feature dimensionality reduction based on the deep features. The matrix parameters in the correlation model are generated based on the dimensionality-reduced features, and the matrix parameters are substituted into the correlation model to output the correlation features. The correlation model is generated in the efficient attention module after training.

[0105] The efficient attention module consists of multiple Transformers and is primarily used to capture the correlation features between the polarization image and the normal prior. In this embodiment of the invention, the efficient attention module can use an embedding layer to embed the spatial features X. f Transform into feature vector X FAfter normalization, self-attention is calculated, followed by a residual connection, then another residual connection containing normalization and linear mapping, and finally the feature map X is reconstructed through a restoration layer. f The normalization layer and residual connection structure is a mechanism for stabilizing deep network training, which can mitigate ill-posed gradients and model degradation. The self-attention layer learns the intra-image correlations by computing self-attention, while the linear mapping layer composed of fully connected layers is used for global information fusion.

[0106] After deep feature extraction by the lightweight convolutional module, the Transformer can reduce the computational complexity by obtaining key feature information through dimensionality reduction to calculate self-attention. First, the feature vector X... F Dimensionality reduction is performed, and then the query matrix Q, key matrix K, and value matrix V are generated separately. Through correlation operations between the query matrix and the key matrix, the attention map A can be obtained.

[0107] A = soft max(QK) T )

[0108] Here, softmax is the normalization function, and then the weighted sum of the value matrix is ​​calculated to obtain the self-attention. Then, the original size is restored by the dimensionality increase operation e(·), and the correlation model is established.

[0109] Y = e(AV)

[0110] B4. The second convolutional layer is used to remap the spatial features based on the correlation features to obtain the target surface normal map.

[0111] Based on the above steps, the neural network outputs a more accurate normal map of the target surface.

[0112] like Figure 4 , Figure 5 As shown, this embodiment of the invention provides a target surface normal reconstruction device. The device embodiment can be implemented through software, hardware, or a combination of both. From a hardware perspective, as... Figure 4 The diagram shown is a hardware architecture diagram of an electronic device for reconstructing the normals of a target surface provided in an embodiment of the present invention. (Except for...) Figure 4 In addition to the processor, memory, network interface, and non-volatile memory shown, the electronic device in the embodiment may also include other hardware, such as a forwarding chip responsible for processing packets. Taking software implementation as an example, such as... Figure 5 As shown, a device in a logical sense is formed by the CPU of its host electronic device reading the corresponding computer program from non-volatile memory into memory and running it. This embodiment provides a target surface normal reconstruction device, comprising:

[0113] The acquisition unit 501 is used to acquire the approximate normal map reconstructed based on the polarization information of the target surface;

[0114] The stitching unit 502 is used to stitch the approximate normal map and the original polarization image in the channel dimension to obtain a stitched image;

[0115] The reconstruction unit 503 is used to input the stitched image into a pre-trained neural network, which outputs a remapped target surface normal map based on the mapping relationship between the polarization image and the surface normals learned through deep learning.

[0116] In one embodiment of the present invention, the training method of the neural network includes:

[0117] Multiple sample pairs of the standard object are obtained by using the standard object as the target; each sample pair includes a sample approximate normal map, a sample polarization image, and a sample surface normal map; the sample approximate normal map is reconstructed based on the polarization information of the standard object surface;

[0118] For each sample pair, the approximate normal map and polarization image of the sample in the sample pair are stitched together in the channel dimension to obtain a stitched image. The stitched image is used as input and the surface normal map of the sample in the sample pair is used as output to train the neural network; thus, a trained neural network is obtained using multiple sample pairs.

[0119] In one embodiment of the present invention, the neural network comprises, from input to output, a first convolutional layer, a lightweight convolutional module, an efficient attention module, and a second convolutional layer.

[0120] The reconstruction unit is specifically configured to: extract spatial features of the stitched image using the first convolutional layer; extract depth features based on the spatial features using the lightweight convolutional module; perform feature dimensionality reduction based on the depth features using the efficient attention module; generate matrix parameters in the correlation model based on the dimensionality-reduced features; and substitute the matrix parameters into the correlation model to output the correlation features; the correlation model is generated in the efficient attention module after training; and remap the spatial features based on the correlation features using the second convolutional layer to obtain the target surface normal map.

[0121] In one embodiment of the present invention, the lightweight convolution module includes: a plurality of residual feature blocks; the residual feature block includes: two residual units connected in parallel and a 1×1 convolutional layer and a 3×3 convolutional layer connected in series after the two residual units connected in parallel;

[0122] When the reconstruction unit performs depth feature extraction based on the spatial features using the lightweight convolution module, it specifically includes: extracting depth features F for each residual feature block using the following formula. R (X f ):

[0123] F R (X f )=f 3×3 (f 1×1 (cat c (RU(X f ),RU(RU(X f )))))+X f

[0124] RU(X f )=f k×k (σ(f k×k (X f )))+X f

[0125] Among them, cat c (·) represents splicing on the channel dimension, RU(·) represents the residual unit, X f Characterizing the spatial features, f k×k (·) represents the convolution function with a kernel size of k×k, and σ(·) represents the activation function.

[0126] In one embodiment of the present invention, the correlation model includes:

[0127] Y = e(AV)

[0128] A = soft max(QK) T )

[0129] Where Y represents the relevance feature, e(·) represents the dimensionality increase operation, A represents the attention feature, V represents the value matrix, softmax represents the normalization function, Q represents the query matrix, and K represents the key matrix.

[0130] In one embodiment of the present invention, the acquisition unit is specifically used for: acquiring an infrared radiation image of the target in a set polarization direction based on a polarization imaging measurement device; calculating Stokes parameter images S0, S1, and S2 using the infrared radiation image; calculating a polarization degree image of the target surface based on the Stokes parameter image; determining the correlation between the polarization degree and the zenith angle and the object's refractive index using the polarization degree image, so as to roughly reconstruct the normal map of the target surface based on the correlation to obtain an approximate normal map.

[0131] In one embodiment of the present invention, the original polarization image is at least one of the Stokes parametric image and the polarization degree image.

[0132] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on a target surface normal reconstruction device. In other embodiments of the present invention, a target surface normal reconstruction device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0133] The information interaction and execution process between the modules in the above-mentioned device are based on the same concept as the method embodiment of the present invention, and the specific details can be found in the description of the method embodiment of the present invention, and will not be repeated here.

[0134] This invention also provides an electronic device, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements a target surface normal reconstruction method according to any embodiment of this invention.

[0135] This invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform a target surface normal reconstruction method according to any embodiment of this invention.

[0136] Specifically, a system or apparatus equipped with a storage medium may be provided, on which software program code implementing the functions of any of the embodiments described above is stored, and the computer (or CPU or MPU) of the system or apparatus may read and execute the program code stored in the storage medium.

[0137] In this case, the program code read from the storage medium can itself implement the function of any of the above embodiments, and therefore the program code and the storage medium storing the program code constitute part of the present invention.

[0138] Examples of storage media used to provide program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code can be downloaded from a server computer via a communication network.

[0139] Furthermore, it should be clear that not only can the program code read by the computer be executed, but also the operating system or other components operating on the computer can be instructed based on the program code to perform some or all of the actual operations, thereby realizing the function of any of the embodiments described above.

[0140] Furthermore, it is understood that the program code read from the storage medium is written to the memory set in the expansion board inserted into the computer or to the memory set in the expansion module connected to the computer. Then, based on the instructions of the program code, the CPU or other components installed on the expansion board or expansion module execute some and all of the actual operations, thereby realizing the function of any of the above embodiments.

[0141] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0142] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as ROM, RAM, magnetic disk, or optical disk.

[0143] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for reconstructing the normals of a target surface, characterized in that, include: Obtain the approximate normal map reconstructed based on the polarization information of the target surface; The approximate normal map and the original polarization image are stitched together in the channel dimension to obtain the stitched image; The stitched image is input into a pre-trained neural network, which outputs a remapped target surface normal map based on the mapping relationship between polarization images and surface normals learned through deep learning. The neural network, from input to output, comprises: a first convolutional layer, a lightweight convolutional module, an efficient attention module, and a second convolutional layer. The neural network, based on the mapping relationship between polarization images and surface normals learned through deep learning, outputs a remapped target surface normal map. This includes: extracting spatial features from the stitched image using a first convolutional layer; extracting deep features based on the spatial features using a lightweight convolutional module; performing feature dimensionality reduction based on the deep features using an efficient attention module; generating matrix parameters in a correlation model based on the dimensionality-reduced features; and substituting the matrix parameters into the correlation model to output correlation features. The correlation model is generated in the efficient attention module after training. Finally, the second convolutional layer remaps the spatial features based on the correlation features to obtain the target surface normal map. The lightweight convolution module includes: multiple residual feature blocks; each residual feature block includes: two residual units connected in parallel and a 1×1 convolutional layer and a 3×3 convolutional layer connected in series after the two residual units connected in parallel; The step of using the lightweight convolution module to extract deep features based on the spatial features includes: Each of the residual feature blocks is used to extract depth features using the following formula. : in, The representation is spliced ​​along the channel dimension. Characterizing the residual unit, Characterizing the spatial features, Characterizing the kernel size is The convolution function, Characterize the activation function; The correlation model includes: Wherein, Y represents the correlation characteristic. The representations are as follows: A represents the attention feature, V represents the value matrix, softmax represents the normalization function, Q represents the query matrix, and K represents the key matrix.

2. The method according to claim 1, characterized in that, The training method for the neural network includes: Multiple sample pairs of the standard object are obtained by using the standard object as the target; each sample pair includes a sample approximate normal map, a sample polarization image, and a sample surface normal map; the sample approximate normal map is reconstructed based on the polarization information of the standard object surface; For each sample pair, the approximate normal map and polarization image of the sample in the sample pair are stitched together in the channel dimension to obtain a stitched image. The stitched image is used as input and the surface normal map of the sample in the sample pair is used as output to train the neural network; thus, a trained neural network is obtained using multiple sample pairs.

3. The method according to any one of claims 1-2, characterized in that, The approximate normal map reconstructed based on the polarization information of the target surface includes: Infrared radiation images of the target in a set polarization direction are obtained based on a polarization imaging measurement device; Stokes parametric image calculated using the infrared radiation image , and ; Calculate the polarization degree image of the target surface based on the Stokes parametric image; The polarization degree image is used to determine the correlation between polarization degree, zenith angle, and object refractive index, so as to roughly reconstruct the normal map of the target surface based on the correlation and obtain an approximate normal map.

4. The method according to claim 3, characterized in that, The original polarization image is at least one of the Stokes parametric image and the polarization degree image.

5. A target surface normal reconstruction device, characterized in that, For performing the method as described in any one of claims 1-4 above, comprising: The acquisition unit is used to acquire the approximate normal map reconstructed based on the polarization information of the target surface; The stitching unit is used to stitch the approximate normal map and the original polarization image in the channel dimension to obtain a stitched image; The reconstruction unit is used to input the stitched image into a pre-trained neural network, which outputs a remapped target surface normal map based on the mapping relationship between the polarization image and the surface normals learned through deep learning.

6. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor, when executing the computer program, implements the method as described in any one of claims 1-4.

7. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method of any one of claims 1-4.

Images