A target reconstruction method based on a differentiable SAR image renderer

Abstract

The application provides a target reconstruction method based on a differentiable SAR image renderer, which is used for reconstructing a three-dimensional target through a differentiable renderer according to a SAR image, and comprises the following steps: step S1, defining a three-dimensional rendering scene; step S2, reconstructing the target, in step S2, a contour map is used for reconstructing a non-ground target, and an illumination map and a shadow are used for reconstructing a ground target, wherein the reconstruction process in step S2 comprises the following steps: step S2-1, according to the target type, extracting a contour map or an illumination map and a shadow from the SAR image as a true value and inputting the true value into the differentiable renderer as a rendering target; step S2-2, the initialization input of the differentiable renderer is a spherical mesh, the error between the rendering image of the differentiable renderer and the true value is transmitted to the input scene parameters in the reverse direction along the forward rendering pipeline of the differentiable renderer, the numerical value of the scene parameters is modified multiple times through a gradient descent algorithm, and a three-dimensional scene conforming to the target two-dimensional image is reconstructed.

Description

Technical Field

[0001] This invention belongs to the field of radar image processing technology, specifically relating to a target reconstruction method based on a differentiable SAR image renderer. Background Technology

[0002] Synthetic Aperture Radar (SAR) has become an important tool for Earth remote sensing, capable of high-resolution imaging under all-weather and all-time conditions. Traditional SAR can only acquire two-dimensional SAR images, and side-looking imaging methods result in foreshortening and layover effects. To overcome these problems, researchers have proposed several three-dimensional (3D) SAR systems and imaging methods, aiming to directly acquire the 3D electromagnetic scattering structure of targets and eliminate foreshortening and layover effects caused by the imaging mechanism of SAR images. This is of great significance for target interpretation, remote sensing mapping, and digital cities.

[0003] Traditional SAR 3D imaging uses point clouds as the primary form of 3D representation. The earliest 3D imaging techniques were represented by Interferometric SAR (InSAR) and StereoSAR, which solved the 3D position of individual scatterers based on pixels from multiple registered viewpoints. However, in complex scenes such as urban built-up areas, echoes from multiple scatterers at different heights can be mapped onto the same resolution cell, a phenomenon known as overlay. These two techniques cannot handle overlay and therefore lack 3D resolution capability. Following Knaell's initial proposal of the concept of 3D SAR, various countries have conducted related research, primarily developing two 3D imaging techniques: Tomographic SAR (TomoSAR) and Array InSAR. Through multiple observations, these two methods have achieved elevation-level resolution by synthesizing apertures in the elevation direction.

[0004] Since the successful application of CNNs to image classification tasks in 2012, deep learning has achieved tremendous success in the field of computer vision. Due to the great success of data-driven methods, an increasing number of studies are using deep neural networks to reconstruct 3D geometry from 2D images. For training networks, there are two approaches: 3D supervision and 2D supervision. The former requires providing ground truth values ​​of the 3D shape, while the latter only requires providing 2D images and is more promising. In the field of SAR 3D reconstruction, researchers have also explored deep learning methods.

[0005] Acquiring SAR image samples is difficult, and acquiring multiple strongly coherent complex SAR data is even more challenging. To ensure coherence between SAR images from TomoSAR reorbit observations, precise track control is required, which is highly difficult to implement. Therefore, SAR 3D imaging technologies, represented by TomoSAR, are unsuitable for applications with high timeliness requirements. Unlike optical images, due to the unique scattering characteristics and imaging mechanism of the microwave band in SAR, target morphology varies greatly in SAR images. This means that 3D reconstruction neural network models pre-trained based on optical images may not be directly applicable to SAR image processing. To address these issues, a target reconstruction algorithm that fits the SAR imaging geometry, utilizes SAR intensity maps, and does not require ground-value supervised training is needed. Summary of the Invention

[0006] This invention is made to solve the above-mentioned problems, and aims to provide a target reconstruction method based on a differentiable SAR image renderer.

[0007] This invention provides a target reconstruction method based on a differentiable SAR image renderer, used to reconstruct a three-dimensional target from a SAR image using a differentiable renderer. The method includes the following steps: Step S1, defining the three-dimensional rendering scene of the differentiable renderer; Step S2, reconstructing the target. In Step S2, contour maps are used for reconstruction of non-ground targets, and illumination maps and shadows are used for reconstruction of ground targets. Reconstruction using illumination maps and shadows includes: a target reconstruction method based on illumination maps, a target reconstruction method based on shadows, and a target reconstruction method based on both illumination maps and shadows. The reconstruction process in Step S2 includes: Step S2-1, extracting contour maps or illumination maps and shadows from the SAR image as ground truth based on the target type and inputting them into the differentiable renderer as the rendering target; Step S2-2, initializing the input of the differentiable renderer as a spherical grid, and along the opposite direction of the forward rendering pipeline of the differentiable renderer, transferring the error between the rendered image and the ground truth to the input scene parameters. The scene parameters are modified multiple times using a gradient descent algorithm to reconstruct a three-dimensional scene that matches the two-dimensional image of the target.

[0008] The target reconstruction method based on a differentiable SAR image renderer provided by this invention may also have the following features: Step S1 includes the following sub-steps: Step A1, establish a world coordinate system O-XYZ, and place the target to be imaged at the origin O; Step A2, set the radar nominal position O′, the radar's motion direction is O′X′, the illumination direction is O′Z′, determine the direction of the third axis O′Y′ through O′X′ and O′Z′, establish a radar coordinate system O′-X′Y′Z′ centered on the radar, and define the plane O′Z′X′ as the mapping plane according to the projection mapping algorithm, and the (k,l)th mapping unit is represented as m. (k,l) Define plane O′X′Y′ as the projection plane, and denote the (i,l)th element as p. (i,l) Step A3: Select a grid as the representation of the imaging target, which includes a vertex set. and a triangular element set That is, there are N v vertices and N f Each face value Represents the spatial coordinates of the i-th vertex. f represents the indices of the three vertices belonging to the j-th triangle element. j There is a texture attribute S j and S j Set it to a scalar; Step A4, set the vertex set of the mesh target. Vertex {v} transformed to radar coordinate system i}, then the surface element f j Vertex coordinates M j The definition is as follows:

[0009]

[0010] In formula (1), (x j,n ,y j,n ,z j,n ) is f j The x, y, z coordinates of the nth vertex.

[0011] The target reconstruction method based on a differentiable SAR image renderer provided by this invention may also have the following feature: wherein, the reconstruction using a contour map in step S2 includes the following sub-steps:

[0012] Step B1, after slant range transformation in the radar coordinate system, the surface element f j Contains mapping unit m (k,l) The possibility is defined as probability. Euclidean distance d(m) between the two (k,l) ,f j The relevant calculation formula is as follows:

[0013]

[0014] In formula (2), A sign indicator bit represents the mapping unit m. (k,l) In f j Is it inside or outside? σ is a scalar that controls the sharpness of the probability distribution. When σ→0, the probability map converges to the exact shape of the surface boundary.

[0015] Step B2: The contour map is a two-dimensional projection of the target onto the mapping plane, and the value of each resolution cell in the contour map is... The definition is as follows:

[0016]

[0017] In formula (3), N f Indicates the number of face elements in the mesh target;

[0018] Step B3, taking into account To connect the contour maps and coordinates M j intermediate variables, For M j The derivative is defined as follows:

[0019]

[0020] Step B4: Define a hybrid loss function to supervise the geometric reconstruction process. This hybrid loss function not only measures the error between the rendered image and the ground truth, but also constrains the smoothness of the surface of the facets. The formula is as follows:

[0021] L = L sil +λ1L lap +λ2L flat (5)

[0022] In formula (5), λ1 = 0.03, λ2 = 0.0003, and the weights decrease as importance decreases. sil Contour map I for rendering sil and contour plot truth value The negative crossover ratio (NCR) represents the difference between the rendered contour map and the ground truth contour map, and is expressed by the following formula:

[0023]

[0024] In formula (6), ⊙ represents the dot product, and I sil and The higher the degree of overlap, the more L sil The smaller the value of , the better.

[0025] In formula (5), L lap L is the sum of squares of the coordinates in the Laplace transform domain, used to evaluate the distance between adjacent nodes. lap The smaller the value, the more compact the mesh element space. lap The definition is as follows:

[0026]

[0027] In formula (7), The coordinates of the grid vertex set V after the Laplace transform. yes The coordinates of the nth axis of the i-th vertex;

[0028] In formula (5), L flat The sum of squares of the cosines of the angles between adjacent face elements is defined as follows:

[0029]

[0030] In formula (8), E is the set of all edges in the deformed mesh, and θ i It is the included angle between two triangular facets that share the i-th edge;

[0031] Step B5: After each iteration, obtain the gradient as the magnitude of the correction that should be made to the target based on the current rendered image. Opposite Yuan f j Based on the returned gradient, its coordinate matrix M is... j The formula has been adjusted and is as follows:

[0032]

[0033] In formula (9), the left side of the formula is the adjusted coordinate, the right side is the original coordinate, and μ represents the learning rate during gradient descent.

[0034] The target reconstruction method based on a differentiable SAR image renderer provided by this invention may also have the following feature: wherein the target reconstruction method based on the illumination map in step S2 includes the following sub-steps:

[0035] Step C1, in the rendered image obtained by the differentiable renderer, each resolution unit m (k,l) Cumulative scattering value at The definition is as follows:

[0036]

[0037] Step C2, the illumination map is defined as the radar visible area, i.e. The corresponding region, for mapping unit m(k,l) , surface element f j Contribution to it The probability is defined as:

[0038]

[0039] The definitions of the two components in formula (11) are as follows:

[0040]

[0041]

[0042] In formula (12), p represents the (i,l)th projection unit. (i,l) Assigned to f j energy, express For nearby m (k,l) The contribution ratio, with p (i,l) The reflection point and m on the mapping plane (k,l) The distance between them is related.

[0043] For m (k,l) In all N f The probability that at least one of the face elements is illuminated. The formula is as follows:

[0044]

[0045] Step C3, the partial derivative of the illumination image with respect to the primitive mesh is defined as:

[0046]

[0047] Step C4: Define a hybrid loss function to supervise the geometric reconstruction process. This hybrid loss function measures the error between the rendered image of the illumination map and the ground truth image, as shown in the following formula:

[0048] L = L ill +λ1L lap +λ2L flat (16)

[0049] In formula (16), L ill It is the rendered illumination map I ill and illumination map true value The negative intersection-union ratio (NOU) is used to measure the difference between the rendered image and the ground truth, and is defined as follows:

[0050]

[0051] Step C5, rendering the image The strength of the amplitude and The probabilities are positively correlated. An illumination map is created based on the target image of the SAR image as the ground truth, and this illumination map is used as the rendering target of the differentiable renderer. The geometric coordinates M of each surface element are adjusted in multiple iterations. j After convergence, the three-dimensional target is obtained.

[0052] The target reconstruction method based on a differentiable SAR image renderer provided by this invention may also have the following feature: wherein the target reconstruction method based on shadows in step S2 includes the following sub-steps:

[0053] Step D1, place the face element f j After projection and slant range transformation to the mapping plane, define it and the mapping unit m. (k,l) The intersection probability in the mapping plane is The definition of a ground distance contour map is:

[0054]

[0055] The upper boundary of the shadow is also the illuminated area I. ill The lower boundary of the shadow, which is the dividing line between the bright and dark areas, depends on the farthest boundary of the target's projection onto the ground along the radar beam illumination direction. The shadow is then defined as:

[0056] I sha =I gsil -I gsil ⊙I ill (19)

[0057] In step D2, the gradient calculation for each cell and each surface element in the shaded region is independent of each other. Opposite Yuan f j The derivative of the coordinate matrix is:

[0058]

[0059] Step D3: Define a hybrid loss function to supervise the geometry reconstruction process, where L sha It is rendering shadow map I sha And the truth value of the shadow map Negative cross union ratio between them;

[0060] Step D4: Segment the shadow of the corresponding target from the multi-view SAR image, use the segmentation result as the ground truth and as the rendering target of the differentiable renderer, inversely derive the mesh model corresponding to the shadow, complete the 3D reconstruction, and perform a quantitative evaluation of the reconstruction result after the reconstruction is completed.

[0061] The target reconstruction method based on a differentiable SAR image renderer provided by this invention may also have the following feature: wherein the target reconstruction method based on illumination map and shadow in step S2 includes the following sub-steps:

[0062] Step E1, considering the rendering illumination map I ill and rendering shadow map I shd True value of illumination map And the truth value of the shadow map Mutually exclusive, define a blending loss function L that merges the illumination map and shadows. comb , used to replace L ill or L sha The formula is as follows:

[0063]

[0064] L comb Opposite Yuan f j coordinate matrix M j The partial derivative is:

[0065]

[0066] Step E2: Segment the shadow of the corresponding target from the multi-view SAR image, linearly scale the SAR image to obtain the illumination map of the target, use the shadow and illumination map obtained from the SAR image as the ground truth and as the rendering target of the differentiable renderer, and inversely reproduce the corresponding mesh model to complete the 3D reconstruction.

[0067] The role and effect of invention

[0068] According to the target reconstruction method based on a differentiable SAR image renderer of this invention, expressions for contour maps, illumination maps, and shadows are proposed and implemented in a differentiable renderer. In the differentiable renderer, a continuous function between the 2D image and 3D scene elements is established through probabilistic approximation of the forward process. Along the reverse direction of the forward rendering pipeline, the error between the rendered image and the ground truth is passed to the input scene parameters. By repeatedly modifying the values ​​of these parameters using a gradient descent algorithm, a 3D scene conforming to the 2D image can be reconstructed. This invention utilizes the SAR image and shadows of the target to reconstruct a geometric appearance with a high degree of overlap with the ground truth. Furthermore, the target reconstruction method based on a differentiable SAR image renderer of this invention is applicable to SAR images on any platform, exhibiting high robustness and good compatibility, and has promising prospects for widespread application. Attached Figure Description

[0069] Figure 1 This refers to the definition of the relevant coordinate system for the 3D rendering scene in the embodiments of the present invention;

[0070] Figure 2 This is a schematic diagram illustrating the process of reconstructing the target in an embodiment of the present invention;

[0071] Figure 3 This is a computational framework diagram for geometric reconstruction based on contour maps in an embodiment of the present invention;

[0072] Figure 4 This is a T72 mesh model reconstructed using a contour map in an embodiment of the present invention;

[0073] Figure 5 This is a comparative schematic diagram of the illumination map and the contour map in an embodiment of the present invention;

[0074] Figure 6 This is a comparison of the reconstruction results of the contour map and the illumination map in an embodiment of the present invention;

[0075] Figure 7 This is a schematic diagram of the target shadow in an embodiment of the present invention;

[0076] Figure 8 This is a rendering framework diagram of shadows in an embodiment of the present invention;

[0077] Figure 9 This is a comparison between the simulated shadow value and the true value in an embodiment of the present invention;

[0078] Figure 10 This is the target model reconstructed using shadows in an embodiment of the present invention;

[0079] Figure 11 This is a target model reconstructed using shadow and illumination maps in an embodiment of the present invention;

[0080] Figure 12 This is an example of comparing the SAR image rendered using the reconstructed T72 model with the ground truth in an embodiment of the present invention. Detailed Implementation

[0081] To make the technical means and effects of the present invention easy to understand, the present invention will be specifically described below in conjunction with embodiments and accompanying drawings.

[0082] <Example>

[0083] This embodiment provides a target reconstruction method based on differentiable SAR image rendering, used to reconstruct a three-dimensional target from a SAR image using a differentiable renderer, including the following steps:

[0084] Step S1: Define the 3D rendering scene for the differentiable renderer.

[0085] Figure 1 This refers to the relevant coordinate system definition of the 3D rendering scene in the embodiments of the present invention.

[0086] Figure 1 This includes the grid target to be imaged, the radar's position and direction of motion, the imaging area, etc. Figure 1 As shown, step S1 includes the following sub-steps:

[0087] Step A1: Establish the world coordinate system O-XYZ and place the target to be imaged at the origin O;

[0088] Step A2: Set the nominal radar position O′, the radar's motion direction as O′X′, and the illumination direction as O′Z′. Determine the direction of the third axis O′Y′ using O′X′ and O′Z′, and establish a radar coordinate system O′-X′Y′Z′ centered on the radar. According to the projection mapping algorithm, define the plane O′Z′X′ as the mapping plane, and the (k,l)th mapping unit is represented as m. (k,l) Define plane O′X′Y′ as the projection plane, and denote the (i,l)th element as p. (i,l) ;

[0089] Step A3: Select a grid as the representation of the imaging target, which includes a vertex set. and a triangular element set That is, there are N v vertices and N f Each face value Represents the spatial coordinates of the i-th vertex. f represents the indices of the three vertices belonging to the j-th triangle element. j There is a texture attribute S j and S j Set it to a scalar;

[0090] Step A4, set the vertex set of the mesh target Vertex {v} transformed to radar coordinate system i}, then the surface element f j Vertex coordinates M j The definition is as follows:

[0091]

[0092] In formula (1), (x j,n ,y j,n ,z j,n ) is f j The x, y, z coordinates of the nth vertex.

[0093] Step S2: Reconstruct the target.

[0094] In step S2, contour maps are used to reconstruct non-ground targets, and illumination maps and shadows are used to reconstruct ground targets. The method of reconstructing using illumination maps and shadows includes: a target reconstruction method based on illumination maps, a target reconstruction method based on shadows, and a target reconstruction method based on both illumination maps and shadows.

[0095] Figure 2 This is a schematic diagram illustrating the process of reconstructing the target in an embodiment of the present invention.

[0096] like Figure 2 As shown, the reconstruction process in step S2 includes:

[0097] Step S2-1: Based on the different types of targets, extract the contour map or illumination map and shadow from the SAR image as ground truth and send them into the differentiable renderer as the rendering target.

[0098] Step S2-2: The initial input of the differentiable renderer is a spherical mesh. Along the opposite direction of the forward rendering pipeline of the differentiable renderer, the error between the rendered image of the differentiable renderer and the true value is passed to the input scene parameters. The values ​​of the scene parameters are modified multiple times through the gradient descent algorithm to reconstruct a three-dimensional scene that conforms to the target two-dimensional image.

[0099] Figure 3 This is a computational framework diagram for geometric reconstruction based on contour maps in an embodiment of the present invention.

[0100] like Figure 3 As shown, when using contour maps for geometric reconstruction, the rendered contour image is compared with the ground truth contour image. The difference between them is then propagated back to the input using the Back Propagation (BP) algorithm. By correcting the input scene parameters, the unknown input geometry can be inferred. The specific process is as follows:

[0101] Step S2, which uses a contour map for reconstruction, includes the following sub-steps:

[0102] Step B1, after slant range transformation in the radar coordinate system, the surface element f j Contains mapping unit m (k,l) The possibility is defined as probability. Euclidean distance d(m) between the two (k,l) ,f j The relevant calculation formula is as follows:

[0103]

[0104] In formula (2), A sign indicator bit represents the mapping unit m. (k,l) In fj Is it inside or outside? σ is a scalar that controls the sharpness of the probability distribution. When σ→0, the probability map converges to the exact shape of the surface boundary.

[0105] Step B2: The contour map is a two-dimensional projection of the target onto the mapping plane, independent of the surface texture and the depth of the surface from the imaging plane. The value of each resolving unit in the contour map... The definition is as follows:

[0106]

[0107] In formula (3), N f Indicates the number of face elements in the mesh target;

[0108] Step B3, taking into account To connect the contour maps and coordinates M j intermediate variables, For M j The derivative is defined as follows:

[0109]

[0110] Step B4: Define a hybrid loss function to supervise the geometric reconstruction process. This hybrid loss function not only measures the error between the rendered image and the ground truth, but also constrains the smoothness of the surface of the facets. The formula is as follows:

[0111] L = L sil +λ1L lap +λ2L flat (5)

[0112] In formula (5), λ1 = 0.03, λ2 = 0.0003, and the weights decrease as importance decreases. Their selection is also related to the absolute magnitude of each loss function. Since L... lap and L flat The values ​​were too large, so their contribution to L was reduced. sil Contour map I for rendering sil and contour plot truth value The negative crossover ratio (NCR) represents the difference between the rendered contour map and the ground truth contour map, and is expressed by the following formula:

[0113]

[0114] In formula (6), ⊙ represents the dot product, and I sil and The higher the degree of overlap, the more L sil The smaller the value of , the better.

[0115] In formula (5), L lap L is the sum of squares of the coordinates in the Laplace transform domain, used to evaluate the distance between adjacent nodes. lap The smaller the value, the more compact the mesh element space. lap The definition is as follows:

[0116]

[0117] In formula (7), The coordinates of the grid vertex set V after the Laplace transform. yes The coordinates of the nth axis of the i-th vertex;

[0118] In formula (5), L flat The sum of squares of the cosines of the angles between adjacent face elements is defined as follows:

[0119]

[0120] In formula (8), E is the set of all edges in the deformed mesh, and θ i It is the included angle between two triangular facets sharing the i-th edge. Decrease L flat The goal is to make as many face elements as possible coplanar, which will make the output deformed mesh smoother.

[0121] Step B5: After each iteration, obtain the gradient as the magnitude of the correction that should be made to the target based on the current rendered image. Opposite Yuan f j Based on the returned gradient, its coordinate matrix M is... j The formula has been adjusted and is as follows:

[0122]

[0123] In formula (9), the left side of the formula is the adjusted coordinate, the right side is the original coordinate, and μ represents the learning rate during gradient descent.

[0124] In this embodiment, the impact of different batch sizes and different loss functions on the reconstruction results was considered. A set of control experiments was set up using the T72 tank as the reconstruction target. For each target, contours from 32 viewpoints were used for 3D reconstruction.

[0125] Figure 4 This is a T72 mesh model reconstructed using a contour map in an embodiment of the present invention.

[0126] Figure 4In the table, (a) is the T72 truth value, (b), (c), and (d) are the reconstruction results when the batch size is 1, 4, and 8, respectively, and (e), (f), and (g) are the results when bs is fixed at 8, after deleting L from the blending function. flat L lap And delete L at the same time flat and L lap The reconstruction results.

[0127] like Figure 4 As shown, the geometry of the 3D vehicles reconstructed using different batch sizes (bs) does not differ significantly. With the batch size fixed at bs = 8, and L in the mixture loss function removed... flat and L lap During partial reconstruction, based on the reconstruction results, L was found flat The component has a greater positive impact on the surface smoothness of the target mesh.

[0128] The target reconstruction method based on illumination maps in step S2 includes the following sub-steps:

[0129] Step C1, in the rendered image obtained by the differentiable renderer, each resolution unit m (k,l) Cumulative scattering value at The definition is as follows:

[0130]

[0131] Step C2, the illumination map is defined as the radar visible area, i.e. The corresponding region, for mapping unit m (k,l) , surface element f j Contribution to it The probability is defined as:

[0132]

[0133] The definitions of the two components in formula (11) are as follows:

[0134]

[0135]

[0136] In formula (12), p represents the (i,l)th projection unit. (i,l) Assigned to f j energy, Represents ρ j (i,l) For nearby m (k,l) The contribution ratio, with p (i,l) The reflection point and m on the mapping plane (k,l)The distance between them is related.

[0137] For m (k,l) In all N f The probability that at least one of the face elements is illuminated. The formula is as follows:

[0138]

[0139] Step C3, the partial derivative of the illumination image with respect to the primitive mesh is defined as:

[0140]

[0141] Step C4: Define a hybrid loss function to supervise the geometric reconstruction process. This hybrid loss function measures the error between the rendered image of the illumination map and the ground truth image, as shown in the following formula:

[0142] L = L ill +λ1L lap +λ2L flat (16)

[0143] In formula (16), L ill It is the rendered illumination map I ill and illumination map true value The negative intersection-union ratio (NOU) is used to measure the difference between the rendered image and the ground truth, and is defined as follows:

[0144]

[0145] Step C5, rendering the image The strength of the amplitude and The probabilities are positively correlated. An illumination map is created based on the target image of the SAR image as the ground truth, and this illumination map is used as the rendering target of the differentiable renderer. The geometric coordinates M of each surface element are adjusted in multiple iterations. j After convergence, the three-dimensional target is obtained.

[0146] Figure 5 This is a comparative schematic diagram of the illumination map and the contour map in an embodiment of the present invention.

[0147] like Figure 5 As shown, contour diagram I sil The process is equivalent to compressing the target in the slant range O′Z′ in the O′Y′Z′ plane. The target surface element facing away from the radar is also mapped as a contour map, which leads to the following according to I sil The contour map generated by the formula covers part of the shadow area, and the target image in the SAR image is restated as illumination map I. ill .

[0148] Figure 6This is a comparison of the reconstruction results of the contour map and the illumination map in an embodiment of the present invention.

[0149] Figure 6 In the image, (a), (b), and (c) represent the true value of the corresponding target, the reconstruction result using the contour map, and the reconstruction result using the illumination map, respectively.

[0150] like Figure 6 As shown, by comparing the reconstruction results, it can be found that when the target is at a high altitude, the reconstruction result is not as good as that of the illumination map because the labeled contour map differs greatly from the true value of the contour map; while for targets such as aircraft that are not high, the model reconstructed by the contour map is worse than that reconstructed by the illumination map, with better details and a smoother surface.

[0151] The shadow-based target reconstruction method in step S2 includes the following sub-steps:

[0152] Step D1, place the face element f j After projection and slant range transformation to the mapping plane, define it and the mapping unit m. (k,l) The intersection probability in the mapping plane is The definition of a ground distance contour map is:

[0153]

[0154] Figure 7 This is a schematic diagram of the target shadow in an embodiment of the present invention.

[0155] like Figure 7 As shown, the upper boundary of the shadow is also the illuminated area I. ill The lower boundary of the shadow, which is the dividing line between the bright and dark areas, depends on the farthest boundary of the target's projection onto the ground along the radar beam illumination direction. The shadow is then defined as:

[0156] I sha =I gsil -I gsil ⊙I ill (19)

[0157] In step D2, the gradient calculation for each cell and each surface element in the shaded region is independent of each other. Opposite Yuan f j The derivative of the coordinate matrix is:

[0158]

[0159] Step D3: Define a hybrid loss function to supervise the geometry reconstruction process, where L sha It is rendering shadow map I sha And the truth value of the shadow map Negative cross union ratio between them;

[0160] Step D4: Segment the shadow of the corresponding target from the multi-view SAR image, use the segmentation result as the ground truth and as the rendering target of the differentiable renderer, inversely derive the mesh model corresponding to the shadow, complete the 3D reconstruction, and perform a quantitative evaluation of the reconstruction result after the reconstruction is completed.

[0161] Figure 8 This is a rendering framework diagram of shadows in an embodiment of the present invention.

[0162] like Figure 8 As shown, the rendering of shadows involves two paths: (1) rendering I according to the expression of the illumination map. ill (2) The target is projected through ground distance and slant distance to obtain the ground distance contour map I. gsil Based on I ill and I gsil The shadow I can be determined sha The upper and lower boundaries.

[0163] Figure 9 This is a comparison between the simulated shadow value and the true value in an embodiment of the present invention.

[0164] like Figure 9 As shown, Figure 9 As shown in (a), SAR images of the aircraft model from multiple angles are first obtained. Due to the aircraft's own occlusion, it casts a shadow on the ground when facing away from the radar illumination direction. Through image processing, the parts with lower scattering intensity are segmented from the ground, such as... Figure 9 As shown in (b). Compared to the true shadow value obtained by segmentation, the shadow simulated according to formula (19) in this embodiment is smoother, as shown in... Figure 9 As shown in (c). Furthermore, the shadow blocks in the true shadow values ​​are not continuous, but the simulated shadow values ​​in high-probability areas are consistent with the true values.

[0165] Figure 10 This is the target model reconstructed using shadows in an embodiment of the present invention.

[0166] Figure 10 In the image, the left image represents the ground truth model of the target, while the right image represents the target model reconstructed using shadows.

[0167] like Figure 10 As shown, taking six types of targets, including cubes, as examples, and assuming a fixed ground surface, their SAR images are simulated. The shadows of the corresponding targets are segmented from the multi-view SAR images, and the segmentation results are used as ground truth to invert the mesh models corresponding to these shadows to complete the 3D reconstruction.

[0168] The target reconstruction method based on illumination maps and shadows in step S2 includes the following sub-steps:

[0169] Step E1, considering the rendering illumination map I ill and rendering shadow map I shd True value of illumination map And the truth value of the shadow map Mutually exclusive, define a blending loss function L that merges the illumination map and shadows. comb , used to replace L ill or L sha The formula is as follows:

[0170]

[0171] L comb Opposite Yuan f j coordinate matrix M j The partial derivative is:

[0172]

[0173] Step E2: Segment the shadow of the corresponding target from the multi-view SAR image, linearly scale the SAR image to obtain the illumination map of the target, use the shadow and illumination map obtained from the SAR image as the ground truth and as the rendering target of the differentiable renderer, and inversely reproduce the corresponding mesh model to complete the 3D reconstruction.

[0174] Figure 11 This is a target model reconstructed using shadow and illumination maps in an embodiment of the present invention.

[0175] Figure 11 In the image, the left image represents the ground truth model of the target, while the right image represents the target model reconstructed using shadows.

[0176] like Figure 11 As shown, the target image and shadow are segmented from the SAR image from multiple perspectives, and then sent to the differentiable renderer to invert the mesh model corresponding to the target.

[0177] In this embodiment, the T72 tank is reconstructed using a target reconstruction method based on illumination maps and shadows. The reconstructed T72 model is then used to generate a SAR image, which is compared with the ground truth SAR image. The specific process is as follows:

[0178] Step 1: Select samples from 8 viewing angles with uniform azimuth distribution from the MSTART72 ground truth, and extract the T72 target image and its shadow from the ground background.

[0179] Step 2: The generated illumination map and shadow map are used as ground truth and fed into the differentiable renderer as the rendering target. The initial input of the renderer is a spherical mesh.

[0180] Step 3: In each iteration, the current model renders the illumination map and shadow map, and compares them with the ground truth to obtain formula (21)L. comb According to formula (22), the gradient of the error with respect to the coordinates after the current input grid transformation is calculated. The gradient is then passed to the grid coordinates and adjusted using the backpropagation algorithm.

[0181] Step 4: Repeat step 3 for 200 epochs. It is found that the error between the rendered illumination map and shadows and the true value is very small. At this time, the mesh input into the renderer is the reconstructed target.

[0182] Step 5: Using the reconstructed T72 target, select three specific viewpoints, input them into the renderer, and generate SAR images at the corresponding viewpoints.

[0183] Figure 12 This is an example of comparing the SAR image rendered using the reconstructed T72 model with the ground truth in an embodiment of the present invention.

[0184] like Figure 12 As shown, to better compare the generated SAR image with the ground truth, the generated target SAR image is overlaid with the ground background from the ground truth SAR image. Furthermore, parameters for the speckle distribution are extracted from the shadows of the ground truth SAR image to generate a similar speckle-based noise floor, which is then superimposed onto the entire rendered image. The comparison between the generated target SAR image and the ground truth SAR image shows that the SAR image rendered using the reconstructed T72 model is essentially consistent with the ground truth SAR image, indicating that the three-dimensional target has been accurately reconstructed.

[0185] In this embodiment, the SAR image size is 128×128 pixels, and the experimental hardware and software configuration is GeForce RTX2080Ti and PyTorch. The time required for target reconstruction is related to the number of facets in the initial input, the number of effective pixels in the image, and the number of images used for inversion.

[0186] The role and effect of the embodiments

[0187] According to the target reconstruction method based on a differentiable SAR image renderer involved in this embodiment, expressions for contour maps, illumination maps, and shadows are proposed and implemented in a differentiable renderer. In the differentiable renderer, a continuous function between the 2D image and 3D scene elements is established by probabilistic approximation of the forward process. Along the reverse direction of the forward rendering pipeline, the error between the rendered image and the ground truth is passed to the input scene parameters. By repeatedly modifying the values ​​of these parameters using a gradient descent algorithm, a 3D scene conforming to the 2D image can be reconstructed. This embodiment utilizes the SAR image and shadows of the target to reconstruct a geometric appearance with a high degree of overlap with the ground truth. Furthermore, the target reconstruction method based on a differentiable SAR image renderer in this embodiment is applicable to SAR images on any platform, exhibiting high robustness and good compatibility, and has promising prospects for widespread application.

[0188] The above embodiments are preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention.

Claims

1. A target reconstruction method based on differentiable SAR image rendering, used to reconstruct a three-dimensional target from a SAR image using a differentiable renderer, characterized in that, Includes the following steps: Step S1: Define the 3D rendering scene of the differentiable renderer; Step S2: Reconstruct the target. In step S2, contour maps are used for reconstruction of non-ground targets, and illumination maps and shadows are used for reconstruction of ground targets. Reconstruction using illumination maps and shadows includes: illumination map-based target reconstruction methods, shadow-based target reconstruction methods, and illumination map and shadow-based target reconstruction methods. The shadow-based target reconstruction method in step S2 includes the following sub-steps: Step D1, place the face element After projection and slant range transformation onto the mapping plane, define its relationship with the mapping unit. The intersection probability in the mapping plane is: Then the definition of the ground distance contour map is: (18) The upper boundary of the shadow is also the illuminated area. The lower boundary of the shadow, which is the dividing line between the bright and dark areas, depends on the farthest boundary of the target's projection onto the ground along the radar beam illumination direction. The shadow is then defined as: (19) In step D2, the gradient calculation for each cell and each surface element in the shaded region is independent of each other. Opposite Yuan The derivative of the coordinate matrix is: (20) Step D3: Define a hybrid loss function to supervise the geometric reconstruction process, where, It is a rendering of shadow maps. And the truth value of the shadow map Negative cross union ratio between them; Step D4: Segment the shadow of the corresponding target from the SAR image from multiple perspectives, use the segmentation result as the ground truth and as the rendering target of the differentiable renderer, invert the mesh model corresponding to the shadow, complete the three-dimensional reconstruction, and perform a quantitative evaluation of the reconstruction result after the reconstruction is completed. The target reconstruction method based on illumination map and shadow in step S2 includes the following sub-steps: Step E1, considering the rendering of the illumination map and rendering shadow maps True value of illumination map And the truth value of the shadow map Mutual exclusion, defining a blending loss function that merges illumination maps and shadows. , used to replace or The formula is as follows: (21) Opposite Yuan coordinate matrix The partial derivative is: (22) Step E2: Segment the shadow of the corresponding target from the SAR image from multiple perspectives, linearly scale the SAR image to obtain the illumination map of the target, use the shadow and illumination map obtained from the SAR image as ground truth and as the rendering target of the differentiable renderer, invert the corresponding mesh model, and complete the three-dimensional reconstruction. The reconstruction process in step S2 includes: Step S2-1: Based on the different types of targets, extract the contour map or the illumination map and the shadow from the SAR image as ground truth and send them to the differentiable renderer as the rendering target; Step S2-2: The initial input of the differentiable renderer is a spherical mesh. Along the opposite direction of the forward rendering pipeline of the differentiable renderer, the error between the rendered image of the differentiable renderer and the true value is passed to the input scene parameters. The values ​​of the scene parameters are modified multiple times through the gradient descent algorithm to reconstruct a three-dimensional scene that conforms to the target two-dimensional image.

2. The sample generation method based on a differentiable SAR image renderer according to claim 1, Its features are: Step S1 includes the following sub-steps: Step A1, Establish the world coordinate system Place the target to be imaged at the origin. Place; Step A2, set the radar nominal position The radar's direction of motion is The direction of illumination is ,pass and Determine the third axis In the direction of establishing a radar coordinate system centered on the radar, According to the projection mapping algorithm, the plane is... Defined as the mapping plane, the first Each mapping unit is represented as , will the plane Defined as the projection plane, the first Each unit is represented as ; Step A3: Select a grid as the representation method for the imaging target, which includes a vertex set. and a triangular element set That is, vertices and Each face value Indicates the first The spatial coordinates of each vertex Indicates belonging to the first The indices of the three vertices of each triangle element, each triangle element There is a texture property and will Set it to a scalar; Step A4, set the vertex set of the mesh target Vertex transformed to radar coordinate system Then the face element vertex coordinates The definition is as follows: (1) In formula (1), yes The vertices coordinate.

3. The sample generation method based on a differentiable SAR image renderer according to claim 1, Its features are: The reconstruction using the contour map in step S2 includes the following sub-steps: Step B1, after slant range transformation in the radar coordinate system, the surface element... Includes mapping unit The possibility is defined as probability. , and the Euclidean distance between the two The relevant calculation formula is as follows: (2) In formula (2), A sign indicator bit represents the mapping unit. exist Is it inside or outside? , As a scalar that controls the sharpness of the probability distribution, when When the probability graph converges to the exact shape of the surface element boundary; Step B2, the contour map is a two-dimensional projection of the target onto the mapping plane, and the value of each resolution unit in the contour map is... The definition is as follows: (3) In formula (3), Indicates the number of face elements in the mesh target; Step B3, taking into account To connect the contour maps and coordinates intermediate variables, right The definition of the derivative is as follows: (4) Step B4: Define a hybrid loss function to supervise the geometric reconstruction process. This hybrid loss function not only measures the error between the rendered image and the ground truth, but also constrains the smoothness of the surface of the facets. The formula is as follows: (5) In formula (5), , The weight decreases as importance decreases. Contour map for rendering and contour plot truth value The negative crossover ratio (NCR) represents the difference between the rendered contour map and the ground truth contour map, and is expressed by the following formula: (6) In formula (6), Dot product, and The higher the degree of overlap, The smaller the value of , the better. In formula (5), This is the sum of squares of the coordinates in the Laplace transform domain, used to evaluate the distance between adjacent nodes. The smaller the value, the more compact the mesh element space. The definition is as follows: (7) In formula (7), It is a set of grid vertices Coordinates after Laplace transform yes The Middle The vertex of the first vertex The coordinates of each axis; In formula (5), It is the sum of the squares of the cosines of the angle between adjacent face elements, defined as follows: Down: (8) In formula (8), It is the set of all edges in the deformed mesh. It is shared. The included angle between two triangular elements on the edge; Step B5: After each iteration, obtain the gradient as the magnitude of the correction that should be made to the target based on the current rendered image. opposite Yuan The coordinate matrix is ​​adjusted based on the returned gradient. The formula has been adjusted and is as follows: (9) In formula (9), the left side represents the adjusted coordinates, and the right side represents the original coordinates. This represents the learning rate during gradient descent.

4. The sample generation method based on a differentiable SAR image renderer according to claim 1, Its features are: The target reconstruction method based on the illumination map in step S2 includes the following sub-steps: Step C1, in the rendered image obtained by the differentiable renderer, each resolution unit Cumulative scattering value at The definition is as follows: (10) Step C2, the illumination map is defined as the radar visible area, i.e. The corresponding region, for the mapping unit , face Contribution to it The probability is defined as: (11) The definitions of the two components in formula (11) are as follows: (12) (13) In formula (12), Indicates the first Projection unit Assigned to energy, express For the nearby The contribution ratio, and The reflection point on the mapping plane and The distance between them is related. right In all The probability that at least one of the face elements is illuminated. The formula is as follows: (14) Step C3, the partial derivative of the illumination map with respect to the primitive mesh is defined as: (15) Step C4, define a hybrid loss function to supervise the geometry reconstruction process. The hybrid loss function measures the error between the rendered image and the ground truth of the illumination map, and is expressed by the following formula: (16) In formula (16), It is a rendered illumination map and illumination map true value The negative crossover union ratio (NCR) is used to measure the difference between the rendered image and the ground truth, and is defined as follows: (17)； Step C5, the rendered image The strength of the amplitude and The probabilities are positively correlated. An illumination map is created based on the target image of the SAR image as the ground truth, and this illumination map is used as the rendering target of the differentiable renderer. The geometric coordinates of each pixel are adjusted in multiple iterations. After convergence, the three-dimensional target is obtained.

Images