View view generation method of cross-modal fusion and multi-frequency coding in ultra-low orbit scene

Abstract

The invention discloses a cross-modal fusion and multi-frequency coding view angle image generation method in an ultra-low orbit scene, and solves the problem of view angle condition generation and single image three-dimensional reconstruction method based on diffusion prior in the ultra-low orbit scene of an aircraft target. Due to insufficient condition information fusion capability, limited radiation or illumination characterization and insufficient cross-view-angle geometric constraint, a problem that a new view angle image with visual coherence and geometric consistency of texture details is difficult to obtain at the same time under large-range view angle and scale change is solved; according to the method, multi-order angle information of direction distribution is effectively captured through perceptual coding of a light direction vector, then the position of a light starting point coordinate is coded, and finally the pose of a camera pose parameter is coded, so that the influence of a relative pose on depth perception and perspective distortion in an image generation process can be depicted; the three coding results are used for a decoding stage of the potential diffusion model, and geometric consistency of detail recovery under a new view angle is effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision and image generation, and particularly relates to a view synthesis method based on cross-modal fusion and multi-frequency encoding in an ultra-low orbit scene. BACKGROUND

[0002] Humans are able to infer the three-dimensional structure and appearance of an object based on a single image, relying on long-term visual experience and semantic-geometric priors, even if the object has not been directly observed in reality. This ability is of fundamental significance to tasks such as navigation, object manipulation, and artistic creation. In contrast, traditional three-dimensional reconstruction methods (such as Structure-from-Motion (SfM), Multi-View Stereo (MVS), and implicit representation methods based on volume rendering (such as NeRF)) usually rely on sufficient multi-view observations, CAD annotations, or class-level priors, and their applicability in open-world tasks is limited.

[0003] In recent years, text-to-image pre-training diffusion models based on large-scale data (such as Stable Diffusion) have been shown to contain certain implicit geometric information (such as depth, normal, and foreground-background segmentation). Researchers have injected two-dimensional image priors into NeRF or point cloud networks, and achieved three-dimensional reconstruction based on a single image through mechanisms such as Score Distillation Sampling (SDS). However, such methods still generally face problems such as cross-view inconsistency, texture blurring, and detail loss.

[0004] Under this background, Zero123 (Zero1to3) proposes to jointly inject an input RGB image and a relative camera pose (rotation, translation) into a pre-trained U-Net network to construct a view-dependent latent diffusion model, which can generate a new image at a given target view and optimize NeRF to achieve zero-shot three-dimensional reconstruction. Although Zero123 is mainly trained on synthetic multi-view data (such as rendering data based on Objaverse), it shows good generalization ability on real images and artistic style images. However, when there is a large view change (such as a large angle rotation or significant side / elevation change), geometric inconsistency and texture drift phenomena are still likely to occur, and this is more pronounced on complex structures or special appearances (such as insects, people, and transparent objects).

[0005] To improve the phenomenon of geometric inconsistency and texture drift, various strategies have been proposed. For example, Cascade-Zero 123 constructs a two-stage cascade generation process: first, the basic model generates adjacent views, and then these views and the source image are input into the refinement model to obtain the target view, thereby improving the geometric consistency and texture detail quality; SyncDreamer synchronously generates multiple views in the inversion stage, and through the cross-view feature association of three-dimensional perception, it promotes the information sharing between intermediate states to obtain consistent multi-view results; Consistent-1-to-3 (also known as Consistent123) introduces a guided attention mechanism based on geometric / polar constraints and a multi-view attention aggregation mechanism to strengthen the geometric consistency and visual coherence of texture details between different views; Zero-to-Hero filters and reweights the attention map in the reasoning stage, which can alleviate the deformation and misplacement problem to a certain extent without retraining.

[0006] Although the above methods have made significant progress in conventional view change scenarios, for aircraft operating in ultra-low orbit scenarios, the coverage is still insufficient when using imaging payloads for observation. Imaging payloads are imaging devices installed on observation aircraft to observe another aircraft body as a target. Under the condition that both the imaging platform and the target are aircraft, high-speed relative motion, rapid changes in viewing axis and baseline, significant fluctuations in the size of the target in the field of view with relative height and attitude, and complex and variable background interference often occur. Influenced by this, existing view condition generation and single Figure Three dimensional reconstruction methods based on diffusion priors have problems such as insufficient condition information fusion capability, limited radiation or illumination representation, and insufficient cross-view geometric constraints, making it difficult to simultaneously obtain visually coherent texture details and geometrically consistent new view images under large-scale view and scale changes. Especially for small-scale components such as thin deployable components on aircraft (such as solar panels), which are thin, narrow, and strongly reflective, due to their extremely small thickness, high aspect ratio, and the tendency to produce perspective foreshortening and high light reflection, and low pixel occupancy and easily confused boundaries in imaging, existing methods are prone to distortion such as missing, breaking, jittering, or misconnection, making it difficult to stably reconstruct continuous and complete geometry and appearance. SUMMARY

[0007] The purpose of the present application is to solve the problem that existing view condition generation methods based on diffusion priors in the ultra-low orbit scenario of an aircraft have insufficient condition information fusion capability, limited radiation or illumination representation, and insufficient cross-view geometric constraints, making it difficult to simultaneously obtain visually coherent texture details and geometrically consistent new view images under large-scale view and scale changes. The present application provides a view image generation method based on cross-modal fusion and multi-frequency encoding in an ultra-low orbit scenario.

[0008] In order to achieve the above object, the present application adopts the following technical solutions: A view angle image generation method for cross-modal fusion and multi-frequency coding in an ultra-low orbit scenario, characterized by comprising the following steps: Step 1. In the ultra-low orbit scenario, a plurality of view angle sequence images of an aircraft target and corresponding camera extrinsic parameter matrices are obtained, and the corresponding camera pose parameters are calculated according to the width, height and set field of view angle of the images. One of the view angle sequence images is taken as a main view angle image, and the remaining view angle sequence images are taken as auxiliary view angle images. Step 2. The high-dimensional semantic features of the main view angle image are extracted by using a CLIP encoder, the high-dimensional semantic features are spliced with the corresponding camera pose parameters of the main view angle image, and the spliced result is input into a lightweight cross-modal attention module for cross-modal attention fusion to generate a main view angle high-dimensional conditional vector. Step 3. For each auxiliary view angle image, a spherical harmonic function embedding model based on multi-frequency coding is used to perform perceptual coding on the light direction vector, Fourier position coding is performed on the light starting point coordinates by using a multi-frequency sine and cosine function expansion, and pose coding is performed on the corresponding camera pose parameters. The above coding results are spliced, fused and compressed according to the dimensions to obtain the conditional features corresponding to each auxiliary view angle image. Step 4. A latent diffusion model composed of a U-Net denoising network, a variational autoencoder and a decoder is constructed. The multi-view sequence images of the aircraft target are input into the variational autoencoder to obtain clean latent representation tensors , and the latent representation tensors are obtained by adding noise in the latent space according to the forward diffusion, which are taken as the input of the U-Net denoising network. Step 5. The main view angle high-dimensional conditional vector and the conditional features corresponding to each auxiliary view angle image are input into the U-Net denoising network to guide the iterative denoising of the latent representation tensors , restore the latent representation under the target view angle, and obtain the new view angle image of the ultra-low orbit aircraft target after mapping by the decoder. For each view angle, the difference between the obtained new view angle image of the ultra-low orbit aircraft target and the image of the view angle in the multi-view sequence image is used to train the lightweight cross-modal attention module, the spherical harmonic function embedding model and the U-Net denoising network. Step 6. Any view angle image of the aircraft target to be tested in the ultra-low orbit scenario and the extrinsic parameter matrix of the camera in the new view angle are input into the trained lightweight cross-modal attention module, the spherical harmonic function embedding model and the U-Net denoising network to obtain the new view angle image of the aircraft target to be tested.

[0009] Further, in step 1, the camera pose parameters include position parameters representing the position of the camera in the world coordinate system, attitude parameters representing the orientation of the camera in the world coordinate system, and view cone parameters representing the main view direction vector and the scale information of the imaging plane.

[0010] Further, step 2 is specifically: Step 2.1, extracting high-dimensional semantic features of the main-view image using a CLIP encoder; Step 2.2, sequentially splicing, one-time linear mapping, and applying layer normalization to the high-dimensional semantic features of the main-view image and the corresponding camera pose parameters of the main-view image to form a cross-modal initial feature representation ; Step 2.3, inputting the cross-modal initial feature representation into a lightweight cross-modal attention module composed of two layers of Transformer encoders to learn the nonlinear coupling relationship between image semantics and geometric pose and obtain a conditional feature vector with cross-modal correlation ; Step 2.4, projecting the conditional feature vector with cross-modal correlation into a cross-attention conditional vector to obtain a high-dimensional conditional vector of the main view.

[0011] Further, step 2.2 is specifically: Step 2.2.1, splicing the high-dimensional semantic features of the main-view image with the camera pose parameters in the feature dimension, the expression is: ; wherein, represents the spliced vector, represents the dimension; Step 2.2.2, performing one-time linear mapping on the spliced vector and applying layer normalization to form a cross-modal initial feature representation , the expression is: ; wherein, represents the normalization operation, represents the weight matrix of linear mapping, , represents the bias vector of linear mapping, .

[0012] Further, in step 2.3, both of the two layers of the Transformer encoder contain multi-head self-attention mechanism and feedforward network, the expressions are respectively: ; ; wherein, represents the first layer of the Transformer encoder conditional feature vector; represents the second layer of the Transformer encoder conditional feature vector, that is, the conditional feature vector with cross-modal correlation.

[0013] Further, step 2.4 is specifically: The conditional feature vector with cross-modal correlation is linearly mapped back to the semantic dimension of the original CLIP encoder to generate a cross-attention conditional vector , and the main perspective high-dimensional conditional vector is obtained; the expression is: ; In the formula, represents the linear mapping weight matrix, , represents the linear mapping bias matrix, .

[0014] Further, step 3 is specifically: Step 3.1, the spherical harmonic function embedding model based on multi-frequency coding is adopted to normalize each ray direction vector of each auxiliary perspective image to obtain the ray direction distribution, the spherical harmonic function is used to perceptually encode the ray direction distribution, and the order is selected to obtain dimensional directional encoding feature vector ; Step 3.2, the ray origin coordinates are Fourier position encoded by using the multi-frequency sine function expansion based on neural radiation field, specifically: Define six frequency bands , , then each coordinate dimension is mapped to: ; In the formula, represents the mapping result of the coordinate dimension , j represents the component of the ray origin coordinate direction, and takes the value of x, y, and z; , then the ray origin coordinates are: ; The ray origin coordinates are linearly transformed to 32 dimensions to obtain the position encoding of the ray origin coordinates; Step 3.3, the camera pose parameters corresponding to each auxiliary perspective image are input into the pose encoder, and are mapped into a 46-dimensional pose encoding feature vector by two layers of feedforward network; Step 3.4, direction encoding feature vector , position encoding of light ray origin coordinates , pose encoding feature vector Concatenate by dimension to unified ray condition feature , the expression is: ; The unified ray condition feature is input into a two-layer linear network for fusion and compression to 78 dimensions to obtain the corresponding condition feature of each auxiliary view image .

[0015] Further, in step 4, the multi-view sequence images of the aircraft target are respectively input into the variational autoencoder to obtain the clean latent representation tensor , and the expression for the forward diffusion and noise addition in the latent space is: ; Wherein, is the latent representation tensor, is the cumulative noise modulation coefficient of time step t, i is the view index, and t is the diffusion time step, is a standard Gaussian noise with the same shape as , and represents the covariance of .

[0016] Further, in step 5, the main view high-dimensional condition vector is input into the cross-attention layer of the U-Net denoising network, and the main view high-dimensional condition vector is used as the key value to modulate the semantic distribution of the latent representation tensor ; The condition feature corresponding to each auxiliary view image is input into the convolution branch of the U-Net denoising network, and the condition feature corresponding to each auxiliary view image is fused with the latent representation tensor point by point through the convolution branch, which is used to provide geometric priors.

[0017] Advantages of the present application: 1. The present application provides a cross-modal fusion and multi-frequency encoding view generation method in an ultra-low orbit scene, which calculates the camera pose parameters according to the main view image of the aircraft target and its corresponding extrinsic matrix, constructs the geometric mapping relationship of the target view, and then introduces a lightweight cross-modal attention module for fusion to generate a main view high-dimensional condition vector for guiding the diffusion process, which is used in the decoding stage of the latent diffusion model, can guide the focusing and weighting of features in different spatial positions, and improve the texture detail coherence and geometric consistency of the generated free view image.

[0018] 2. The application provides a viewgraph generation method for cross-modal fusion and multi-frequency coding in an ultra-low orbit scenario, which can effectively capture multi-order angle information of directional distribution according to the perception coding of the light direction vector of the aircraft target auxiliary viewgraph image, and can enhance the network's ability to model the continuity of details under complex angle changes; then the position coding of the light starting point coordinates is performed, which helps the network to distinguish the relative layout of the viewgraph in different spatial positions; finally, the pose coding of the camera pose parameters is performed, which can depict the influence of the relative attitude on the depth perception and perspective distortion in the image generation process; the above three coding results are used in the decoding stage of the latent diffusion model, which can effectively improve the geometric consistency of detail recovery in the new view. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 is a flowchart of an embodiment of the viewgraph generation method for cross-modal fusion and multi-frequency coding in an ultra-low orbit scenario of the application; Figure 2 is a flowchart of step 3 of an embodiment of the viewgraph generation method for cross-modal fusion and multi-frequency coding in an ultra-low orbit scenario of the application; Figure 3 is a schematic diagram of the light direction vector and the light starting point coordinates in an embodiment of the viewgraph generation method for cross-modal fusion and multi-frequency coding in an ultra-low orbit scenario of the application; Figure 4 is an effect diagram of generating a new viewgraph image in an embodiment of the viewgraph generation method for cross-modal fusion and multi-frequency coding in an ultra-low orbit scenario of the application. DETAILED DESCRIPTION

[0020] The technical solutions of the application will be described in detail below with reference to the drawings and embodiments. Obviously, the described embodiments are only a part of the embodiments of the application, not all the embodiments. Based on the embodiments of the application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the application.

[0021] An embodiment of the application provides a viewgraph generation method for cross-modal fusion and multi-frequency coding in an ultra-low orbit scenario, as shown in Figure 1 , comprising the following steps: Step 1, in the ultra-low orbit scenario, a plurality of viewgraph sequence images and corresponding camera extrinsic matrixes of the aircraft target are obtained , and the camera pose parameters are calculated according to the width, height and set field of view angle of the image, including: (1) Translation Parameters: representing the position of the camera in the world coordinate system, denoted as: ; respectively represent the coordinate positions on the x, y, z axes of the camera in the world coordinate system.

[0022] obtained by the translation component of the extrinsic matrix .

[0023] (2) Rotation Parameters: represent the orientation of the camera in the world coordinate system, which can be extracted from the rotation component of the extrinsic matrix .

[0024] For the convenience of network processing, the rotation matrix R is converted into a compact form of three-dimensional Euler angles (yaw, pitch, roll) in this embodiment.

[0025] (3) Intrinsic Projection Parameters: The main viewing direction vector and imaging plane scale information calculated according to the width, height of the image and the set field of view angle (FOV).

[0026] One view sequence image in the multi-view sequence image of the aircraft target is taken as the main view image, and the remaining view sequence images are taken as the auxiliary view images.

[0027] Step 2, the high-dimensional semantic features of the main view image are extracted by using the CLIP encoder, the high-dimensional semantic features of the main view image are spliced with the camera pose parameters, and the lightweight cross-modal attention module (i.e. CCSF in Figure 1 ) is input to perform cross-modal attention fusion to generate a main view high-dimensional conditional vector; specifically: Step 2.1, the high-dimensional semantic features of the main view image are extracted by using the CLIP encoder; Step 2.2, the high-dimensional semantic features of the main view image are spliced with the camera pose parameters corresponding to the main view image in turn, one-time linear mapping, and layer normalization are applied to form a cross-modal initial feature representation ; Specifically: Step 2.2.1, the high-dimensional semantic features of the main view image are spliced with the camera pose parameters in the feature dimension, and the expression is: ; denote the spliced vector, denote the dimension; Step 2.2.2, the spliced vector is one-time linearly mapped, and layer normalization is applied to form a cross-modal initial feature representation The 772-dimensional coarse stitching feature is reduced to 512 dimensions, and subsequent self-attention learning is stabilized by normalization. The expression is: ; wherein, denotes a normalization operation, denotes a weight matrix of a linear mapping, , denotes a bias vector of a linear mapping, .

[0028] Step 2.3, the cross-modal initial feature representation is regarded as a token sequence with a length of 1, input into a lightweight cross-modal attention module composed of two layers of Transformer encoders, and the nonlinear coupling relationship between image semantics and geometric pose is learned through multi-head self-attention and a feed-forward network to obtain a conditional feature vector with cross-modal correlation ; Both layers of the Transformer encoder include a multi-head self-attention mechanism and a feed-forward network, and the expressions are respectively: ; ; wherein, denotes a conditional feature vector of the first layer of the Transformer encoder; denotes a conditional feature vector of the second layer of the Transformer encoder, i.e., a conditional feature vector with cross-modal correlation.

[0029] In this embodiment, the multi-head self-attention mechanism is used to capture deep dependency relationships within the feature dimension, and the feed-forward network is used to enrich the expression capability.

[0030] Step 2.4, the conditional feature vector with cross-modal correlation is linearly mapped back to the semantic dimension of the original CLIP encoder to generate a cross-attention conditional vector , obtaining a high-dimensional conditional vector of the main perspective; the expression is: ; wherein, denotes a linear mapping weight matrix, , denotes a linear mapping bias matrix, .

[0031] The main view high-dimensional condition vector is used for injection into the cross attention layer of the U-Net denoising network, so that the denoising process can simultaneously rely on image semantics and camera geometric information, and content consistency and structural constraints in new view synthesis are realized.

[0032] Step 3, for the problem of complex depth change and occlusion in the ultra-low orbit scene, the embodiment proposes a geometric perception condition construction based on light rays, specifically: Figure 2 and Figure 3 As shown, for each auxiliary view image, a spherical harmonic function embedding model (PASHE) based on multi-frequency coding is introduced, Figure 1 The spherical harmonic function is used for perception coding of the light direction vector, which can extract multi-order angular features to represent the direction dependence, and the multi-frequency sine function expansion is used for Fourier position coding of the light ray starting point coordinates to capture high-frequency details in the spatial distribution; and the corresponding camera pose parameters are encoded, and a compact pose embedding vector is obtained, and the above encoding results are spliced, fused and compressed according to the dimensions to obtain the condition features of each auxiliary view image; specifically: Step 3.1, for each auxiliary view image, each light direction vector is normalized, the spherical harmonic function is used to encode the light direction distribution, and the order is selected, to obtain a dimensional direction encoding feature vector ; Step 3.2, for the starting point coordinates of each light direction vector, a multi-frequency sine function expansion based on neural radiation field is used for Fourier position coding, specifically: Six frequency bands , are defined, and each coordinate dimension is mapped to: ; In the formula, represents the mapping result of the coordinate dimension , and j represents the component of the light ray starting point coordinates, taking values of x, y and z; The light ray starting point coordinates are: ; The light ray starting point coordinates are linearly transformed to 32 dimensions to obtain the position coding of the light ray starting point coordinates; Step 3.3, the camera pose parameters corresponding to each auxiliary view image are input into the pose encoder, and are mapped into a 46-dimensional pose encoding feature vector through two layers of feedforward network; Step 3.4, the direction encoding feature vector , the position encoding of the light ray origin coordinate , pose encoding feature vector Concatenate by dimension to unified ray condition feature , the expression is: ; The unified ray condition feature is input into a two-layer linear network for fusion and compression to 78 dimensions to obtain the corresponding condition feature of each auxiliary view image.

[0033] In each iteration step of diffusion sampling, the corresponding condition feature of each auxiliary view image is injected into the convolution branch of the U-Net denoising network as a geometric modulation signal parallel to the latent variable, thereby finely guiding the generator of the U-Net backbone structure to model the occlusion boundary, depth mutation and scale change.

[0034] Step 4, construct a latent diffusion model composed of a U-Net denoising network, a variational autoencoder, and a decoder, input the multi-view sequence images of the aircraft target into the variational autoencoder respectively to obtain the corresponding clean latent variable ; then add noise in the latent space by forward diffusion to obtain the latent representation tensor , the expression is:

[0035] , wherein is the cumulative noise modulation coefficient of time step t; i is the view index, and t is the diffusion time step. is a standard Gaussian noise (mean 0, covariance I) with the same shape as , and the noise samples of different views and time steps are independent of each other and are usually resampled step by step; the obtained latent representation tensor will be input into the U-Net denoising network; the latent result obtained after denoising is restored to the pixel domain by the decoder to generate the target view image.

[0036] Step 5, input the main view high-dimensional condition vector into the cross-attention layer of the U-Net denoising network, and use the main view high-dimensional condition vector as the key value to modulate the semantic distribution of the latent representation tensor , and input the corresponding condition feature of each auxiliary view image into the convolution branch of the U-Net denoising network, and the corresponding condition feature of each auxiliary view image is fused with the latent representation tensor point by point through the convolution branch, which is used to provide geometric priors; the latent representation tensor is iteratively denoised by the main view high-dimensional condition vector and the corresponding condition feature of each auxiliary view image to restore the latent representation and after decoder mapping, a new view image of the target of the ultra-low orbit aircraft is obtained For each view, the lightweight cross-modal attention module (CCSF), the spherical harmonic embedding model (PASHE), and the U-Net denoising network are trained according to the difference between the new view image of the target of the ultra-low orbit aircraft and the image of the view in the multi-view sequence image.

[0037] Step 6, input any view image of the target aircraft to be tested under the ultra-low orbit scene and the extrinsic parameter matrix of the camera under the new view into the trained lightweight cross-modal attention module, the spherical harmonic embedding model, and the U-Net denoising network to obtain the new view image of the target aircraft to be tested.

[0038] The new view image is tested, and a single view image and a corresponding camera extrinsic parameter matrix are input, wherein the extrinsic parameter matrix is used to specify the target view and attitude expected to be generated. The model uses the trained lightweight cross-modal attention module and the spherical harmonic embedding module for forward inference to generate the image under the corresponding target view. During the test process, the model weight is kept frozen, only forward generation is performed, and no gradient update is performed.

[0039] Figure 4 The new view image generated by the view image generation method of cross-modal fusion and multi-frequency encoding in the ultra-low orbit scene of the embodiment is shown in the following table. The first column is the original image of the satellite solar panel in the ultra-low orbit scene observed by the imaging load on the simulated aircraft, and the second to fifth columns are other new view images generated from the first column input image, which are rotated at an angle interval of 60 degrees around the center, with the pitch angle and roll angle unchanged. The performance of the model in view migration and multi-view consistency is verified. Figure One

[0040] According to Figure 4 It can be clearly seen that the satellite solar panel still maintains a straight structure, sharp edges, and no obvious stretching or breaking under large angle changes; thin components such as antennas can also maintain integrity and visibility during rotation, with almost no occlusion, inversion, or deformation artifacts. The correspondence between pixels and view directions is explicitly established through the perception coding of light direction vectors, the Fourier position coding of light origin coordinates, and the pose coding of camera pose parameters, so that the new view image has stronger geometric robustness at the detail level; the lightweight cross-modal attention module ensures the consistency of appearance and illumination between different views through the fusion of semantics and camera pose parameters, so that the metal reflection and highlight area are naturally continuous in multi-view. The synergistic effect of the two makes the generated new view satellite solar panel have higher realism and consistency in structure, with clean occlusion boundaries, reduced background residual images, and more stable and natural overall generation.

[0041] ​The above merely illustrates the specific embodiments of the present application, but the protection scope of the present application is not limited thereto, any change or replacement within the technical scope disclosed by the present application should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for generating viewpoint maps through cross-modal fusion and multi-frequency coding in ultra-low orbit scenarios, characterized in that, Includes the following steps: Step 1: In the ultra-low orbit scenario, acquire multi-view sequence images and corresponding camera extrinsic matrix for the aircraft target, and calculate the corresponding camera pose parameters according to the width, height and set field of view of the image. Take one of the view sequence images as the main view image and the remaining view sequence images as the auxiliary view images. Step 2: Use the CLIP encoder to extract high-dimensional semantic features from the main view image, concatenate the high-dimensional semantic features with the corresponding camera pose parameters of the main view image, and input them into a lightweight cross-modal attention module for cross-modal attention fusion to generate a high-dimensional conditional vector of the main view. Step 3: For each auxiliary viewpoint image, a spherical harmonic function embedding model based on multi-frequency coding is used to perceptually encode the light direction vector, and a multi-frequency sine and cosine function expansion is used to perform Fourier position coding on the coordinates of the light source, and the corresponding camera pose parameters are also coded. The above coding results are spliced, fused and compressed according to the dimensions to obtain the corresponding conditional features of each auxiliary viewpoint image. Step 4: Construct a latent diffusion model consisting of a U-Net denoising network, a variational autoencoder, and a decoder; input the multi-view sequence images of the aircraft target into the variational autoencoder to obtain clean latent representation tensors. The latent representation tensor is obtained by forward diffusion in the latent space. , as input to the U-Net denoising network; Step 5: Input the high-dimensional conditional vector of the main viewpoint and the corresponding conditional features of each secondary viewpoint image into the U-Net denoising network to guide the latent representation tensor. Iterative denoising is performed to restore the latent representation under the target's perspective, and after being mapped by the decoder, a new perspective image of the ultra-low orbit target is obtained. For each perspective, based on the difference between the new perspective image of the ultra-low orbit vehicle target obtained and the image of that perspective in the multi-view sequence image, the lightweight cross-modal attention module, the spherical harmonic function embedding model, and the U-Net denoising network are trained. Step 6: Input the image of the target aircraft under test from any perspective in the ultra-low orbit scenario and the extrinsic parameter matrix of the camera from the new perspective into the trained lightweight cross-modal attention module, spherical harmonic function embedding model and U-Net denoising network to obtain the new perspective image of the target aircraft under test.

2. The viewpoint map generation method for cross-modal fusion and multi-frequency coding in ultra-low orbit scenarios according to claim 1, characterized in that, In step 1, the camera pose parameters include position parameters representing the camera's position in the world coordinate system, attitude parameters representing the camera's orientation in the world coordinate system, and frustum parameters representing the main view direction vector and the scale information of the imaging plane.

3. The viewpoint map generation method for cross-modal fusion and multi-frequency coding in ultra-low orbit scenarios according to claim 1, characterized in that, Step 2 is as follows: Step 2.1: Extract high-dimensional semantic features from the main viewpoint image using the CLIP encoder; Step 2.2: Sequentially concatenate the high-dimensional semantic features of the main view image with the corresponding camera pose parameters of the main view image, perform a one-time linear mapping, and apply layer normalization to form a cross-modal initial feature representation. ; Step 2.3: Represent the initial features across modalities. The input is a lightweight cross-modal attention module consisting of two Transformer encoder layers, which learns the nonlinear coupling relationship between image semantics and geometric pose to obtain conditional feature vectors with cross-modal correlation. ; Step 2.4: Conditional feature vectors with cross-modal correlation After projection transformation into a cross-attention conditional vector, a high-dimensional conditional vector of the main viewpoint is obtained.

4. The viewpoint map generation method for cross-modal fusion and multi-frequency coding in ultra-low orbit scenarios according to claim 3, characterized in that, Step 2.2 specifically involves: Step 2.2.1: Extract high-dimensional semantic features from the main viewpoint image. With camera pose parameters Concatenation is performed along the feature dimension, expressed as follows: ； This represents the concatenated vector. Indicates dimension; Step 2.2.2: Process the concatenated vector A one-time linear mapping is performed, followed by layer normalization, to form an initial feature representation across modalities. The expression is: ； in, This indicates a normalization operation. The weight matrix represents the linear mapping. , The bias vector represents the linear mapping. .

5. The viewpoint map generation method for cross-modal fusion and multi-frequency coding in ultra-low orbit scenarios according to claim 3, characterized in that, In step 2.3, both layers of the Transformer encoder include a multi-head self-attention mechanism and a feedforward network, with the following expressions: ； ； in, This represents the conditional feature vector of the first-layer Transformer encoder; This represents the conditional feature vector of the second-layer Transformer encoder, which is a conditional feature vector with cross-modal correlation.

6. The viewpoint map generation method for cross-modal fusion and multi-frequency coding in ultra-low orbit scenarios according to claim 5, characterized in that, Step 2.4 specifically involves: Conditional feature vectors with cross-modal correlation Linearly map back to the semantic dimension of the original CLIP encoder to generate cross-attention conditional vectors. This yields the high-dimensional conditional vector from the main perspective; the expression is: ； In the formula, Represents the linear mapping weight matrix. , Represents the bias matrix of the linear mapping. .

7. The viewpoint map generation method for cross-modal fusion and multi-frequency coding in ultra-low orbit scenarios according to claim 5, characterized in that, Step 3 specifically involves: Step 3.1: Employ a spherical harmonic function embedding model based on multi-frequency coding for each ray direction vector in each secondary viewpoint image. After normalization, the ray direction distribution is obtained. A spherical harmonic function is then used to perform perceptual encoding on the ray direction distribution, and the order is selected. ,get 3D directional encoding feature vector ; Step 3.2: Set the coordinates of the ray's origin. Fourier position coding is performed using a multi-frequency sine and cosine function expansion based on neural radiation fields, specifically as follows: Define six frequency bands , Then each coordinate dimension Mapped to: ； In the formula, Represents coordinate dimension The mapping result, j represents the component of the coordinate direction of the starting point of the ray, with values ​​of x, y, and z; The coordinates of the starting point of the ray are: ; The coordinates of the ray's origin are linearly transformed to 32 dimensions to obtain the position code of the ray's origin coordinates. ; Step 3.3: Input the camera pose parameters corresponding to each auxiliary viewpoint image into the pose encoder, and map them into a 46-dimensional pose encoding feature vector through a two-layer feedforward network. ; Step 3.4: Encode the direction feature vector Position encoding of the starting coordinates of the light ray Pose encoding feature vector Concatenate the dimensional features into a unified ray condition characteristic. The expression is: ； Unify the characteristics of radiation conditions The input is fused and compressed to 78 dimensions into a two-layer linear network to obtain the conditional features corresponding to each secondary viewpoint image. .

8. The viewpoint map generation method for cross-modal fusion and multi-frequency coding in ultra-low orbit scenarios according to claim 1, characterized in that, In step 4, the multi-view sequence images of the aircraft target are input into a variational autoencoder to obtain a clean latent representation tensor. The expression for adding noise in the latent space via forward diffusion is: ； in, For the potential representation tensor, Let be the cumulative noise modulation coefficient at time step t, i be the view index, and t be the diffusion time step. To and Standard Gaussian noise of the same shape express The covariance.

9. The viewpoint map generation method for cross-modal fusion and multi-frequency coding in ultra-low orbit scenarios according to claim 1, characterized in that, In step 5, the high-dimensional conditional vector of the main viewpoint is input into the cross-attention layer of the U-Net denoising network, and the high-dimensional conditional vector of the main viewpoint is used as a key value to modulate the latent representation tensor. The semantic distribution; The conditional features corresponding to each secondary viewpoint image are input into the convolutional branch of the U-Net denoising network. The conditional features corresponding to each secondary viewpoint image are then processed by the convolutional branch and combined with the latent representation tensor. Point-by-point fusion is used to provide geometric priors.

Images