A long tail data generation method based on feedforward stylization reconstruction

Abstract

This invention discloses a long-tail data generation method based on feedforward stylized reconstruction, comprising the following steps: constructing multimodal geometric perception input and feature encoding; reconstructing a semantically guided scene based on a feature-level linear modulation mechanism; and performing 3D consistent and controllable style transfer based on the feedforward reconstruction model. This invention constructs a feedforward reconstruction network based on an improved DINOv2. By performing multimodal adaptation on the backbone network input layer and explicitly injecting decoupled high-level semantic features using a feature-level linear modulation mechanism, the geometric accuracy of autonomous driving scene reconstruction is improved. Based on the 3D scene reconstructed by feedforward, this invention employs a distillation strategy to transfer the generation capability of the 2D diffusion model to a 3D Gaussian field. This invention effectively avoids phenomena such as object deformation and disappearance in the generated data, and can generate geometrically consistent long-tail weather data such as rain and snow through text commands.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving perception data generation, and in particular to a long-tail data generation method based on feedforward stylized reconstruction. Background Technology

[0002] With the deepening development of autonomous driving technology, the robustness of environmental perception algorithms in complex long-tail scenarios, especially under extreme weather conditions, has become a key factor determining system safety. However, the distribution of data in the real world exhibits a significant long-tail distribution characteristic. Compared to the massive amount of data from sunny days, samples from adverse weather conditions such as rain, snow, and fog are scarce and costly to obtain. This severe imbalance in data distribution leads to a significant deficiency in the generalization ability of existing perception models when facing extreme conditions. Although using 3D reconstruction technology to obtain explicit geometry and perform scene style editing provides a new approach to data augmentation, it still faces significant bottlenecks in practical applications: on the one hand, traditional reconstruction methods based on iterative optimization, while achieving high reconstruction accuracy, usually require retraining for specific scenarios, lacking cross-scenario generalization ability and failing to meet the efficiency requirements of large-scale data generation; on the other hand, while emerging feedforward reconstruction networks have the advantage of fast inference, they often suffer from severe geometric ambiguity and structural collapse due to a lack of high-level semantic constraints, making it difficult to support the generation of high-quality autonomous driving data. Furthermore, in the stylization generation stage, existing methods mostly rely on 2D diffusion models for video editing. Due to the lack of explicit 3D geometric constraints, the generated video sequences often exhibit phenomena such as object flickering and deformation of rigid objects, resulting in generated data that fails to provide accurate geometry to meet the realism requirements of perception systems. Furthermore, because such methods directly generate 2D images, lacking 3D display representation, they struggle to provide sufficient 3D data support for downstream tasks such as closed-loop simulation of perception algorithms. Therefore, there is an urgent need to develop a method for generating long-tail data for autonomous driving that integrates semantic guidance to resolve geometric ambiguities in feedforward reconstruction and achieves controllable style transfer with 3D scene consistency. Summary of the Invention

[0003] To address the aforementioned technical problems, this invention provides a long-tail data generation method based on feedforward stylization reconstruction. This method not only enables high-precision feedforward 3D scene reconstruction but also allows for geometrically consistent style transfer, thereby efficiently generating long-tail data covering various weather conditions.

[0004] To achieve the above objectives, the technical solution of the present invention is as follows: A method for generating long-tail data based on feedforward stylization reconstruction includes the following steps: A. Constructing multimodal geometric perception input and feature encoding A1. Read the data and calculate the geometric embedding, the data including RGB image data. The camera's intrinsic and extrinsic parameter matrices, where and These represent the height and width of the image, respectively; the Planck ray corresponding to each pixel is calculated based on the principle of ray projection, and the Planck ray is determined by the direction vector of the ray. and torque vector The composition, after normalization, is used as a geometric embedding feature, denoted as... .

[0005] A2. Construct a multimodal input tensor, converting the original RGB image into a 3-channel input tensor. With 6-channel Planck ray embedding By concatenating the data along the channel dimension, a 9-channel multimodal input tensor containing both visual and geometric information is constructed. .

[0006] A3. Initialize the weights of the input layer of the backbone network by expanding them. DINOv2 is selected as the backbone network for feature extraction. Since the embedding layer of the backbone network during pre-training only adapts to 3-channel input, the original 3-channel RGB weights are retained to address the channel mismatch issue. The weights of the six newly added geometric channels are Gaussian initialized using scaling variance, and a new weight matrix is ​​calculated. The formula is as follows:

[0007] In the formula, To scale the variance, This indicates that a Gaussian distribution with a mean of 0 is used to generate random noise.

[0008] A4. Complete positional encoding adaptation and temporal embedding injection. Addressing the issue of inconsistency between the input image resolution and the DINOv2 backbone network input resolution, a bicubic interpolation algorithm is used to encode the original positional data. Adjustments are made to obtain a positional encoding adapted to the current resolution. The formula is as follows:

[0009] In the formula, This represents the bicubic interpolation algorithm. This indicates the pixel size of the image block.

[0010] Subsequently, learnable temporal embeddings are introduced. To process multi-frame temporal information and generate the final input feature sequence for the backbone network, the formula is as follows:

[0011] A5. Implement a hierarchical freezing transfer learning strategy, freezing the first 12 layers of the DINOv2 backbone network, as well as all normalized layers and positional coding layers, keeping their parameters unchanged, and only fine-tuning the last 12 layers, setting their learning rate to 0.1 times that of the subsequent decoder head.

[0012] B. Reconstructing a semantic guidance scenario based on a feature-level linear modulation mechanism B1. Extract semantic features and calculate attention weights, then apply the visual features output by the backbone network. Input is given to a lightweight semantic head, and the dimensionality-reduced semantic features are extracted. Then, the learnable projection vectors are used. Calculate the spatial attention weights for each image patch:

[0013] In the formula, Indicates the first Spatial attention weights for each image patch and They represent the first The and the first The semantic feature vector of each image patch, This indicates the total number of image patches.

[0014] B2. Aggregate and generate a global semantic vector, based on the calculated attention weights. The semantic features of all image patches are weighted and summed to obtain a global semantic vector. .

[0015] B3. Generate modulation parameters and perform feature injection, using a conditional network to integrate the global semantic vector. The mapping is to a set of affine transformation parameters, which include a scaling factor. Translation factor And apply the feature linear modulation mechanism to the original visual features Element-wise modulation is performed to generate semantically enhanced features. The formula is as follows:

[0016] In the formula, Used to limit the modulation amplitude This is element-wise multiplication.

[0017] B4. Decode the 3D Gaussian properties and construct a joint loss, modulating the features. Input the decoding head, regress and predict the geometric and appearance attributes of the 3D Gaussian primitives. The geometric attributes include depth, rotation, and scaling, and the appearance attributes include opacity and color. A joint loss function is constructed for supervision, as shown in the following formula:

[0018] In the formula, This represents learning to perceive the similarity of image patches. This represents the logarithmic L1 depth loss. This represents semantic supervision loss, i.e., semantic feature map. With semantic truth value The sum of the absolute values ​​of the differences between corresponding elements. Indicates the loss due to sky regularization. , and These represent the balanced weight coefficients for depth, sky, and semantic loss terms, respectively.

[0019] C. 3D Consistent and Controllable Style Transfer Based on Feedforward Reconstruction Model C1. Generate a geometry-guided stylized image. Using the 3D scene reconstructed in step B as a geometric prior, render the RGB image of the current viewpoint. Combine the target weather description text and generate a stylized image using the InstructPix2Pix diffusion model. During the denoising process, dual-scale guided noise prediction is used. The formula is as follows:

[0020] In the formula, This is the final noise prediction value. , and Noise prediction is performed under unconditional, image-only, and combined conditions, respectively. and These are the guiding scales for images and text, respectively.

[0021] C2. To address the inter-frame flickering issue in the generated image sequence, a photometric tone matching algorithm is applied to calculate the original rendered image. The mean and standard deviation of the generated stylized image will be used to determine the stylized image. The brightness distribution is forced to align, as shown in the following formula:

[0022] In the formula, This represents the stylized image after final tone alignment. This represents the initial stylized image generated by the diffusion model. This represents the original rendered image based on feedforward reconstruction. and These represent the mean and standard deviation of the image brightness channel, respectively.

[0023] C3. Configure the geometry freeze optimization strategy, load the 3D Gaussian parameters trained in step B4, freeze all properties related to the geometry structure so that they do not receive gradient updates during backpropagation, and only the color is frozen. The geometry-related properties, set as optimization parameters, include position. Rotation Scaling and opacity .

[0024] C4. Perform 3D style distillation based on a mask, generating a geometric confidence mask using the accumulated opacity during the 3D Gaussian rendering process. This is used to filter generation noise in open areas, targeting the image generated in step C2, and minimizing distillation loss. The formula is as follows:

[0025] In the formula, This represents element-wise multiplication. This represents an image rendered based on the current 3D Gaussian field. This refers to a stylized image after color matching.

[0026] Through iterative optimization, 2D style features are distilled onto frozen 3D geometry, ensuring that the final generated scene maintains strict geometric consistency across multiple viewpoints.

[0027] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention constructs a feedforward reconstruction network based on an improved DINOv2. By performing multimodal adaptation on the input layer of the backbone network and explicitly injecting decoupled high-level semantic features using a feature-level linear modulation mechanism, the geometric accuracy of autonomous driving scene reconstruction is improved.

[0028] 2. Based on the feedforward reconstructed 3D scene, this invention employs a distillation strategy to transfer the generation capabilities of the 2D diffusion model to a 3D Gaussian field. This invention effectively avoids phenomena such as object deformation and disappearance in the generated data, and can generate geometrically consistent long-tail weather data such as rain and snow through text commands. Attached Figure Description

[0029] Figure 1 This is a flowchart illustrating the present invention.

[0030] Figure 2 This is a diagram of the semantically guided feedforward reconstruction network structure based on DINOv2.

[0031] Figure 3 This is a schematic diagram of the lightweight semantic header structure.

[0032] Figure 4 This is a flowchart of a 3D consistent and controllable weather style transfer framework. Detailed Implementation

[0033] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings. Figure 1 As shown, an autonomous driving scene generation method based on semantic-guided feedforward reconstruction and geometrically frozen distillation includes the following steps: A. Constructing multimodal geometric perception input and feature encoding A1. First, read the data and calculate the geometric embedding. This specifically includes reading RGB image data. ,in , The image height and width are given respectively. Simultaneously, the camera's intrinsic and extrinsic parameter matrices are read. Based on the principle of ray projection, the Planck ray corresponding to each pixel is calculated. This Planck ray is determined by the ray's direction vector. and torque vector The composition, after normalization, is used as a geometric embedding feature, denoted as... .

[0034] A2. Construct a multimodal input tensor. Input the original RGB image with 3 channels... With 6-channel Planck ray embedding By concatenating the data along the channel dimension, a 9-channel multimodal input tensor containing both visual and geometric information is constructed. .

[0035] A3. Initialize the expansion of the weights in the input layer of the backbone network, such as... Figure 2 The input layer is shown. DINOv2 is selected as the backbone network for feature extraction. Since the embedding layer during pre-training of this network only adapts to 3-channel input, the original 3-channel RGB weights are retained to address the channel mismatch issue. The weights of the six newly added geometric channels are Gaussian initialized using scaling variance, and a new weight matrix is ​​calculated. :

[0036] In the formula, To scale the variance, This indicates that a Gaussian distribution with a mean of 0 is used to generate random noise.

[0037] A4. Complete positional encoding adaptation and temporal embedding injection. To address the issue of inconsistency between the input image resolution and the input resolution of the DINOv2 pre-trained model, a bicubic interpolation algorithm is used to encode the original positional data. Adjustments are made to obtain a positional encoding adapted to the current resolution. :

[0038] In the formula, This represents the bicubic interpolation algorithm. This indicates the pixel size of the image block.

[0039] Subsequently, learnable temporal embeddings are introduced. To process multi-frame temporal information, the feature sequence input to the final backbone network is generated:

[0040] A5. Implement a layer-freezing transfer learning strategy, freezing the first 12 layers and all positional encoding layers of the DINOv2 backbone network, keeping their parameters unchanged, and only fine-tuning the last 12 layers. The features output by the backbone network are then processed as follows: Figure 2 The normalization layer shown is used for standardization to obtain visual features. .

[0041] B. Semantic-guided scene reconstruction based on feature-level linear modulation mechanism B1. Extract semantic features and calculate attention weights. For example... Figure 2 As shown in the semantic guidance branch, the visual features output by the backbone network are... Input is given to a lightweight semantic header. For example... Figure 3 As shown in the structure, the semantic head consists of linear layer 1, activation layer, and linear layer 2, used to extract the dimensionality-reduced semantic features. Then, learnable projection vectors are used. Calculate the spatial attention weights for each image patch :

[0042] In the formula, Indicates the first Spatial attention weights for each image patch and They represent the first The and the first The semantic feature vector of each image patch, This indicates the total number of image patches.

[0043] B2. Aggregate and generate a global semantic vector. For example... Figure 2 As shown in the global vector module, based on the calculated attention weights The semantic features of all image patches are weighted and summed to obtain a global semantic vector. .

[0044] B3. Generate modulation parameters and perform feature injection, using a conditional network to integrate the global semantic vector. The mapping is to a set of affine transformation parameters, including scaling factors. Translation factor and apply such as Figure 2 The semantic modulation module demonstrates a feature linear modulation mechanism for the original visual features. Element-wise modulation is performed to generate semantically enhanced features. :

[0045] In the formula, Used to limit the modulation amplitude This is element-wise multiplication.

[0046] B4. Decode the 3D Gaussian properties and construct the joint loss. For example... Figure 2 As shown in the bottom Gaussian ellipsoid parameter inference module, the modulation features are... Input an appearance head, regress and predict the geometric and appearance attributes of a 3D Gaussian primitive. The geometric attributes include depth, rotation, and scaling, and the appearance attributes include opacity and color. A joint loss function is constructed for supervision, as shown in the following formula:

[0047] In the formula, This represents learning to perceive the similarity of image patches. This represents the logarithmic L1 depth loss. This represents the semantic supervision loss, specifically the semantic feature map. With semantic truth value The sum of the absolute values ​​of the differences between corresponding elements. Indicates the loss due to sky regularization. , and These represent the balanced weight coefficients for depth, sky, and semantic loss terms, respectively.

[0048] C. 3D Consistent and Controllable Style Transfer Based on Feedforward Reconstruction Model C1. Generate geometry-guided stylized images. For example... Figure 4 As shown in the flowchart, the original Gaussian parameters reconstructed in step B are used as geometric priors, and the rendered image is obtained through rasterization. Combined with the target weather text description, a stylized image is generated using the InstructPix2Pix diffusion model. During the denoising process, dual-scale guided noise prediction is employed. The formula is as follows:

[0049] In the formula, This is the final noise prediction value. , and Noise prediction is performed under unconditional, image-only, and combined conditions, respectively. and These are the guiding scales for images and text, respectively.

[0050] C2. To address the inter-frame flickering problem in the generated image sequence, the following methods are applied: Figure 4 The color matching algorithm shown calculates the original rendered image. The mean and standard deviation of the generated stylized image will be used to determine the stylized image. Forced alignment of brightness distribution:

[0051] In the formula, This is the final noise prediction value. , and Noise prediction is performed under unconditional, image-only, and combined conditions, respectively. and These are the guiding scales for images and text, respectively.

[0052] C3. Then configure the geometry freeze optimization strategy. For example... Figure 4 As shown in the style fine-tuning module, the 3D Gaussian parameters trained in step B4 are loaded, and a geometry freeze strategy is implemented, that is, all attributes related to the geometry are frozen so that they do not receive gradient updates during backpropagation, and only the color is updated. Set as an optimizable parameter.

[0053] C4. 3D style distillation based on masks. For example... Figure 4 As shown on the right, a geometric confidence mask is generated using the cumulative opacity during the 3D Gaussian rendering process. This is used to filter generation noise in open areas such as the sky. Using the image generated in step C2 as the target, a masked L1Loss loop closure is constructed to minimize the distillation loss. :

[0054] In the formula, This represents element-wise multiplication. This represents an image rendered based on the current 3D Gaussian field. This refers to a stylized image after color matching.

[0055] Through iterative optimization, 2D style features are distilled onto frozen 3D geometry, ensuring that the final generated scene maintains strict geometric consistency across multiple viewpoints.

[0056] This invention is not limited to this embodiment. Any equivalent concept or modification within the technical scope disclosed in this invention shall be included within the protection scope of this invention.

Claims

1. A method for generating long-tail data based on feedforward stylized reconstruction, characterized in that: Includes the following steps: A. Constructing multimodal geometric perception input and feature encoding; B. Reconstructing semantic guidance scenarios based on feature-level linear modulation mechanisms; B1, extract semantic features and calculate attention weights, visual features output by the backbone network are input to a lightweight semantic head to extract reduced dimension semantic features , and then use a learnable projection vector to calculate the spatial attention weight of each image block; B2, aggregate the global semantic vector, according to the calculated attention weight , the semantic features of all image blocks are weighted and summed to obtain the global semantic vector ; B3, generating modulation parameters and performing feature injection, mapping global semantic vectors into a set of affine transformation parameters including scaling factors and translation factors using conditional networks, and applying an element-wise modulation mechanism on original visual features to generate semantic-enhanced features ;​​​​ B4, decode 3D Gaussian attributes and construct joint loss, modulated features input decoding head, regress predicted 3D Gaussian primitives' geometry attributes including depth, rotation and scale, and appearance attributes including opacity and color, and construct a joint loss function for supervision; C. 3D consistent and controllable style transfer based on feedforward reconstruction model; C1. Generating a geometry-guided stylized image, using the 3D scene reconstructed in step B as a geometry prior, rendering an RGB image of the current viewpoint, combining the target weather description text, and using an InstructPix2Pix diffusion model to generate a stylized image, using a bi-scale guided computation of noise prediction in the denoising process ; C2, for the problem of inter-frame flicker existing in the generated image sequence, a photometric tone mapping algorithm is applied to calculate the mean and standard deviation of the original rendering image , and the brightness distribution of the generated stylized image is forced to align; ; C3. Configure geometry frozen optimization strategy, load the trained 3D Gaussian parameters from step B4, freeze all geometry related attributes so that they do not receive gradient updates during backpropagation, only color is set to be optimized parameter, the geometry related attributes include position , rotation , scale and opacity ; C4. Perform 3D style distillation based on a mask, generating a geometric confidence mask using the accumulated opacity during the 3D Gaussian rendering process. This is used to filter generation noise in open areas, targeting the image generated in step C2, and minimizing distillation loss. ; Through iterative optimization, 2D style features are distilled onto frozen 3D geometry, ensuring that the final generated scene maintains strict geometric consistency across multiple viewpoints.

2. The method for generating long-tail data based on feedforward stylization reconstruction according to claim 1, characterized in that: Step A describes a method for constructing multimodal geometrically aware inputs and feature encodings, which includes the following steps: A1. Read the data and calculate the geometric embedding, the data including RGB image data. The camera's intrinsic and extrinsic parameter matrices, where and These represent the height and width of the image, respectively; the Planck ray corresponding to each pixel is calculated based on the principle of ray projection, and the Planck ray is determined by the direction vector of the ray. and torque vector The composition, after normalization, is used as a geometric embedding feature, denoted as... ; A2. Construct a multimodal input tensor, converting the original RGB image into a 3-channel input tensor. With 6-channel Planck ray embedding By concatenating the data along the channel dimension, a 9-channel multimodal input tensor containing both visual and geometric information is constructed. ; A3. Initialize the weights of the input layer of the backbone network by expanding them. DINOv2 is selected as the backbone network for feature extraction. Since the embedding layer of the backbone network during pre-training only adapts to 3-channel input, the original 3-channel RGB weights are retained to address the channel mismatch issue. The weights of the six newly added geometric channels are Gaussian initialized using scaling variance, and a new weight matrix is ​​calculated. The formula is as follows: In the formula, To scale the variance, This indicates that a Gaussian distribution with a mean of 0 is used to generate random noise; A4. Complete positional encoding adaptation and temporal embedding injection. Addressing the issue of inconsistency between the input image resolution and the DINOv2 backbone network input resolution, a bicubic interpolation algorithm is used to encode the original positional data. Adjustments are made to obtain a positional encoding adapted to the current resolution. The formula is as follows: In the formula, This represents the bicubic interpolation algorithm. Indicates the pixel size of an image block; Subsequently, learnable temporal embeddings are introduced. To process multi-frame temporal information and generate the final input feature sequence for the backbone network, the formula is as follows: A5. Implement a layer-freezing transfer learning strategy, freezing the first 12 layers of the DINOv2 backbone network, as well as all normalized layers and position coding layers, keeping their parameters unchanged, and only fine-tuning the last 12 layers, setting their learning rate to 0.1 times that of the subsequent decoder head.

3. The method for generating long-tail data based on feedforward stylization reconstruction according to claim 1, characterized in that: Step B utilizes learnable projection vectors Calculate the spatial attention weights for each image patch: In the formula, Indicates the first Spatial attention weights for each image patch and They represent the first The and the first The semantic feature vector of each image patch, Indicates the total number of image patches; Step B3 describes generating semantically enhanced features. The formula is as follows: In the formula, Used to limit the modulation amplitude This is element-wise multiplication; Step B4 describes the construction of a joint loss function for supervision, as shown in the following formula: In the formula, This represents learning to perceive the similarity of image patches. This represents the logarithmic L1 depth loss. This represents semantic supervision loss, i.e., semantic feature map. With semantic truth value The sum of the absolute values ​​of the differences between corresponding elements; This indicates the loss due to sky regularization. , and These represent the balanced weight coefficients for depth, sky, and semantic loss terms, respectively.

4. The method for generating long-tail data based on feedforward stylization reconstruction according to claim 1, characterized in that: Step C1 describes generating a stylized image using the InstructPix2Pix diffusion model, and employing dual-scale guided noise prediction during the denoising process. The formula is as follows: In the formula, This is the final noise prediction value. , and Noise prediction is performed under unconditional, image-only, and combined conditions, respectively. and These are the guiding scales for images and text, respectively; Step C2 describes generating a stylized image. The brightness distribution is forced to align, as shown in the following formula: In the formula, This represents the stylized image after final tone alignment. This represents the initial stylized image generated by the diffusion model. This represents the original rendered image based on feedforward reconstruction. and These represent the mean and standard deviation of the image brightness channel, respectively. Step C4 aims to minimize distillation loss using the image generated in step C2. The formula is as follows: In the formula, This represents element-wise multiplication. This represents an image rendered based on the current 3D Gaussian field. This refers to a stylized image after color matching.

Images