A method and apparatus for variational decoding based on cross-layer feature compensation

Abstract

This disclosure relates to a variational decoding method and apparatus based on cross-layer feature compensation. The method includes: inputting an input image into an encoder for forward propagation to obtain latent variables and four intermediate features; the encoder including at least four downsampling stages; inputting the latent variables into a decoder symmetrical to the encoder structure; the decoder including four upsampling stages; the i-th upsampling stage outputting an i-th decoder feature; in the i-th upsampling stage of the decoder, if a target intermediate feature corresponding to the i-th upsampling stage exists among the four intermediate features, fusing the target intermediate feature with the i-th decoder feature to obtain an i-th fused feature; if i is less than 4, inputting the i-th fused feature into the (i+1)-th upsampling stage; if i equals 4, outputting a reconstructed image based on the i-th fused feature. This solves the technical problem of low image quality generated by variational autoencoder architectures in related technologies.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, specifically to a variational decoding method and apparatus based on cross-layer feature compensation. Background Technology

[0002] The standard variational autoencoder (VAE) architecture consists of two parts: an encoder and a decoder. The encoder is responsible for mapping high-dimensional input data to a low-dimensional latent space and outputting a compact latent variable representation. The decoder is responsible for reconstructing the original data from the latent variables, realizing the spatial transformation from compressed representation to complete data.

[0003] However, due to the low spatial resolution of the latent vectors, a large amount of high-frequency spatial information is discarded during the encoding process, and the upsampling operation in the decoder lacks constraints on fine edge structures, resulting in blurred and jagged edges in the reconstructed image. In the reconstruction task of images in autonomous driving scenarios, the above defects manifest as problems such as blurred road boundaries and broken lane markings, indicating that the variational autoencoder architecture in related technologies suffers from the technical problem of generating images with low quality. Summary of the Invention

[0004] This disclosure provides a variational decoding method and apparatus based on cross-layer feature compensation to solve the technical problem of low image quality in variational autoencoder architecture.

[0005] In a first aspect, embodiments of this disclosure provide a variational decoding method based on cross-layer feature compensation, comprising: inputting an input image into an encoder to perform forward propagation, obtaining latent variables and four intermediate features, wherein the encoder includes at least four downsampling stages, and the intermediate features are feature maps output by the downsampling stages; inputting the latent variables into a decoder symmetrical to the encoder structure, wherein the decoder includes four upsampling stages, the i-th upsampling stage outputs the i-th decoder feature, i being an integer greater than 0 and less than or equal to 4; in the i-th upsampling stage of the decoder, if there is a target intermediate feature among the four intermediate features corresponding to the i-th upsampling stage, fusing the target intermediate feature with the i-th decoder feature to obtain the i-th fused feature; if i is less than 4, inputting the i-th fused feature into the (i+1)-th upsampling stage; and if i is equal to 4, outputting a reconstructed image based on the i-th fused feature.

[0006] Optionally, the target intermediate feature is fused with the i-th decoder feature to obtain the i-th fused feature, including: determining the target projection layer corresponding to the i-th upsampling stage among the four projection layers, wherein the projection layer is used to perform point convolution projection on the intermediate feature; projecting the channel dimension of the target intermediate feature to be consistent with the channel dimension of the i-th decoder feature through the target projection layer to obtain the target projection feature; and fusing the target projection feature with the i-th decoder feature to obtain the i-th fused feature.

[0007] Optionally, fusing the target projection feature with the i-th decoder feature to obtain the i-th fused feature includes: when the spatial dimensions of the target projection feature and the i-th decoder feature are consistent, adding the target projection feature and the i-th decoder feature element by element to obtain the i-th fused feature, wherein the spatial dimension is used to indicate the spatial resolution of the feature.

[0008] Optionally, the input image is input to the encoder for forward propagation to obtain four intermediate features, including one of the following: returning the intermediate features at each of the four downsampling stages of the encoder, adding the four intermediate features to a buffer list, wherein the forward propagation function of the four downsampling stages includes a feature buffer function; or, after the forward hook function of the four downsampling stages, outputting the four intermediate features to the buffer list.

[0009] Optionally, when the input image is a road image, the four intermediate features include at least one of the following: a first feature map, used to carry road edge response, lane marking texture, road surface material contrast, and obstacle contour gradient information; a second feature map, used to carry traffic sign components, vehicle area contours, and intermediate texture information of road markings; a third feature map, used to carry vehicle-level semantics, road scene layout, and lane semantic consistency information; and a fourth feature map, used to carry driving scene-level semantics, road core skeleton, and global contextual information of the traffic environment.

[0010] Optionally, before inputting the input image into the encoder to perform forward propagation, the process includes: acquiring a pre-trained encoder and decoder, and freezing the parameter weights of the encoder; initializing four projection layers, and performing multiple rounds of training on the four projection layers and the decoder; in the p-th round of training, based on the p-th training sample in the training sample set, performing the following operations: inputting the p-th training sample into the encoder to perform forward propagation, obtaining four reference intermediate features and a p-th reference latent variable, where p is an integer greater than 0; inputting the p-th reference latent variable and the four reference intermediate features into the decoder, and after four upsampling stages, obtaining the p-th reference image; obtaining a first loss based on the p-th reference latent variable, and determining a second loss, a third loss, and a fourth loss based on the p-th reference image and the p-th training sample, wherein the first loss is used to indicate the latent... The spatial structure features are analyzed as follows: the first loss is determined based on the difference between the variational posterior distribution and the standard Gaussian prior distribution corresponding to the reference latent variable; the second loss is used to indicate the pixel-level error between the training sample and the reference image, determined by pixel-by-pixel comparison of the reference image and the training sample; the third loss is used to indicate the perceptual similarity between the training sample and the reference image, determined based on the feature differences between the reference image and the training sample in the perceptual network; and the fourth loss is used to indicate the edge error between the training sample and the reference image, determined based on the mean square error of the edge response maps corresponding to the reference image and the training sample. If the training termination condition is not met, the parameters of the four projection layers and the decoder are updated based on the first, second, third, and fourth losses; if the training termination condition is met, the training ends.

[0011] Optionally, initializing the projection layer includes: determining the input channel dimension of the input channel corresponding to the projection layer; determining the standard deviation by the reciprocal of the square root of the input channel dimension, and initializing the weights of the projection layer based on the Gaussian distribution determined by the standard deviation; and initializing the bias term of the projection layer to a preset bias value.

[0012] Secondly, embodiments of this disclosure provide a variational decoding apparatus based on cross-layer feature compensation, comprising: an encoding module for inputting an input image into an encoder to perform forward propagation, obtaining latent variables and four intermediate features, wherein the encoder includes four downsampling stages, and the intermediate features are feature maps output by the downsampling stages; a decoding module for inputting the latent variables into a decoder symmetrical to the encoder structure, wherein the decoder includes four upsampling stages, the i-th upsampling stage outputs the i-th decoder feature, where i is an integer greater than 0 and less than or equal to 4; a feature fusion module for fusing the target intermediate feature with the i-th decoder feature in the i-th upsampling stage of the decoder, when there is a target intermediate feature among the four intermediate features corresponding to the i-th upsampling stage, to obtain the i-th fused feature; a first judgment module for inputting the i-th fused feature into the (i+1)-th upsampling stage when i is less than 4; and a second judgment module for outputting a reconstructed image based on the i-th fused feature when i is equal to 4.

[0013] Optionally, the feature fusion module is further configured to: determine the target projection layer corresponding to the i-th upsampling stage among the four projection layers, wherein the projection layer is used to perform point convolution projection on the intermediate features; project the channel dimension of the target intermediate features to be consistent with the channel dimension of the i-th decoder feature through the target projection layer to obtain the target projection feature; and fuse the target projection feature with the i-th decoder feature to obtain the i-th fused feature.

[0014] Optionally, the feature fusion module is further configured to: add the target projection feature and the i-th decoder feature element by element to obtain the i-th fused feature, provided that the spatial dimensions of the target projection feature and the i-th decoder feature are consistent, wherein the spatial dimension is used to indicate the spatial resolution of the feature.

[0015] Optionally, the encoding module is also configured to: return intermediate features in each of the four downsampling stages of the encoder, add the four intermediate features to a buffer list, wherein the forward propagation function of the four downsampling stages includes a feature buffer function; and output the four intermediate features to the buffer list after passing the forward hook function of the four downsampling stages.

[0016] Optionally, when the input image is a road image, the four intermediate features include at least one of the following: a first feature map, used to carry road edge response, lane marking texture, road surface material contrast, and obstacle contour gradient information; a second feature map, used to carry traffic sign components, vehicle area contours, and intermediate texture information of road markings; a third feature map, used to carry vehicle-level semantics, road scene layout, and lane semantic consistency information; and a fourth feature map, used to carry driving scene-level semantics, road core skeleton, and global contextual information of the traffic environment.

[0017] Optionally, the variational decoding device based on cross-layer feature compensation further includes: a training module for acquiring pre-trained encoders and decoders and freezing the parameter weights of the encoder; initializing four projection layers and performing multiple rounds of training on the four projection layers and the decoder; in the p-th round of training, based on the p-th training sample in the training sample set, performing the following operations: inputting the p-th training sample into the encoder to perform forward propagation, obtaining four reference intermediate features and the p-th reference latent variable, where p is an integer greater than 0; inputting the p-th reference latent variable and the four reference intermediate features into the decoder, and obtaining the p-th reference image after four upsampling stages; obtaining a first loss based on the p-th reference latent variable, and determining a second loss, a third loss, and a fourth loss based on the p-th reference image and the p-th training sample, wherein the first loss is used to indicate The structured features of the latent space are defined as follows: the first loss is determined based on the difference between the variational posterior distribution and the standard Gaussian prior distribution corresponding to the reference latent variable; the second loss is used to indicate the pixel-level error between the training sample and the reference image, determined by pixel-by-pixel comparison of the reference image and the training sample; the third loss is used to indicate the perceptual similarity between the training sample and the reference image, determined based on the feature differences between the reference image and the training sample in the perceptual network; and the fourth loss is used to indicate the edge error between the training sample and the reference image, determined based on the mean square error of the edge response maps corresponding to the reference image and the training sample. If the training termination condition is not met, the parameters of the four projection layers and the decoder are updated based on the first, second, third, and fourth losses; if the training termination condition is met, the training ends.

[0018] Optionally, the training module is also used to: determine the input channel dimension of the input channel corresponding to the projection layer; determine the standard deviation by the reciprocal of the square root of the input channel dimension, and initialize the weights of the projection layer based on the Gaussian distribution determined by the standard deviation; and initialize the bias term of the projection layer to a preset bias value.

[0019] Thirdly, embodiments of this disclosure provide a computer-readable storage medium having a computer program stored thereon, characterized in that the program, when executed by a processor, implements the method as described in the first aspect above.

[0020] Fourthly, embodiments of this disclosure provide a computing device, including: a memory for storing a computer program; and a processor for executing the computer program stored in the memory, wherein when the computer program is executed, it implements the method as described in the first aspect above.

[0021] Through the above methods, the multi-scale information that should have been discarded at each downsampling level of the encoder is preserved and passed to the decoder by the intermediate feature synchronous capture mechanism during the encoder's forward propagation process; the real structural information captured by the encoder is injected into the decoder reconstruction process through the cross-layer fusion mechanism of the decoder; and a multi-scale constraint chain from global structure to local edges is formed through the step-by-step transfer mechanism of fused features. Thus, the decoder obtains additional information channels beyond the highly compressed latent vectors, alleviating high-frequency loss caused by information bottlenecks and achieving a synergistic improvement in the structural consistency and detail fidelity of the reconstructed image. Attached Figure Description

[0022] The accompanying drawings, which are included to provide a further understanding of this disclosure and form part of this disclosure, illustrate exemplary embodiments of the present disclosure and are used to explain the disclosure, but do not constitute an undue limitation of the disclosure. In the drawings: Figure 1 This is a flowchart of an optional variational decoding method based on cross-layer feature compensation according to an embodiment of the present disclosure; Figure 2 This is a schematic diagram of the structure of an optional encoder according to an embodiment of the present disclosure; Figure 3 This is a schematic diagram of the structure of an optional decoder according to an embodiment of the present disclosure; Figure 4 This is a schematic diagram of an optional variational decoding method based on cross-layer feature compensation according to an embodiment of the present disclosure; Figure 5 This is a schematic diagram of an optional variational decoding device based on cross-layer feature compensation according to an embodiment of the present disclosure; Figure 6 This is a schematic diagram of an optional electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0023] The present disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Through these descriptions, the features and advantages of the present disclosure will become clearer and more apparent.

[0024] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments. Although various aspects of embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless specifically indicated otherwise.

[0025] Furthermore, the technical features involved in the different embodiments of this disclosure described below can be combined with each other as long as they do not conflict with each other.

[0026] According to one aspect of the embodiments of this disclosure, a variational decoding method based on cross-layer feature compensation is provided. Figure 1 This is a flowchart illustrating a variational decoding method based on cross-layer feature compensation provided in this disclosure; as follows: Figure 1 As shown, the method may include the following steps: Step S101: Input the input image into the encoder to perform forward propagation, and obtain latent variables and 4 intermediate features. The encoder includes at least 4 downsampling stages, and the intermediate features are feature maps output by the downsampling stages.

[0027] The number of encoder and decoder layers is not limited to 4 layers. The input image can be input into the encoder to perform forward propagation, and latent variables and N intermediate features can be obtained. The encoder includes at least N downsampling stages, and the intermediate features are feature maps output by the downsampling stages. N is an integer greater than 0.

[0028] The input image refers to the raw visual data to be processed, typically a two-dimensional pixel matrix containing height, width, and color channel information. For example, in an autonomous driving scenario, the input image can be a road scene photograph captured by an onboard camera, containing visual elements such as roads, vehicles, pedestrians, and traffic signs.

[0029] The encoder progressively compresses the spatial resolution and extracts abstract features through stepwise downsampling operations, ultimately outputting a compact latent representation. Forward propagation is the complete computational process where the input image starts from the input layer, passes through each layer of the network sequentially, and is unidirectionally passed to the output layer.

[0030] Figure 2 This is a schematic diagram of the structure of an optional encoder according to an embodiment of the present disclosure; as shown... Figure 2 As shown, the encoder may include n downsampling stages. After n downsampling stages, a latent vector can be obtained. Figure 2 In this model, N can be 2, and the output feature maps of the second and (n-1)th downsampling stages are used as intermediate features. The intermediate features are the feature maps output by each downsampling stage. A downsampling stage is a specific processing layer in the encoder that reduces the spatial resolution of the feature maps through operations such as pooling or stride convolution. Each downsampling stage outputs one intermediate feature map. Latent variables are low-dimensional compressed representations of the encoder's output, carrying the core semantic information of the input image. Latent variables typically have extremely low spatial resolution and a small number of channels, representing a highly abstract generalization of the input image by the encoder.

[0031] In a standard variational autoencoder, the intermediate feature maps of each encoder level proceed to the next layer after completing the computation of the current layer, ultimately retaining only the lowest-resolution latent variables as output. The process of obtaining latent variables through the encoder's forward propagation is well-established in relevant technologies and will not be elaborated upon here. However, the extraction of intermediate features is not addressed in these technologies. The following explains how to obtain N intermediate features: In an optional implementation, the input image is input to the encoder for forward propagation to obtain four intermediate features, including one of the following: returning the intermediate features at each of the four downsampling stages of the encoder, adding the four intermediate features to a buffer list, wherein the forward propagation function of the four downsampling stages includes a feature buffer function; or, after the forward hook function of the four downsampling stages, outputting the four intermediate features to the buffer list.

[0032] When the number of layers in the encoder and decoder is not limited to 4, the input image can be input into the encoder to perform forward propagation and obtain N intermediate features, including one of the following: returning intermediate features in each of the N downsampling stages of the encoder, adding the N intermediate features to the buffer list, wherein the forward propagation function of the N downsampling stages includes a feature buffer function; or outputting the N intermediate features to the buffer list after the forward hook function of the N downsampling stages.

[0033] The cache list can be a data structure for storing intermediate features, such as a list or dictionary. It can save the output features of each downsampling stage in hierarchical order for easy retrieval in subsequent decoding stages. The feature caching function can be a specific code segment embedded in the encoder's forward propagation function, responsible for copying intermediate features at a specified level and adding them to the cache list. This function is triggered after the downsampling stage calculation is completed, achieving explicit feature saving.

[0034] In an optional implementation, the encoder's forward propagation function definition is modified to add a feature caching function to the internal logic. Specifically, in the forward propagation method of the encoder class, an empty cache list object is defined to receive the output features of each downsampling stage. The encoder forward propagation executes each downsampling stage sequentially. After the calculation of each downsampling stage is completed, the feature caching function is called to copy the output features of that stage and append them to the cache list. The feature caching function, as an integral part of the forward propagation function, is tightly integrated with the downsampling calculation logic. After all downsampling stages are completed, the encoder outputs the latent variable at the end. The forward propagation function ultimately returns two objects: the final latent variable and a cache list containing intermediate features from each level. The features in the cache list are arranged in the order of the downsampling stages, with the first element corresponding to the output of the first downsampling stage and the last element corresponding to the output of the last downsampling stage.

[0035] In another alternative implementation, after encoder instantiation and before forward propagation, a forward hook function is registered for each target downsampling layer (downsampling phase). Hook registration can be accomplished through an interface provided by the deep learning framework, specifying the target layer and the callback function. The hook callback function is defined (e.g., the target layer object, the layer's input tensor, and the layer's output tensor). Inside the callback function, the output tensor is copied and saved to an externally predefined cache dictionary.

[0036] During encoder forward propagation, data flows normally through each layer. When computation reaches the target layer of a registered hook, the framework automatically calls the corresponding callback function and saves the layer's output to a cache dictionary. After forward propagation, intermediate features are extracted from the cache dictionary and sorted by layer index to form an ordered set. After forward propagation, registered hooks can be removed to release related resources and avoid interfering with subsequent computations.

[0037] Through the above-described embodiments of this disclosure, two optional caching implementation methods are provided: explicit output and forward hooks. The explicit output method is suitable for deeply customized scenarios, enabling efficient integration; the hook mechanism method is suitable for rapid deployment scenarios, enabling plug-and-play functionality. Both methods ensure that intermediate features are captured and stored in an orderly manner in real time and accurately during the encoder's forward propagation, providing a reliable information source for hierarchical feature compensation in the subsequent decoding stage.

[0038] It should be noted that the captured intermediate features are stored in memory as tensors, with each feature having four dimensions: batch size, channels, height, and width. Memory usage can be calculated based on the feature map size and number. Taking a single input image as an example, the total memory usage of the four cached features is determined by the spatial resolution, number of channels, and numerical precision of each feature.

[0039] For example, for batch B=1, the four extracted features are: channel 128, height and width are half of the original height and width; channel 256, height and width are one-quarter of the original height and width; channel 512, height and width are one-eighth of the original height and width; and channel 512, height and width are one-sixteenth of the original height and width. Then the total video memory usage... It could be: If the original height and width are 512, then the total video memory usage is approximately 52.43MB. This is relatively low for most graphics processing units (GPUs).

[0040] Back Figure 1 In step S102, the latent variables are input into a decoder that is symmetrical to the encoder structure. The decoder includes four upsampling stages, and the i-th upsampling stage outputs the i-th decoder feature, where i is an integer greater than 0 and less than or equal to 4.

[0041] Optionally, if the number of layers in the codec is not limited to 4, the latent variables are input into the decoder that matches the encoder, wherein the decoder includes M upsampling stages, the i-th upsampling stage outputs the i-th decoder feature, M is an integer greater than or equal to N, and i is an integer greater than 0 and less than or equal to M.

[0042] The decoder is a neural network structure symmetrical to or matching the encoder, responsible for restoring latent variables to reconstructed data with the same or similar dimensions as the original input. The decoder gradually restores spatial resolution and shrinks channel dimensions through successive upsampling operations, ultimately outputting a reconstructed image. The upsampling stage is a specific processing layer in the decoder that improves the spatial resolution of the feature maps through operations such as interpolation or transposed convolution, while adjusting the channel dimensions to gradually approach the final output requirements. Each upsampling stage outputs a decoder feature.

[0043] Provided that the number of decoder stages M is greater than the number of encoder buffer features N, there is no limit to the number of upsampling layers in the encoder and the number of downsampling layers in the decoder.

[0044] Step S103: In the i-th upsampling stage of the decoder, if there is a target intermediate feature among the four intermediate features that corresponds to the i-th upsampling stage, the target intermediate feature is fused with the i-th decoder feature to obtain the i-th fused feature.

[0045] Optionally, if the number of layers in the codec is not limited to 4, in the i-th upsampling stage of the decoder, if there is a target intermediate feature among the N intermediate features that corresponds to the i-th upsampling stage, the target intermediate feature is fused with the i-th decoder feature to obtain the i-th fused feature.

[0046] Figure 3 This is a schematic diagram of the structure of an optional decoder according to an embodiment of the present disclosure; as shown... Figure 3As shown, the decoder can include multiple upsampling stages. After the latent variables are input into the decoder, the reconstructed image is obtained after multiple upsampling stages. In the i-th upsampling stage of the decoder (any upsampling stage), the i-th decoder feature can be output normally. If the target intermediate feature exists in the corresponding stage, the target intermediate feature can be fused with the i-th decoder feature to obtain the i-th fused feature. The matching criteria for the target intermediate feature can be pre-defined, that is, the intermediate feature of a certain downsampling stage of the encoder can be pre-matched with a certain upsampling stage of the decoder, or intermediate features with the same spatial dimension can be considered to be matched with the i-th decoder feature. Fusion is to combine two feature maps of the same dimension to generate a new feature representation. The fused feature can be used as the final output of the current upsampling stage, input to the next stage, or used as the basis for the reconstructed image.

[0047] It's important to note that the intermediate features output from each downsampling stage of the encoder have different channel dimensions. Channel dimension indicates the size of the feature map along the channel axis, representing the number of independent feature channels it contains. For example, shallow edge features might have 128 channels, mid-level semantic features might have 256 channels, and deep structural features might have 512 channels. Similarly, the decoder features output from each upsampling stage of the decoder also have independent channel configurations; for example, some stages might have 512 channels, while others might have 256 channels. Since the channel design of the encoder and decoder is usually optimized independently, the number of channels in the intermediate features often differs from that of the encoder features. Therefore, channel alignment is required before feature fusion, which can be achieved using a projection layer.

[0048] In an optional implementation, fusing the target intermediate feature with the i-th decoder feature to obtain the i-th fused feature includes: determining the target projection layer corresponding to the i-th upsampling stage among the four projection layers, wherein the projection layer is used to perform point convolution projection on the intermediate feature; projecting the channel dimension of the target intermediate feature to be consistent with the channel dimension of the i-th decoder feature through the target projection layer to obtain the target projection feature; and fusing the target projection feature with the i-th decoder feature to obtain the i-th fused feature.

[0049] The number of projection layers is the same as the number of intermediate features. Therefore, when the number of intermediate features is N, the number of projection layers is also N. Each projection layer can be a neural network layer composed of 1×1 convolutional kernels, responsible for performing linear transformations along the channel dimension. The projection layer does not change the spatial resolution of the feature map; it only adjusts the number of channels to make the channel dimension of the input features consistent with the target channel dimension. The target projection layer is the specific projection layer that corresponds to the current i-th upsampling stage among the N projection layers. The input channel dimension of this projection layer matches the channel dimension of the target intermediate features, and the output channel dimension matches the channel dimension of the decoder features.

[0050] In an optional implementation, each projection layer corresponds to an intermediate feature output from the encoder's downsampling stage. The projection layer is implemented using a 1×1 convolution with a 1×1 kernel size, a stride of 1, and zero padding. The input channel dimension of each projection layer is consistent with the channel dimension of the corresponding intermediate feature, and the output channel dimension is consistent with the channel dimension of the corresponding decoder feature. For example, if an intermediate feature has 128 channels and the corresponding decoder feature has 512 channels, then the projection layer is configured with 128 input channels and 512 output channels.

[0051] After computation is performed in the i-th upsampling stage of the decoder, the i-th decoder feature is obtained. At this point, the target projection layer corresponding to this stage needs to be determined from among the N projection layers. This can be done by traversing the configuration information of the N projection layers to find the projection layer whose input channel dimension matches the current target intermediate feature channel dimension and whose output channel dimension matches the current decoder feature channel dimension. This projection layer is the target projection layer.

[0052] The 1×1 convolution operation in the projection layer is performed pixel-by-pixel in the spatial dimension. For each spatial location on the feature map, the 1×1 convolution kernel receives the values ​​of all channels at that location and generates the values ​​of the output channels through linear combination. Since the convolution kernel size is 1×1, it does not involve the interaction of adjacent spatial locations, so the height and width of the feature map remain unchanged, and only the channel dimension is transformed from the input dimension to the output dimension.

[0053] Through the above-described embodiments of this disclosure, the projection layer solves the obstacle of cross-layer dimensional mismatch with a very small number of parameters (each projection layer only involves the product of the number of input channels and the number of output channels plus the offset of the number of output channels), thus maintaining the integrity of the original decoder architecture.

[0054] It should be noted that in an ideal symmetric codec architecture, the intermediate features output by the encoder's downsampling stage and the decoder features output by the decoder's upsampling stage have the same spatial resolution, allowing for direct element-wise addition fusion. However, real-world architectures may have asymmetric designs: the encoder may contain additional downsampling stages, or the decoder may contain additional upsampling stages; the output of a certain encoder level may not perfectly match the output resolution of the corresponding decoder level due to specific design choices.

[0055] Therefore, verification can be performed before fusion to check whether the spatial dimensions of the target projection features and the decoder features are consistent. If they are consistent, fusion can be performed directly; if they are inconsistent, the spatial dimensions of the target projection features can be adjusted through downsampling or upsampling operations to match the decoder features.

[0056] In an optional implementation, fusing the target projection feature with the i-th decoder feature to obtain the i-th fused feature includes: when the spatial dimensions of the target projection feature and the i-th decoder feature are consistent, adding the target projection feature and the i-th decoder feature element by element to obtain the i-th fused feature, wherein the spatial dimension is used to indicate the spatial resolution of the feature.

[0057] Optionally, if the spatial dimension of the target projection feature is greater than the spatial dimension of the i-th decoder feature, the target projection feature is downsampled to obtain a downsampled projection feature, and the downsampled projection feature is added element-wise to the i-th decoder to obtain the i-th fused feature; if the spatial dimension of the target projection feature is less than the spatial dimension of the i-th decoder feature, the target projection feature is upsampled to obtain an upsampled projection feature, and the upsampled projection feature is added element-wise to the i-th decoder to obtain the i-th fused feature.

[0058] Spatial dimension indicates the size of the feature map along the spatial axis, including both height and width. Spatial dimension directly indicates the spatial resolution of the feature map, that is, the level of detail of the feature map in the original image space. High spatial resolution means the feature map has more spatial locations and can carry more detailed structural information; low spatial resolution means the feature map has fewer spatial locations and mainly carries abstract semantic information.

[0059] In an optional implementation, after obtaining the target projection features and decoder features, their spatial dimensions are first compared. Specifically, the height and width values ​​of the target projection features and the height and width values ​​of the decoder features are extracted. The two sets of values ​​are then compared to see if they are completely equal.

[0060] If the height of the target projection feature is equal to the height of the decoder feature, and the width of the target projection feature is equal to the width of the decoder feature, then the spatial dimensions are determined to be consistent. In this case, element-wise addition fusion is directly performed.

[0061] If the height of the target projection feature is greater than the height of the decoder feature, or the width of the target projection feature is greater than the width of the decoder feature, then the spatial dimension of the target projection feature is determined to be greater than the spatial dimension of the decoder feature. In this case, downsampling is performed first, followed by fusion. Downsampling can be achieved through average pooling. The target projection feature is divided into multiple non-overlapping spatial windows, the size of which is determined by the ratio of the spatial dimensions of the target projection feature to those of the decoder feature. The average value of all values ​​within each window is calculated, and this average value is used as the value at the corresponding position in the downsampled feature map. Pooling is performed independently on each channel without changing the channel dimension. After downsampling, downsampled projection features are generated, and element-wise addition is performed between the downsampled projection features and the decoder features to obtain the fused features.

[0062] If the height of the target projection feature is less than the height of the decoder feature, or the width of the target projection feature is less than the width of the decoder feature, then the spatial dimension of the target projection feature is determined to be less than the spatial dimension of the decoder feature. In this case, upsampling is performed first, followed by fusion. Upsampling can be implemented using bilinear interpolation. For each spatial location in the upsampled feature map, the weighted average of the four neighboring pixels is calculated based on its relative position in the original low-resolution feature map. The weights are determined by the distance from the location to the center of the four neighboring pixels; the closer the pixel is, the greater its weight. After upsampling, an upsampled projection feature is generated. Then, the upsampled projection feature and the decoder feature are added element-wise to obtain the fused feature.

[0063] Element-wise addition iterates through all spatial locations and channel indices of the two feature maps. For each channel at each spatial location, the value of the target projected feature at that location is added to the value of the decoder feature at that location to obtain the value of the fused feature at that location. After the operation, the fused feature has the exact same spatial and channel dimensions as the two input features.

[0064] Through the above-described embodiments of this disclosure, by using spatial dimension consistency judgment and conditional adjustment operations, this step enables cross-layer feature fusion to adapt to the asymmetric architecture design between the encoder and decoder, expanding the applicability of the technical solution. While maintaining element-wise addition as the unified fusion method, downsampling or upsampling is used to address the spatial resolution mismatch problem, ensuring that multi-scale information from the encoder can be injected into the corresponding layers of the decoder. All three cases end with additive fusion, allowing the decoder reconstruction process to obtain real information constraints from the original input, ultimately improving the structural consistency and detail fidelity of the reconstructed image.

[0065] return Figure 1 In step S104, if i is less than 4, the i-th fusion feature is input into the (i+1)-th upsampling stage.

[0066] When the current stage is not the last upsampling stage, the fused features are passed as input to the next upsampling stage, and the decoding process continues. This is the normal data flow of the decoder's step-by-step upsampling. Therefore, whether the last upsampling stage has been reached can be determined by checking if the current stage i is greater than the number of upsampling stages. That is, if the number of upsampling stages in the decoder is M, the current stage i can be compared with M.

[0067] Step S105: When i equals 4, output the reconstructed image based on the i-th fusion feature.

[0068] When the current stage is the last upsampling stage, the final reconstructed image is output based on the fused features. At this point, the decoder has completed all upsampling operations from latent variables to pixel space, and the final fused features are mapped to an image of the same dimension as the original input through output layer convolution.

[0069] Through the above methods, the multi-scale information that should have been discarded at each downsampling level of the encoder is preserved and passed to the decoder by the intermediate feature synchronous capture mechanism during the encoder's forward propagation process; the real structural information captured by the encoder is injected into the decoder reconstruction process through the cross-layer fusion mechanism of the decoder; and a multi-scale constraint chain from global structure to local edges is formed through the step-by-step transfer mechanism of fused features. Thus, the decoder obtains additional information channels beyond the highly compressed latent vectors, alleviating high-frequency loss caused by information bottlenecks and achieving a synergistic improvement in the structural consistency and detail fidelity of the reconstructed image.

[0070] Optionally, in this embodiment, the variational decoding method based on cross-layer feature compensation described above can be applied, but is not limited to, to image compression and real-time reconstruction in autonomous driving scenarios. In autonomous driving vehicle systems, multiple cameras continuously acquire high-resolution road scene images, generating a large amount of data that needs to be stored or transmitted. The above-disclosed solution can be applied to an in-vehicle image compression system: the encoder compresses the original image into compact latent variables while caching multi-scale intermediate features; the decoder uses these cached features at the receiving end to perform cross-layer compensation reconstruction, restoring images with clear high-frequency details such as road boundaries, lane markings, and obstacle outlines. Compared to standard variational autoencoder reconstruction, this solution significantly improves the fidelity of edge regions at the same compression rate, ensuring that the compressed image still meets the training and inference requirements of downstream perception models, while reducing the pressure on in-vehicle storage and communication bandwidth.

[0071] Optionally, in this embodiment, the variational decoding method based on cross-layer feature compensation described above can be applied, but is not limited to, to multi-sensor data simulation generation. In autonomous driving simulation platforms, a large amount of realistic training data is needed to supplement the deficiencies in real-vehicle data collection. The above-disclosed scheme can serve as a decoding enhancement module for a conditional diffusion generation system: the lidar depth map and reference image are respectively processed by an encoder to extract latent representations and multi-scale features; after the diffusion model performs conditional denoising in the joint latent space, the decoder uses two sets of cached features to perform cross-layer compensation, outputting a target image and a reference image with faithful geometric structure and clear texture details. This enhances the realism of the simulation data in key areas such as road edges and traffic signs, thereby improving the performance of the perception model in the transition from simulation to reality.

[0072] Optionally, in this embodiment, the variational decoding method based on cross-layer feature compensation described above can be applied, but is not limited to, image enhancement in nighttime or inclement weather. In low-visibility scenarios such as autonomous driving at night or rain / fog, the image quality captured by the camera is severely degraded, with blurred lane lines and unclear pedestrian outlines. The above-disclosed solution can be applied to an image enhancement system: the encoder performs forward propagation on the degraded image, caching intermediate features containing original edge and structural information; the decoder injects these real features into the corresponding layers through cross-layer fusion during reconstruction, compensating for information loss caused by degradation. Compared to a standard decoder, the edge loss term in this solution is specifically constrained for high-frequency regions, resulting in enhanced images with higher sharpness and contrast in safety-critical areas such as road boundaries and obstacle outlines, providing more reliable input for nighttime perception algorithms.

[0073] The above usage scenarios are merely examples; the solutions disclosed herein can be used in any scenario related to image processing. This embodiment does not impose any limitations on them.

[0074] Figure 4 This is a schematic diagram of an optional variational decoding method based on cross-layer feature compensation according to an embodiment of the present disclosure; as shown... Figure 4 As shown, the input image can be vehicles and roads next to bushes. After passing through an encoder containing four downsampling stages, a latent vector can be obtained. The latent vector can be updated by injecting text information (such as changing it to roads next to buildings) through methods such as diffusion models (diffusion model update process). Figure 4 (not shown), and then the latent vector is input into a decoder that includes four upsampling stages to obtain a reconstructed image, where roads and vehicles are modified.

[0075] Specifically, the encoder's input dimensions can be batch size B, 3 channels, height H, and width W. In the first downsampling stage, the operation type is convolution + ReLU activation + downsampling. The convolution kernel size can be 3×3, with a stride of 2. The input dimensions are B×3×H×W, and the output dimensions are B×128×H / 2×W / 2. This stage outputs the first intermediate feature (first feature map), with a spatial resolution of half the input and a channel dimension of 128. In the second downsampling stage, the operation type is residual block ×2 + downsampling. The convolution kernel size is 3×3, with a stride of 2. The input dimensions are B×128×H / 2×W / 2, and the output dimensions are B×256×H / 4×W / 4. This stage outputs the second intermediate feature (second feature map), with a spatial resolution of one-quarter of the input and a channel dimension of 256. The third downsampling stage operates on the principle of residual block ×2 + downsampling. The kernel size can be 3×3, the stride can be 2, the input dimension is B×256×H / 4×W / 4, and the output dimension is B×512×H / 8×W / 8. This stage outputs the third intermediate feature (third feature map), with a spatial resolution of 1 / 8 of the input and a channel dimension of 512. The fourth downsampling stage also operates on the same principle, with residual block ×2 + downsampling. The kernel size can be 3×3, the stride can be 2, the input dimension is B×512×H / 8×W / 8, and the output dimension is B×512×H / 16×W / 16. This stage outputs the fourth intermediate feature (fourth feature map), with a spatial resolution of 1 / 16 of the input and a channel dimension of 512. Latent layers (…) Figure 4 (Not shown in the image), Operation type: Convolution, Convolution kernel size can be 3×3, stride can be 2, Input dimension: B×512×H / 16×W / 16, Output dimension: B×8×H / 32×W / 32, This layer outputs the latent variable Z, with a spatial resolution of 1 / 32 of the input and a channel dimension of 8.

[0076] The decoder employs a four-stage hierarchical upsampling structure. The input is latent variables, and the output is the reconstructed image. Its input dimension is B×8×H / 32×W / 32 (from latent variables). In the first upsampling stage, the operation type is convolution + upsampling. The convolution kernel size can be 3×3, and the upsampling ratio can be 2. The input dimension is B×8×H / 32×W / 32, and the output dimension is B×256×H / 16×W / 16. This stage outputs the first decoder feature with a spatial resolution of H / 16×W / 16 and a channel dimension of 256. In the second upsampling stage, the operation type is residual block ×2 + upsampling. The convolution kernel size can be 3×3, and the upsampling ratio can be 2. The input dimension is B×256×H / 16×W / 16, and the output dimension is B×512×H / 8×W / 8. This stage outputs the second decoder feature with a spatial resolution of H / 8×W / 8 and a channel dimension of 512. The third upsampling stage operates on the principle of residual block ×2 + upsampling. The kernel size can be 3×3, and the upsampling factor can be 2. The input dimension is B×512×H / 8×W / 8, and the output dimension is B×512×H / 4×W / 4. This stage outputs the third decoder feature with a spatial resolution of H / 4×W / 4 and a channel dimension of 512. The fourth upsampling stage operates on the same principle, operating on the same principle. The kernel size can be 3×3, and the upsampling factor can be 2. The input dimension is B×512×H / 4×W / 4, and the output dimension is B×512×H / 2×W / 2. This stage outputs the fourth decoder feature with a spatial resolution of H / 2×W / 2 and a channel dimension of 512. Output layer ( Figure 4 (not shown in the image), Operation type: Residual block × 2 + Convolution, Convolution kernel size can be 3×3 (residual block) + 3×3 (output convolution), stride can be 1, Input dimension: B×512×H / 2×W / 2, Output dimension: B×3×H×W, This layer outputs the final reconstructed image, with 3 channels, and the spatial resolution is restored to the original input size.

[0077] There are four projection layers, each an independent 1×1 convolutional layer, responsible for mapping the channel dimension of the corresponding encoder intermediate features to the channel dimension of the corresponding decoder features. The first projection layer has 128 input channels, 512 output channels, a 1×1 kernel, bias, and 128×512+512=66048 parameters. It corresponds to the first downsampling stage and the fourth upsampling stage, expanding the 128 channels of shallow edge features to 512 channels and mapping them to the high-dimensional decoder space. The second projection layer has 256 input channels, 512 output channels, a 1×1 kernel, bias, and 256×512+512=131584 parameters. It corresponds to the second downsampling stage and the third upsampling stage, expanding the 256 channels of the mid-level semantic features to 512 channels. The third projection layer has 512 input channels, 512 output channels, a 1×1 kernel size, and a bias (yes). It has 512×512+512=262656 parameters. This corresponds to the third downsampling stage and the second upsampling stage, reprojecting the 512 channels of deep structural features to 512 channels, performing semantic space alignment rather than dimensionality transformation. The fourth projection layer has 512 input channels, 256 output channels, a 1×1 kernel size, and a bias (yes). It has 512×256+256=131328 parameters. This corresponds to the fourth downsampling stage and the first upsampling stage, shrinking the 512 channels of core latent features to 256 channels to match the deep channel configuration of the decoder.

[0078] For road images, the intermediate features of the encoder can include features unique to the driving scene. Specifically, in an optional implementation, when the input image is a road image, the four intermediate features include at least one of the following: a first feature map, used to carry road edge response, lane marking texture, road surface material contrast, and obstacle contour gradient information; a second feature map, used to carry traffic sign components, vehicle area contours, and intermediate texture information of road markings; a third feature map, used to carry vehicle-level semantics, road scene layout, and lane semantic consistency information; and a fourth feature map, used to carry driving scene-level semantics, road core skeleton, and global contextual information of the traffic environment.

[0079] Image frequencies can be categorized into high-frequency, mid-frequency, and low-frequency features. The high and low frequencies of an image can be determined by the gradient of pixel changes over a short distance.

[0080] Specifically, the first feature map (shallow edge features) can be captured after the first downsampling stage of the encoder, with a spatial resolution of 1 / 2 of the input image and a channel dimension of 128. At this point, the feature map still maintains a relatively high spatial resolution, with each feature point corresponding to a small local region in the original image, primarily encoding high-frequency and mid-frequency information.

[0081] Among them, road edge response refers to the activation response at significant grayscale or color transition edges in the image, such as road boundaries, shoulder contours, and lane line edges; lane marking texture refers to the local texture patterns of road markings, including the local structural features of markings such as dashed lines, solid lines, and arrows; road material contrast refers to the color and reflectivity differences between different road materials such as asphalt, cement, and gravel; and obstacle contour gradient refers to the boundary gradient direction information between obstacles such as pedestrians, vehicles, and cones and the background. At the decoder's upsampling level, the edge information provided by these features guides the decoder to generate sharper boundary transitions in road boundary and obstacle contour regions, avoiding blurring during the upsampling process and supplementing texture details to prevent smoothing.

[0082] The second feature map (intermediate semantic features) is captured after the second downsampling stage of the encoder, with a spatial resolution of 1 / 4 of the input image and a channel dimension expanded to 256. At this point, the feature map undergoes additional convolution and nonlinear transformation, expanding the receptive field. Each feature point corresponds to the aggregation of information from an 8×8 to 12×12 pixel region in the original image, primarily encoding intermediate frequency information.

[0083] Among them, traffic sign components are local component representations of traffic signs, such as sign borders, internal patterns, and text strokes, etc.; vehicle region contours are component-level boundaries such as the outer contour of the entire vehicle, the boundary between the window and the body, and the front and rear of the vehicle; road marking intermediate textures are more abstract marking patterns, such as the overall structure of zebra crossings, the shape of directional arrows, and the combination of strokes in text markings; at the decoder upsampling level, this feature provides component-level structural constraints to ensure that the reconstructed object has a reasonable component composition, supplements region contour information to prevent distortion and deformation of the overall shape of the object, and conveys the intermediate texture semantics of different material regions.

[0084] The third feature map (deep structural features) is captured after the encoder's third downsampling stage, with a spatial resolution of 1 / 8 of the input image and a channel dimension further expanded to 512. At this point, the receptive field of the feature map has covered a large area of ​​the input image, enabling it to integrate information from a wide spatial region, and it mainly encodes low-frequency information.

[0085] Among them, vehicle-level semantics is an abstract representation of a complete vehicle, including high-level encoding of categories such as cars, trucks, and motorcycles, as well as attributes such as vehicle orientation and posture; road scene layout is the layout relationship and spatial distribution of large areas such as roads, sky, buildings, and green belts; lane semantic consistency is to constrain the appearance consistency of lane lines of the same type in different locations, such as keeping the width, color, and solid / phasing patterns of the entire lane line uniform. At the upsampling level of the decoder, this feature provides global structural constraints to ensure that the reconstructed scene has a reasonable spatial layout and perspective relationship, constrains the consistency of object-level semantics, prevents the same object from having appearance fragmentation during the reconstruction process, and guides the smooth and semantically consistent reconstruction of large areas.

[0086] The fourth feature map (core latent features) is captured after the fourth downsampling stage of the encoder, located in the last high-dimensional feature level before latent variable generation. The spatial resolution is 1 / 16 of the input image, and the channel dimension remains 512. At this point, the receptive field of each feature point nearly covers most of the entire image, primarily encoding low-frequency and global information.

[0087] Among them, driving scene-level semantics is an abstract generalization of the entire driving scene, such as high-level encoding of categories like urban road scenes, highway scenes, intersection scenes, and ramp scenes; the core road skeleton is the most critical structural element in the scene, such as the main road orientation, changes in the number of lanes, intersection topology, and distribution of major traffic participants; the global traffic environment context is the global contextual information of the entire image, including lighting conditions (day / dusk / night), weather conditions (sunny / rainy / foggy), time period, traffic flow, and other environmental information. At the upsampling level of the decoder (usually located in the deep layers of the decoder), this feature provides core structural skeleton constraints, ensuring that the reconstructed scene does not deviate from the core composition and road orientation of the original scene, and conveys global contextual information to guide the accurate reconstruction of overall lighting, tone, and weather atmosphere, compensating for key structural information that may be lost during the compression process of latent vectors.

[0088] The first feature map captures the finest edge and texture details at the highest resolution level, directly compensating for high-frequency loss during the final upsampling stage of the decoder. The second feature map captures part and region structures at a medium resolution level, constraining the rationality of object composition. The third feature map captures object and scene layout at a lower resolution level, ensuring the consistency of the global structure. The fourth feature map captures scene-level semantics and environmental context at the lowest resolution level, ensuring that the reconstruction result does not deviate from the core semantics of the original scene.

[0089] The training process of the variational decoding method based on cross-layer feature compensation disclosed herein is described below: In an optional implementation, before inputting the input image into the encoder to perform forward propagation, the following steps are included: acquiring a pre-trained encoder and decoder, and freezing the parameter weights of the encoder; initializing four projection layers, and performing multiple rounds of training on the four projection layers and the decoder; in the p-th round of training, based on the p-th training sample in the training sample set, the following operations are performed: inputting the p-th training sample into the encoder to perform forward propagation, obtaining four reference intermediate features and a p-th reference latent variable, where p is an integer greater than 0; inputting the p-th reference latent variable and the four reference intermediate features into the decoder, and after four upsampling stages, obtaining the p-th reference image; obtaining a first loss based on the p-th reference latent variable, and determining a second loss and a third loss based on the p-th reference image and the p-th training sample. The training process includes a first loss indicating the structured features of the latent space, determined based on the difference between the variational posterior distribution and the standard Gaussian prior distribution corresponding to the reference latent variable; a second loss indicating the pixel-level error between the training sample and the reference image, determined by pixel-by-pixel comparison of the reference image and the training sample; a third loss indicating the perceptual similarity between the training sample and the reference image, determined based on the feature differences between the reference image and the training sample in the perceptual network; and a fourth loss indicating the edge error between the training sample and the reference image, determined based on the mean square error of the edge response maps corresponding to the reference image and the training sample. If the training termination condition is not met, the parameters of the four projection layers and the decoder are updated based on the first, second, third, and fourth losses. If the training termination condition is met, the training ends.

[0090] It should be noted that training is divided into two sub-stages: pre-training and fine-tuning. The pre-training stage is performed on a standard variational autoencoder to obtain the initial weights of the encoder and decoder. The fine-tuning stage freezes the encoder weights, initializes the projection layer, and jointly trains the parameters of the projection layer and decoder.

[0091] The pre-training phase can load the encoder and decoder, and use the training set to train the encoder and decoder to obtain the pre-trained encoder and decoder.

[0092] During the fine-tuning phase, the encoder's parameter weights are frozen, while the decoder's parameters are partially frozen or fully incorporated into training, depending on requirements. All projection layers are initialized, and then training samples are input into the encoder. The intermediate features are then input into the decoder using the projection layers to obtain the reconstructed reference image. The loss is calculated based on the reference image and training samples, and the parameters are updated according to the loss.

[0093] In an optional implementation, initializing the projection layer includes: determining the input channel dimension of the input channel corresponding to the projection layer; determining the standard deviation by the reciprocal of the square root of the input channel dimension, and initializing the weights of the projection layer based on the Gaussian distribution determined by the standard deviation; and initializing the bias term of the projection layer to a preset bias value.

[0094] As a key component of cross-layer feature compensation, the initial state of the projection layer directly affects subsequent training results. If the weights are initialized too large, it may lead to feature value explosion during forward propagation and gradient vanishing or exploding during backpropagation; if the weights are initialized too small, the signal may attenuate layer by layer in the deep network, failing to propagate effectively. Therefore, initializing the projection layer with a Gaussian distribution whose standard deviation is determined based on the input channel dimension ensures that the output of the projection layer maintains an appropriate numerical range in the early stages of training, avoiding training failure due to improper initial values. Initializing the bias to zero allows the projection layer to rely solely on linear transformations of the weights in the early stages of training, facilitating the gradient descent algorithm to quickly find effective solutions.

[0095] In an optional implementation, the network configuration of the projection layer is analyzed to extract its input channel dimension parameter. This parameter is determined by the channel dimension of the corresponding encoder intermediate feature. For example, if the projection layer is responsible for processing the first intermediate feature, the input channel dimension is 128; if it processes the second intermediate feature, it is 256; if it processes the third intermediate feature, it is 512; and if it processes the fourth intermediate feature, it is 512.

[0096] The standard deviation is calculated based on the input channel dimension. The specific operation is as follows: first, calculate the square root of the input channel dimension; then, take the reciprocal of this square root; finally, multiply by the square root of 2. Taking an input channel dimension of 128 as an example: first, calculate the square root of 128, which is approximately 11.31; then, take the reciprocal, which is approximately 0.0884; finally, multiply by the square root of 2, which is approximately 1.414, resulting in a standard deviation of approximately 0.125. Based on the calculated standard deviation and zero mean, a Gaussian distribution is constructed. Random sampling is then performed from this distribution to generate the weight tensor of the projection layer. Specifically, when the input channel dimension is... In the case of weight It can be determined using the following formula:

[0097] For projection layers with the same input and output channels, a residual initialization strategy can be optionally employed. First, weights with a standard Gaussian distribution are generated, then an identity matrix and a small random perturbation are superimposed to make the initial weights approximate an identity mapping. This ensures that the projection operation is approximately transparent in the early stages of training, reducing training instability caused by the introduction of new modules.

[0098] Create a bias vector with the same dimensions as the output channels of the projection layer, and initialize all elements to the preset value of zero. Then, by calling the tensor filling function, assign zero to all positions in the bias tensor. This initialization ensures that the output of the projection layer in the early stages of training is determined solely by a linear combination of the input features and weights, unaffected by additional bias offsets.

[0099] By initializing the projection layer weights with a Gaussian distribution whose standard deviation is determined based on the input channel dimension, the weights have an appropriate numerical scale in the early stages of training. This avoids feature values ​​being too large or too small during forward propagation, maintaining a stable distribution of network activation values. Zeroing the bias simplifies the optimization process in the early stages of training, making it easier for gradient descent to find effective update directions.

[0100] The specific training process has been described above. The specific loss function will be explained below: The first loss (KL divergence loss) is determined based on the p-th reference latent variable and is used to indicate the structured characteristics of the latent space. Constraining the latent space to follow a standard Gaussian distribution ensures the latent variables have structured properties. This makes the latent space continuous and interpolable, supporting the generation of new latent vectors from the standard Gaussian distribution and guaranteeing the probabilistic generation capability of the encoder and decoder. The first loss prevents latent space collapse, i.e., avoiding the encoder outputting a constant latent vector that loses its expressive power. The weighting coefficients control its influence strength, preventing it from dominating the total loss and balancing reconstruction quality with latent space structure.

[0101] The KL divergence is calculated using the latent variable distribution parameters output by the encoder. The encoder outputs the mean and log-variance parameters of the latent variables, and a variational posterior distribution is constructed based on these two parameters. The KL divergence between this variational posterior distribution and the standard Gaussian prior distribution is calculated. For each dimension of the latent space, the difference between the variational posterior distribution and the standard Gaussian distribution for that dimension is calculated. The difference consists of three parts: the logarithm of the variational posterior variance, the squared variational posterior mean, and the variational posterior variance. These three parts are combined, and the negative half is taken. This result is then summed over all dimensions of the latent space to obtain the KL divergence loss. The smaller this loss value, the closer the latent distribution is to the standard Gaussian distribution, and the better the structured characteristics of the latent space.

[0102] The second loss (reconstruction loss) is determined based on the p-th reference image and the p-th training sample, and is used to indicate the pixel-level error between the training sample and the reference image. The reference image output by the decoder is compared pixel-by-pixel with the original training sample, and the mean squared error is calculated. The differences in color channel values ​​at each pixel location are squared and then averaged to obtain the overall pixel-level reconstruction error. As a major component of the total loss, it drives the decoder and projection layer to learn basic image reconstruction capabilities. Pixel-level constraints enable the network to focus on global color distribution and overall structure recovery, providing a basic reconstruction framework for other losses.

[0103] The reconstructed image output by the decoder is compared pixel-by-pixel with the original input image. For each pixel location in the image, the square of the difference between the pixel value in the reconstructed image and the pixel value in the original image is calculated. The sum of the squared differences for all pixel locations is then divided by the total number of pixels to obtain the mean squared error. For each color channel at each spatial location in both the input and reconstructed images, the numerical difference between the two images at that location is calculated, and the difference is squared. After traversing all spatial locations and all channels, the squared differences are accumulated and then divided by the total number of pixels (including the product of batch size, height, width, and channels) to obtain the reconstruction loss value. The smaller this loss value, the closer the reconstructed image is to the original image at the pixel level.

[0104] The third loss (perceptual loss) is determined based on the p-th reference image and the p-th training sample, and is used to indicate the perceptual similarity between the training sample and the reference image. The original input image and the reconstructed image output from the decoder are fed into a pre-trained visual perception network (such as VGG-16), and feature maps from the intermediate layers of the network are extracted. For each layer's feature map, the difference between the features of the original image and the reconstructed image at corresponding spatial locations is calculated, and the squared differences are summed and averaged. The weighted sum of the feature differences from all layers yields the perceptual loss. This loss guides the network to focus on the semantic structure and texture patterns of the image, rather than simply pursuing precise pixel value matching. This makes the reconstruction results more consistent with human visual habits, improves subjective visual quality, and compensates for the shortcomings of pixel-level losses in texture and detail evaluation.

[0105] A pre-trained VGG-16 network, with its multi-layered features, captures multi-scale visual information ranging from low-level texture to high-level semantics. The original and reconstructed images are input into this network, and the output features of multiple intermediate layers (such as convolutional layers 3, 5, and 7) are selected. For each selected layer, the feature maps of the two images are compared point-by-point in space, the Euclidean distance is calculated, and then the spatial average is taken. Finally, the distances of each layer are summed according to preset weights to obtain the perceptual loss. The smaller this loss value, the more visually similar the two images are.

[0106] The fourth loss (edge ​​loss) is determined based on the p-th reference image and the p-th training sample, and is used to indicate the edge error between the training sample and the reference image. The training sample and the reference image are processed by the edge detection operator to extract edge response maps. A weighted mean square error is calculated in the edge region, with edge pixels having a higher weight than non-edge pixels, strengthening the constraint on edge reconstruction accuracy. It is specifically optimized for reconstruction of high-frequency detail regions in autonomous driving scenarios. Edge structures such as road boundaries, lane markings, and obstacle outlines are safety-critical information; edge loss ensures that these regions remain clear and sharp after reconstruction, avoiding edge degradation caused by upsampling smoothing. As a scenario-specific loss term, it directly addresses the stringent requirements of autonomous driving for high-frequency details. Unlike general image reconstruction, edge loss enables the network to actively focus on edge fidelity during optimization, improving the practicality of the reconstructed image in downstream perception tasks.

[0107] The gradient changes in the image are detected in both the horizontal and vertical directions. The original and reconstructed images are convolved with two separate kernels to obtain horizontal and vertical gradient maps, respectively. The squares of the two gradient maps are then summed and the square root is taken to obtain a combined gradient magnitude map, i.e., an edge response map. A threshold is set based on the gradient magnitude of the original image; regions with magnitudes greater than the threshold are identified as edge regions, and a binary mask is generated. Within these edge regions, the squared difference between the edge responses of the original and reconstructed images is calculated, and then averaged over the edge pixel count to obtain the edge loss. The smaller this loss value, the more consistent the reconstructed image is with the original image in the edge regions.

[0108] The total loss is formed by weighted summation, and the weight coefficients of each loss term are determined through iterative validation set performance. Reconstruction loss, as the foundational term, has the highest weight; perception loss and edge loss, as enhancement terms, have the next highest weight; and KL divergence loss, as the constraint term, has the lowest weight. For example, the weight of the first loss can be 10. -4 The weight of the second loss can be 1, the weight of the third loss can be 0.5, and the weight of the fourth loss can be 0.2.

[0109] During backpropagation, gradients flow through four loss paths simultaneously, eventually converging at the projection layer and decoder parameters. The projection layer learns how to map intermediate features from the encoder into compensation information most beneficial for reconstruction, while the decoder learns how to integrate latent variables and compensation features to generate high-quality output.

[0110] The following describes the optional training termination conditions: Condition 1: The number of training epochs reaches a preset value, for example, 50 epochs for the fine-tuning phase. Each training epoch traverses the entire training sample set, completing forward propagation, loss calculation, and parameter updates for all samples. After reaching this number of epochs, training terminates regardless of whether the loss continues to decrease.

[0111] Condition 2: The total loss is the weighted sum of four losses, including reconstruction loss, KL divergence loss, perception loss, and edge loss. Training is considered converged and terminated when the decrease in total loss over multiple training rounds is less than a preset percentage, or when the total loss value stabilizes within a preset threshold range.

[0112] Condition 3: Periodically evaluate the reconstruction quality on the validation set, and terminate training when the evaluation metrics reach the preset standards. Evaluation metrics include Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM), and Edge-SSIM. For example, if Edge-SSIM exceeds 0.73 for multiple consecutive rounds, the model is deemed to meet the edge fidelity requirements for autonomous driving scenarios.

[0113] The peak signal-to-noise ratio is defined as:

[0114] Where MAXI = 255 (8-bit image), and MSE is the mean square error between the original image and the reconstructed image.

[0115] The edge region structural similarity index is a metric specifically designed to evaluate the structural similarity of edge regions. It calculates the structural similarity index value only at the detected edge pixel locations by performing edge detection on the image.

[0116] in For the set of edge pixels, This is the edge detection threshold.

[0117] Condition 4: A cosine annealing learning rate scheduling strategy is adopted, in which the learning rate gradually decays with the number of training epochs. When the learning rate decays to a preset lower limit, such as approaching zero or reaching the minimum effective learning rate, it is determined that there is limited room for further optimization, and training is terminated.

[0118] It should be noted that the above conditions can be used individually or in combination. In actual training, the number of training rounds is usually used as the primary termination condition, while monitoring the validation set evaluation metrics serves as the basis for early termination.

[0119] As mentioned earlier, the encoder does not necessarily select all downsampling stage outputs as intermediate features. Therefore, ablation experiments can be used to determine which stages, as intermediate features, contribute significantly to the reconstruction quality. The following example uses a four-layer encoder. Based on the complete model (all four layers enabled), the compensation mechanism of each individual layer is removed sequentially, and the performance degradation is observed. Simultaneously, the performance is tested when only single-scale compensation is retained, and the independent contribution of each layer is observed. Table 1 shows the impact of each layer on the reconstruction quality.

[0120] Table 1. Impact of each feature on reconstruction quality

[0121] As shown in Table 1, removal at any single scale leads to performance degradation, verifying the necessity of multi-scale feature transfer. Even the deepest removal results in a 0.5 dB decrease in peak signal-to-noise ratio (PSNR) and a 1.9% decrease in the edge region structural similarity index, indicating that each level is not redundant. Removing F1 reduces the edge region structural similarity index by 7.0%, significantly higher than other levels. This aligns with the characteristic of F1 being rich in high-frequency edge information, validating the crucial role of shallow feature compensation in edge reconstruction. The highest PSNR for single-scale compensation is 29.8 dB (F1 only), far lower than the 31.4 dB of the complete model. This demonstrates that multi-scale collaborative compensation can generate superimposed gains beyond a single scale, with information from each level complementing rather than substituting. While F3 and F4 contribute little directly to the edge region structural similarity index (decreasing by 2.5% and 1.9% respectively after removal), they play a crucial role in the overall PSNR and structural consistency. Removing them reduces the PSNR by 0.6 dB and 0.5 dB respectively, ensuring the global rationality of the reconstructed scene.

[0122] The above embodiments can be combined in any way to form other optional embodiments of this disclosure, which will not be described in detail here.

[0123] According to another aspect of the embodiments of this disclosure, a variational decoding apparatus based on cross-layer feature compensation is also provided for implementing the above-described variational decoding method based on cross-layer feature compensation. For example... Figure 5As shown, the device includes: an encoding module 501, used to input the input image into an encoder to perform forward propagation, obtaining latent variables and four intermediate features, wherein the encoder includes four downsampling stages, and the intermediate features are feature maps output by the downsampling stages; a decoding module 502, used to input the latent variables into a decoder symmetrical to the encoder structure, wherein the decoder includes four upsampling stages, and the i-th upsampling stage outputs the i-th decoder feature, where i is an integer greater than 0 and less than or equal to 4; a feature fusion module 503, used to fuse the target intermediate feature with the i-th decoder feature in the i-th upsampling stage of the decoder when there is a target intermediate feature among the four intermediate features corresponding to the i-th upsampling stage, to obtain the i-th fused feature; a first judgment module 504, used to input the i-th fused feature into the (i+1)-th upsampling stage when i is less than 4; and a second judgment module 505, used to output a reconstructed image based on the i-th fused feature when i is equal to 4.

[0124] Optionally, the feature fusion module 503 is further configured to: determine the target projection layer corresponding to the i-th upsampling stage among the four projection layers, wherein the projection layer is used to perform point convolution projection on the intermediate features; project the channel dimension of the target intermediate features to be consistent with the channel dimension of the i-th decoder feature through the target projection layer to obtain the target projection feature; and fuse the target projection feature with the i-th decoder feature to obtain the i-th fused feature.

[0125] Optionally, the feature fusion module 503 is further configured to: add the target projection feature and the i-th decoder feature element by element to obtain the i-th fused feature when the spatial dimension of the target projection feature and the i-th decoder feature are consistent, wherein the spatial dimension is used to indicate the spatial resolution of the feature.

[0126] Optionally, the encoding module 501 is further configured to: return intermediate features in each of the four downsampling stages of the encoder, add the four intermediate features to a buffer list, wherein the forward propagation function of the four downsampling stages includes a feature buffer function; and output the four intermediate features to the buffer list after passing the forward hook function of the four downsampling stages.

[0127] Optionally, when the input image is a road image, the four intermediate features include at least one of the following: a first feature map, used to carry road edge response, lane marking texture, road surface material contrast, and obstacle contour gradient information; a second feature map, used to carry traffic sign components, vehicle area contours, and intermediate texture information of road markings; a third feature map, used to carry vehicle-level semantics, road scene layout, and lane semantic consistency information; and a fourth feature map, used to carry driving scene-level semantics, road core skeleton, and global contextual information of the traffic environment.

[0128] Optionally, the variational decoding device based on cross-layer feature compensation further includes: a training module for acquiring pre-trained encoders and decoders and freezing the parameter weights of the encoder; initializing four projection layers and performing multiple rounds of training on the four projection layers and the decoder; in the p-th round of training, based on the p-th training sample in the training sample set, performing the following operations: inputting the p-th training sample into the encoder to perform forward propagation, obtaining four reference intermediate features and the p-th reference latent variable, where p is an integer greater than 0; inputting the p-th reference latent variable and the four reference intermediate features into the decoder, and obtaining the p-th reference image after four upsampling stages; obtaining a first loss based on the p-th reference latent variable, and determining a second loss, a third loss, and a fourth loss based on the p-th reference image and the p-th training sample, wherein the first loss is used to indicate The structured features of the latent space are defined as follows: the first loss is determined based on the difference between the variational posterior distribution and the standard Gaussian prior distribution corresponding to the reference latent variable; the second loss is used to indicate the pixel-level error between the training sample and the reference image, determined by pixel-by-pixel comparison of the reference image and the training sample; the third loss is used to indicate the perceptual similarity between the training sample and the reference image, determined based on the feature differences between the reference image and the training sample in the perceptual network; and the fourth loss is used to indicate the edge error between the training sample and the reference image, determined based on the mean square error of the edge response maps corresponding to the reference image and the training sample. If the training termination condition is not met, the parameters of the four projection layers and the decoder are updated based on the first, second, third, and fourth losses; if the training termination condition is met, the training ends.

[0129] Optionally, the training module is also used to: determine the input channel dimension of the input channel corresponding to the projection layer; determine the standard deviation by the reciprocal of the square root of the input channel dimension, and initialize the weights of the projection layer based on the Gaussian distribution determined by the standard deviation; and initialize the bias term of the projection layer to a preset bias value.

[0130] Optionally, in this embodiment, the implementation of each of the above-mentioned unit modules can be referred to the above-mentioned method embodiments, which will not be repeated here.

[0131] According to another aspect of the embodiments of this disclosure, an electronic device for implementing the above-described variational decoding method based on cross-layer feature compensation is also provided, see [link to relevant documentation]. Figure 6 , Figure 6 This is a structural block diagram of an electronic device provided in an embodiment of this disclosure. Figure 6As shown, the electronic device 600 may include a processor 601, a memory 602, and a program or instructions stored in the memory 602 and executable on the processor 601. When executed by the processor 601, the program or instructions implement the various processes of the above-described variational decoding method embodiment based on cross-layer feature compensation, and achieve the same technical effect. To avoid repetition, further details are omitted here. It should be noted that the electronic devices in this disclosure include mobile electronic devices and non-mobile electronic devices.

[0132] According to one aspect of this disclosure, a computer program product is provided, comprising a computer program / instructions containing program code for performing the methods shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via a communication component, and / or installed from a removable medium. When executed by a central processing unit, the computer program performs various functions provided by embodiments of this disclosure.

[0133] According to one aspect of this disclosure, a computer-readable storage medium is provided that stores a computer program that, when executed by a processor, implements the various processes of the above-described variational decoding method embodiment based on cross-layer feature compensation, and achieves the same technical effect. To avoid repetition, further details are omitted here.

[0134] The aforementioned readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0135] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device and system embodiments are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0136] While one or more embodiments of this specification provide the operational steps of the methods described in the embodiments or flowcharts, more or fewer operational steps may be included based on conventional or non-inventive labor. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only possible execution order. In actual device or client product execution, the methods shown in the embodiments or drawings can be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment).

[0137] In the description of this disclosure, it should be noted that the terms "upper", "lower", "inner", "outer", "front", "back", "left", "right", etc., indicate the orientation or positional relationship based on the orientation or positional relationship in the working state of this disclosure, and are only for the convenience of describing this disclosure and simplifying the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this disclosure.

[0138] In the description of this disclosure, it should be noted that, unless otherwise expressly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. Those skilled in the art can understand the specific meaning of these terms in this disclosure based on the specific circumstances.

[0139] The present disclosure has been described above with reference to preferred embodiments; however, these embodiments are merely exemplary and illustrative. Various substitutions and modifications can be made to the present disclosure based on these embodiments, all of which fall within the protection scope of the present disclosure.

Claims

1. A variational decoding method based on cross-layer feature compensation, characterized in that, include: The input image is fed into the encoder to perform forward propagation, resulting in latent variables and four intermediate features. The encoder includes four downsampling stages, and the intermediate features are feature maps output by the downsampling stages. The latent variables are input into a decoder that is symmetrical to the encoder structure, wherein the decoder includes 4 upsampling stages, and the i-th upsampling stage outputs the i-th decoder feature, where i is an integer greater than 0 and less than or equal to 4; In the i-th upsampling stage of the decoder, if there is a target intermediate feature among the four intermediate features that corresponds to the i-th upsampling stage, the target intermediate feature is fused with the i-th decoder feature to obtain the i-th fused feature; When i is less than 4, the i-th fusion feature is input into the (i+1)-th upsampling stage; When i equals 4, the reconstructed image is output based on the i-th fusion feature.

2. The method according to claim 1, characterized in that, The step of fusing the target intermediate feature with the i-th decoder feature to obtain the i-th fused feature includes: Among the four projection layers, a target projection layer corresponding to the i-th upsampling stage is determined, wherein the projection layer is used to perform point convolution projection on the intermediate features; The target projection layer projects the channel dimension of the target intermediate feature to be consistent with the channel dimension of the i-th decoder feature, thereby obtaining the target projection feature; The target projection feature is fused with the i-th decoder feature to obtain the i-th fused feature.

3. The method according to claim 2, characterized in that, The step of fusing the target projection feature with the i-th decoder feature to obtain the i-th fused feature includes: When the spatial dimension of the target projection feature is consistent with that of the i-th decoder feature, the target projection feature and the i-th decoder feature are added element by element to obtain the i-th fused feature, wherein the spatial dimension is used to indicate the spatial resolution of the feature.

4. The method according to claim 2, characterized in that, The input image is fed into the encoder for forward propagation, resulting in four intermediate features. Including one of the following: Intermediate features are returned in each of the four downsampling stages of the encoder, and the four intermediate features are added to a cache list. The forward propagation function of the four downsampling stages includes a feature cache function. After four downsampling stages, the forward hook function outputs the four intermediate features to a cache list.

5. The method according to claim 4, characterized in that, When the input image is a road image, the four intermediate features include at least one of the following: a first feature map, used to carry road edge response, lane marking texture, road surface material contrast, and obstacle contour gradient information; a second feature map, used to carry traffic sign components, vehicle area contours, and intermediate texture information of road markings; a third feature map, used to carry vehicle-level semantics, road scene layout, and lane semantic consistency information; and a fourth feature map, used to carry driving scene-level semantics, road core skeleton, and global contextual information of the traffic environment.

6. The method according to claim 2, characterized in that, Before the input image is input into the encoder to perform forward propagation, the following is included: Obtain the pre-trained encoder and decoder, and freeze the parameter weights of the encoder; Initialize the four projection layers and perform multiple rounds of training on the four projection layers and the decoder; In the p-th training round, based on the p-th training sample in the training sample set, perform the following operations: The p-th training sample is input into the encoder to perform forward propagation, resulting in 4 reference intermediate features and the p-th reference latent variable, where p is an integer greater than 0; The p-th reference latent variable and the four reference intermediate features are input into the decoder, and after four upsampling stages, the p-th reference image is obtained. A first loss is obtained based on the p-th reference latent variable. A second, third, and fourth loss are determined based on the p-th reference image and the p-th training sample. The first loss is used to indicate the structured features of the latent space and is determined based on the difference between the variational posterior distribution and the standard Gaussian prior distribution corresponding to the reference latent variable. The second loss is used to indicate the pixel-level error between the training sample and the reference image and is determined by comparing the reference image and the training sample pixel by pixel. The third loss is used to indicate the perceptual similarity between the training sample and the reference image and is determined based on the feature differences between the reference image and the training sample in the perceptual network. The fourth loss is used to indicate the edge error between the training sample and the reference image and is determined based on the mean square error of the edge response maps corresponding to the reference image and the training sample, respectively. If the training termination condition is not met, update the parameters of the four projection layers and the decoder based on the first loss, the second loss, the third loss, and the fourth loss. Training ends when the aforementioned training termination conditions are met.

7. The method according to claim 6, characterized in that, Initializing the projection layer includes: Determine the input channel dimension of the input channel corresponding to the projection layer; The standard deviation is determined by the reciprocal of the square root of the input channel dimension, and the weights of the projection layer are initialized based on the Gaussian distribution determined by the standard deviation. The bias term of the projection layer is initialized to a preset bias value.

8. A variational decoding device based on cross-layer feature compensation, characterized in that, include: The encoding module is used to input the input image into the encoder to perform forward propagation, and obtain latent variables and four intermediate features. The encoder includes four downsampling stages, and the intermediate features are feature maps output by the downsampling stages. A decoding module is used to input the latent variables into a decoder that is symmetrical to the encoder structure. The decoder includes four upsampling stages, and the i-th upsampling stage outputs the i-th decoder feature, where i is an integer greater than 0 and less than or equal to 4. The feature fusion module is used to fuse the target intermediate feature with the i-th decoder feature when there is a target intermediate feature among the four intermediate features that corresponds to the i-th upsampling stage in the i-th upsampling stage, to obtain the i-th fused feature; The first judgment module is used to input the i-th fusion feature into the (i+1)-th upsampling stage when i is less than 4; The second judgment module is used to output the reconstructed image based on the i-th fusion feature when i equals 4.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method of any one of claims 1 to 7.

10. A computing device, characterized in that, include: Memory, used to store computer programs; A processor is configured to execute a computer program stored in the memory, wherein, when the computer program is executed, it implements the method of any one of claims 1 to 7.

Images