Neural implicit expression intelligent network connection vehicle dynamic scene super-resolution reconstruction method

Abstract

The invention discloses a neural implicit expression intelligent network connection vehicle dynamic scene super-resolution reconstruction method, which comprises the steps of forming low-resolution dynamic scene data according to point cloud data, RGB-D images and speed / distance information of a vehicle and surrounding vehicles thereof; inputting the low-resolution dynamic scene data into the cross-modal attention fusion network model, and outputting a multi-view and multi-moment space-time alignment feature data set; inputting the time-space alignment feature data set into a neural implicit scene model, and outputting implicit scene features; inputting the implicit scene features into a space-time attention enhancement model, and outputting enhanced implicit scene features; and performing super-resolution reconstruction on the dynamic scene according to the enhanced implicit scene features, a binocular parallax consistency algorithm and a center vision enhancement algorithm. According to the method, the problem of detail loss caused by traditional pixel interpolation or point cloud downsampling is solved, the space-time consistency of a dynamic scene is enhanced, and reliable environment scene information is provided for an intelligent network connection vehicle.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent connected vehicles, and specifically to a method and device for super-resolution reconstruction of dynamic scenes of intelligent connected vehicles using neural implicit expression, as well as computing equipment. Background Art

[0002] The rapid development of autonomous driving and intelligent connected vehicle technologies is placing higher demands on vehicle perception systems. Accurate and real-time understanding of the vehicle's surroundings is crucial for autonomous driving decision-making, path planning, and safety assurance.

[0003] Currently, the perception systems of intelligent connected vehicles typically rely on data from multiple sensors, such as lidar (point cloud data), cameras (RGB images), depth sensors (depth maps), and on-board millimeter-wave radar. These sensors work together to provide multimodal information about the vehicle's surroundings. However, lidar's point density is often affected by scanning frequency, distance, and object reflectivity, resulting in sparse data at long distances or in areas with complex textures. Camera images are prone to loss of detail in low-light conditions, rapid motion, or in the presence of distant objects. RGB-D images are limited by the noise and resolution of the sensors themselves. Data from different sensors has different characteristics and noise patterns. At high speeds or in complex dynamic scenes, the rapid motion of the vehicle itself and surrounding objects, such as vehicles and pedestrians, can cause motion blur in the images. Point cloud data can also be distorted or drift due to the mismatch between the sensor scanning speed and the object's motion. Existing methods often struggle to maintain good temporal consistency when processing dynamic objects, resulting in a lack of temporal coherence in the reconstructed scene and jumps in details between frames. Furthermore, traditional super-resolution reconstruction methods focus on enhancing pixel-level detail, lacking a deep understanding and utilization of the semantic information of dynamic objects in the scene (such as vehicle type, direction of travel, and intent). This results in improved geometry or texture, but semantically mismatches with the real world, making it difficult to support high-level autonomous driving decisions. Furthermore, most traditional methods output explicit pixel grids or point clouds, which consume significant resources when storing and transmitting high-resolution data. They also struggle to naturally represent the continuous geometry and appearance of a scene, and are prone to aliasing or discontinuities when viewing angles change or when details are magnified.

[0004] To solve the above problems, the present invention proposes a super-resolution reconstruction method for dynamic scenes of intelligent connected vehicles based on neural implicit expression, which solves the problem of detail loss caused by traditional pixel interpolation or point cloud downsampling, enhances the spatiotemporal consistency of dynamic scenes, and provides reliable environmental scene information for intelligent connected vehicles. Summary of the Invention

[0005] In view of the above problems, the present application provides a neural implicit representation intelligent networked vehicle dynamic scene super-resolution reconstruction method and device and computing equipment.

[0006] According to one aspect of the present application, a neural implicit representation intelligent networked vehicle dynamic scene super-resolution reconstruction method is provided, comprising: forming low-resolution dynamic scene data according to point cloud data, RGB-D images and speed / distance information of the vehicle and its surrounding vehicles; inputting the low-resolution dynamic scene data into a cross-modal attention fusion network model to output a spatio-temporal alignment feature dataset of multiple perspectives and multiple time points; wherein the cross-modal attention fusion network model comprises a modal alignment module and a spatio-temporal alignment module based on binocular disparity estimation; inputting the spatio-temporal alignment feature dataset into a neural implicit scene model to output implicit scene features; wherein the neural implicit scene model comprises a multi-scale perception network module and a hybrid density network module; the multi-scale perception network module comprises a coordinate encoding layer, a multi-scale fusion layer and a gradient conversion layer; inputting the implicit scene features into a spatio-temporal attention enhancement model to output enhanced implicit scene features; wherein the spatio-temporal attention enhancement model comprises a spatio-temporal attention calculation layer, a dynamic attention mask layer and a sub-pixel convolution enhancement layer; performing super-resolution reconstruction on the dynamic scene according to the enhanced implicit scene features, a binocular disparity consistency algorithm and a central visual enhancement algorithm.

[0007] In an optional manner, the modal alignment module of the cross-modal attention fusion network model performs dynamic feature calibration through feature mapping, and the feature mapping formula is: wherein, is the calibrated feature matrix; is the total number of modalities; is the original feature matrix of the mth modality; C is the number of feature channels; H is the brightness of the feature map; W is the width of the feature map; is the singular value decomposition result of is the modality importance weight; is the global feature vector; is the learnable calibration matrix; is the left singular vector matrix of is the singular value matrix of is the inverse matrix of is the right singular vector matrix of is the transpose of is the bias vector after calibration.

[0008] In an optional manner, the spatiotemporal alignment module is optimized using a continuous optical flow field, and the energy function of the continuous optical flow field is: in, is the weight of the smoothing term; Optical flow field spatial gradients; Optical flow field The L2 norm square of the spatial gradient; A coefficient that controls the fidelity of the optical flow field data; is the brightness value of pixel x in the current image I moving along the optical flow field u; is the brightness value of the corresponding pixel x in the next frame image J; is the rate at which the optical flow field u changes with time t; is the two-dimensional pixel area of ​​image I; optical flow field The Euler-Lagrange equations satisfied are: in, is the Laplace operator; is the brightness gradient of image I; is the rate of change of the brightness gradient at position x+u in the current frame image I.

[0009] In an optional manner, the mixed density network module adopts a variational autoencoder, and the lower bound of the variational autoencoder is: in, To infer the model expectations; is the probability distribution of the decoder; is the prior distribution of the latent variable z; To infer the model exist KL divergence of conditional probability; is the conditional distribution of the latent variable z, ; is the mean of the encoder network output; is the variance of the encoder network output; is the standard normal distribution function; The Jacobian of is calculated using the chain rule, which is: in, Indicated by the parameter A reversible transformation function is defined to map the latent variable z to another space; is the transpose of z, which means to find the partial derivative of each component of z; Reversible change It is composed of K continuous flow layers; represents the kth reversible flow layer; represents the input variable of the k-th transformation; is the identity matrix.

[0010] In an optional manner, the cross-modal attention fusion network model further includes an adaptive feature pyramid pooling layer; The adaptive feature pyramid pooling layer uses parallel dilated convolution kernels with dilated rates of {1, 2, 4, 8} to extract multi-scale feature maps, and dynamically generates a spatial attention map based on the input noise level according to the feature response prediction network.

[0011] In an optional manner, the dynamic attention mask layer obtains pixel-level motion vectors based on the U-Ne optical flow estimation network; Calculating the L2 norm of the optical flow vector between adjacent frames according to the pixel-level motion vector to obtain a motion intensity map; Adaptively determining a motion threshold of the motion intensity map according to the Otsu algorithm to generate a binary dynamic area mask; A Hadamard product operation is performed on the binary dynamic area mask and the basic attention weight.

[0012] In an optional manner, performing super-resolution reconstruction on a dynamic scene based on the enhanced implicit scene features, the binocular disparity consistency algorithm, and the center vision enhancement algorithm further includes: generating a preliminary high-resolution color image and a first depth image of the enhanced implicit scene features by a ray tracing method; Fusing the preliminary high-resolution depth image with a second depth image calculated using a binocular disparity consistency algorithm to obtain scene depth information; A dynamic scene super-resolution reconstruction result is generated according to the first depth image and the scene depth information.

[0013] In an optional manner, forming low-resolution dynamic scene data based on point cloud data, RGB-D images, and speed / distance information of the vehicle and its surrounding vehicles further includes: Adaptively downsampling the original high-density point cloud data of the vehicle and its surrounding vehicles based on voxel grid or farthest point sampling, extracting key geometric features through a normal estimation method, and forming a low-density point cloud representation; Performing resolution reduction on the RGB-D image using a multi-scale Gaussian pyramid method, and extracting color, depth, and high-level semantic information using a depth estimation algorithm, wherein the high-level semantic information includes vehicle type, road area, and pedestrian information; The speed / distance information is timestamped and spatially aligned with the downsampled point cloud data and the reduced RGB-D image, and integrated into a context-sensitive multimodal low-resolution feature vector through a deep learning model as the low-resolution dynamic scene data; wherein the speed / distance information includes the relative speed, absolute speed, relative distance, and absolute position of the vehicle and surrounding vehicles.

[0014] According to another aspect of the present invention, a super-resolution reconstruction device for dynamic scenes of intelligent connected vehicles based on neural implicit representation is provided, comprising: A low-resolution data generation module is used to generate low-resolution dynamic scene data based on the point cloud data, RGB-D images, and speed / distance information of the vehicle and its surrounding vehicles; A cross-modal spatiotemporal feature fusion module is used to input the low-resolution dynamic scene data into a cross-modal attention fusion network model and output a multi-view and multi-time spatiotemporal alignment feature dataset; wherein the cross-modal attention fusion network model includes a modality alignment module and a spatiotemporal alignment module based on binocular disparity estimation; An implicit feature generation module, configured to input the spatiotemporal alignment feature dataset into a neural implicit scene model and output implicit scene features; wherein the neural implicit scene model includes a multi-scale perception network module and a mixture density network module; the multi-scale perception network module includes a coordinate encoding layer, a multi-scale fusion layer, and a gradient conversion layer; An implicit feature enhancement module, configured to input the implicit scene features into a spatiotemporal attention enhancement model and output enhanced implicit scene features; wherein the spatiotemporal attention enhancement model comprises a spatiotemporal attention calculation layer, a dynamic attention mask layer, and a sub-pixel convolution enhancement layer; The high-resolution scene generation module is used to perform super-resolution reconstruction of dynamic scenes based on the enhanced implicit scene features, binocular disparity consistency algorithm and central vision enhancement algorithm.

[0015] According to another aspect of the present invention, there is provided a computing device, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the above-mentioned neural implicit expression method for super-resolution reconstruction of dynamic scenes of intelligent connected vehicles.

[0016] According to the scheme provided in the application, low-resolution dynamic scene data is formed according to point cloud data, RGB-D images and speed / distance information of a vehicle and surrounding vehicles of the vehicle; the low-resolution dynamic scene data is input into a cross-modal attention fusion network model to output a spatio-temporal alignment feature dataset of multiple perspectives and multiple time points; wherein the cross-modal attention fusion network model comprises a modal alignment module and a spatio-temporal alignment module based on binocular disparity estimation; the spatio-temporal alignment feature dataset is input into a neural implicit scene model to output an implicit scene feature; wherein the neural implicit scene model comprises a multi-scale perception network module and a hybrid density network module; the multi-scale perception network module comprises a coordinate encoding layer, a multi-scale fusion layer and a gradient conversion layer; the implicit scene feature is input into a spatio-temporal attention enhancement model to output an enhanced implicit scene feature; wherein the spatio-temporal attention enhancement model comprises a spatio-temporal attention calculation layer, a dynamic attention mask layer and a sub-pixel convolution enhancement layer; and a dynamic scene is super-resolution reconstructed according to the enhanced implicit scene feature, a binocular disparity consistency algorithm and a center vision enhancement algorithm. The application overcomes the problem of detail loss caused by traditional pixel interpolation or point cloud down-sampling, enhances the spatio-temporal consistency of a dynamic scene, and provides reliable environmental scene information for an intelligent networked vehicle.

[0017] The above description is only a summary of the technical scheme of the application. In order to enable the technical means of the application to be more clearly understood, the application can be implemented according to the content of the specification, and in order to enable the above and other purposes, features and advantages of the application to be more apparent and easy to understand, the following specific embodiments of the application are described. BRIEF DESCRIPTION OF DRAWINGS

[0018] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The accompanying drawings are included to provide a description of the preferred embodiments and are not meant to limit the present application. Furthermore, the same reference numerals are used throughout the several drawings to represent similar components. In the drawings: Figure 1 A flowchart of a neural implicit expression intelligent networked vehicle dynamic scene super-resolution reconstruction method of an embodiment of the application is shown; Figure 2 A working flowchart of a cross-modal attention fusion network model of an embodiment of the application is shown; Figure 3 A diagram showing multi-scale information in images captured by different receptive fields of an embodiment of the application is shown; Figure 4 A dynamic scene diagram of an embodiment of the application is shown; Figure 5 A framework diagram of a neural implicit expression intelligent networked vehicle dynamic scene super-resolution reconstruction device of an embodiment of the application is shown; Figure 6 A schematic structural diagram of a computing device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0019] Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0020] Figure 1 The flowchart of the method for super-resolution reconstruction of dynamic scenes of intelligent connected vehicles based on neural implicit expression according to an embodiment of the present invention is shown. Figure 1 As shown, the following steps are included: Step S101 : low-resolution dynamic scene data is generated based on the point cloud data, RGB-D image, and speed / distance information of the vehicle and its surrounding vehicles.

[0021] In this embodiment, point cloud data provides three-dimensional geometric structure information, has excellent representation capabilities for the shape and position of vehicles and obstacles, and is insensitive to lighting. RGB-D images combine the rich texture, color, and semantic information (such as lane markings, traffic signs, vehicle type, etc.) of traditional RGB images, while depth maps provide pixel-level depth information. Speed / distance information provides the vehicle's own motion state (speed, acceleration) and its dynamic relationship with surrounding objects (relative speed, distance). Combining these heterogeneous data sources comprehensively depicts dynamic scenes from different dimensions (geometry, texture, semantics, and dynamics), compensating for the limitations of single modalities (such as LiDAR's lack of texture information, RGB's susceptibility to lighting, and the limited accuracy of depth maps).

[0022] For example, there is a car accelerating in front of the intelligent connected vehicle and a pedestrian on the right sidewalk. LiDAR generates high-density 3D point cloud around the vehicle, outlining the contour of the front car, the road edge, and the approximate shape of the pedestrian. The RGB-D camera captures the license plate, body color, surrounding environment (such as traffic lights, buildings) of the front car, and the clothing and facial features of the pedestrian, and the depth map provides pixel-level depth information of the car and pedestrian from the vehicle. The speed of the vehicle itself is 60 km / h, the radar detects that the relative speed of the front car is +10 km / h (indicating that it is accelerating away), and the distance is 30 meters, and the relative speed of the pedestrian is 0 km / h, and the distance is 10 meters. The original high-density point cloud is down-sampled by voxel, for example, reducing 1 million points to 50,000 points but still clearly distinguishing the 3D shape of the front car and the pedestrian and estimating the surface normal. The original 1080p RGB-D image is reduced to 270p (4 times reduction in resolution). Although the resolution is reduced, processing through methods such as multi-scale Gaussian pyramid can still identify that the front is a car (high-level semantic information), its body color, and the right side is a pedestrian. Synchronize and align the down-sampled point cloud and the reduced RGB-D image and encode them into feature vectors. The above low-resolution data (down-sampled point cloud features, reduced RGB-D image features, dynamic information vectors) are input into a pre-trained deep learning fusion model to output low-resolution feature vectors with context association.

[0023] In an alternative way, forming low-resolution dynamic scene data from point cloud data, RGB-D images, and speed / distance information of the vehicle and its surrounding vehicles further comprises: Adaptive down-sampling of the original high-density point cloud data of the vehicle and its surrounding vehicles based on voxel grid or farthest point sampling, extracting key geometric features through normal estimation methods, and forming low-density point cloud representation; Resolution reduction of the RGB-D image through multi-scale Gaussian pyramid method, extracting color, depth, and high-level semantic information through depth estimation algorithm, wherein the high-level semantic information includes vehicle type, road area, and pedestrian information; Timestamp synchronization and spatial coordinate system alignment of the speed / distance information with the down-sampled point cloud data and the reduced RGB-D image, and integration into multi-modal low-resolution feature vectors with context association through a deep learning model, as the low-resolution dynamic scene data; wherein the speed / distance information includes the relative speed, absolute speed, relative distance, and absolute position of the vehicle and its surrounding vehicles.

[0024] In this embodiment, raw, high-density point cloud data of a vehicle and its surroundings is input, with the goal of reducing data volume while preserving key geometric features. The voxel size is adjusted based on the local density or geometric complexity of the scene. For example, smaller voxels are used in areas with rich detail, such as vehicle surfaces or pedestrians, while larger voxels are used in open areas to preserve finer features. By applying different sampling densities in different regions or incorporating other characteristics (such as curvature) into sampling point selection, more points are retained in areas with rich geometric detail. For example, a point in the point cloud is randomly selected as the first sampling point. The maximum distance from all remaining points to the currently selected sampling point is calculated, and the point with the maximum distance is added as the next sampling point. This process repeats until a preset number of sampling points is reached. For each point P in the downsampled point cloud, its K nearest neighboring points are selected. Principal component analysis is performed on these neighboring points, and the eigenvector corresponding to the minimum eigenvalue of the covariance matrix is ​​calculated, which is the normal of that point. The output contains the 3D coordinates of the point and the corresponding normal vector (a low-density point cloud representation).

[0025] For RGB-D images, the system inputs raw RGB and depth images acquired by the vehicle's RGB-D camera. The goal is to reduce the resolution and extract color, depth, and high-level semantic information. A Gaussian filter is applied to the raw RGB-D image to smooth the image and remove noise. The smoothed image is then rescaled (for example, by skipping pixels or averaging pixel blocks). This process repeatedly generates a series of image layers with decreasing resolution, forming a pyramid structure. The system outputs a series of RGB-D images of varying resolutions, with the lower-resolution images used to form low-resolution dynamic scene data. A pre-trained deep learning model (such as a convolutional neural network-based semantic segmentation network, such as DeepLabV3+ or U-Net) is used to process the RGB or RGB-D images and identify and segment different semantic regions and objects within the image. The system outputs reduced-resolution RGB and depth images, along with corresponding semantic segmentation maps or semantic feature maps.

[0026] To ensure that point cloud data, RGB-D images, and speed / distance information are captured at or very close to each other, all sensor data is transformed into a unified coordinate system (e.g., the vehicle's body coordinate system or the world coordinate system). The preprocessed low-density point cloud, reduced-resolution RGB-D image, extracted semantic information, and aligned speed / distance information are fused end-to-end to generate a contextually relevant low-resolution feature vector. The point cloud branch uses networks such as PointNet, PointNet++, and GNNs (Graph Neural Networks) to extract point cloud features. The image branch uses CNNs to extract features from the RGB image, depth map, and semantic map. The speed / distance branch is embedded in the fusion layer via an MLP (Multi-Layer Perceptron) or directly as a vector. The final output is a compact, contextually relevant multimodal low-resolution feature vector, which serves as the initial input to the entire super-resolution reconstruction network. This vector encodes the geometric, visual, depth, semantic, and motion information of the dynamic scene.

[0027] For example, consider a self-driving car traveling on a city road. In front of it are several moving vehicles, some parked vehicles, a pedestrian crossing the road, and traffic signs and lane markings. The vehicle's LiDAR (Light Detection and Ranging) collects point cloud data every second, outlining the three-dimensional contours of the vehicles and pedestrians ahead. Adaptive downsampling (voxel grid) divides the scene into 0.2m voxels. In areas with rich detail, such as trucks and pedestrians, only one representative point is selected from each voxel, but the high voxel density in these areas preserves their essential shapes. In open road areas, the number of points in a voxel is significantly reduced, thereby reducing the amount of data. A normal is calculated for each downsampled point. For example, points on the side of a truck have a nearly horizontal normal, while the normal on the hood is slightly tilted upward. This normal information helps identify the surface orientation of the object. The front-facing RGB-D camera captures 2K (2048x1080) resolution color and depth images, clearly showing vehicles, pedestrians, and traffic signs ahead. These images are then downsampled to a lower resolution using a Gaussian pyramid, for example, from 2K to 512x270. While details may be blurred, the outlines and colors of key objects (such as vehicles and pedestrians) remain discernible. A pre-trained semantic segmentation network (such as DeepLabV3+) is then run to identify cars and trucks ahead and classify their pixels as vehicles, pedestrians crossing the road and classify their pixels as pedestrians, and areas such as roads, lane markings, and traffic signs. The resulting 512x270 resolution RGB and depth images are accompanied by corresponding semantic segmentation maps (with per-pixel class labels). This process yields information-rich low-resolution dynamic scene data, no longer simply discrete point clouds, images, or numbers, but rather highly abstract, semantically rich, and contextually relevant multimodal feature vectors. These feature vectors serve as input to a neural implicit representation model for subsequent super-resolution reconstruction, enabling more accurate environmental perception and decision-making in autonomous driving systems.

[0028] Step S102: input the low-resolution dynamic scene data into a cross-modal attention fusion network model, and output a multi-view and multi-time spatiotemporal alignment feature dataset; wherein the cross-modal attention fusion network model includes a modality alignment module and a spatiotemporal alignment module based on binocular disparity estimation.

[0029] In this embodiment, vehicles, pedestrians, and other objects are constantly moving, and aligning static features alone is not enough to accurately reconstruct dynamic scenes, such as Figure 2 As shown in the figure, the spatiotemporal alignment module of binocular disparity estimation can accurately estimate the motion and depth information of objects in the scene, thereby achieving fine alignment of data from different perspectives (binocular) and at different times (optical flow) in three-dimensional space and time dimensions.

[0030] In an optional manner, the modality alignment module of the cross-modal attention fusion network model performs dynamic feature calibration through feature mapping, and the feature mapping formula is: in, , is the original characteristic matrix of the mth mode; for The singular value decomposition result of ; , is the modal importance weight; , is the global eigenvector; , is the learnable calibration matrix; for The left singular vector matrix of ; for The singular value matrix of The inverse matrix of for The right singular vector matrix of The transpose of is the bias vector after calibration.

[0031] In this embodiment, the weighted SVD reconstruction term is calculated for each modality. Project it into its principal component space and scale it, and extract the main structural information of the modal features by performing SVD decomposition on the original feature matrix, so that the essential information of the features can be retained during the calibration process. After calibration, It combines the strengths of LiDAR and RGB-D and dynamically adjusts the contribution based on the importance of the modality in the current scene. For example, LiDAR contributes more to the geometry of a truck, while RGB-D contributes more to the color of the truck surface and the recognition of pedestrian clothing.

[0032] In an optional manner, the spatiotemporal alignment module is optimized using a continuous optical flow field, and the energy function of the continuous optical flow field is: in, is the weight of the smoothing term; Optical flow field spatial gradients; A coefficient that controls the fidelity of the optical flow field data; is the brightness value of pixel x in the current image I moving along the optical flow field u; is the brightness value of the corresponding pixel x in the next frame image J; is the rate at which the optical flow field u changes with time t; is the two-dimensional pixel area of ​​image I; optical flow field The Euler-Lagrange equations satisfied are: in, is the Laplace operator; is the brightness gradient of image I.

[0033] In this embodiment, the optical flow method can estimate the motion vector of each pixel in the image, rather than just the overall motion. By minimizing the energy function, it can more accurately capture the fine motion of various elements in the scene, such as vehicles, pedestrians, and background, thereby achieving more accurate spatiotemporal alignment. By solving the Euler-Lagrange equation, the continuity and differentiability of the optical flow field are ensured, and it can be jointly optimized through gradient descent in an end-to-end deep learning framework. The continuous optical flow field provides pixel-level temporal correlation, enabling the precise fusion of features at different times, thereby utilizing richer information when reconstructing high-resolution scenes.

[0034] Step S103: input the spatiotemporal alignment feature dataset into a neural implicit scene model and output implicit scene features; wherein the neural implicit scene model includes a multi-scale perception network module and a hybrid density network module; the multi-scale perception network module includes a coordinate encoding layer, a multi-scale fusion layer and a gradient conversion layer.

[0035] In this embodiment, traditional methods typically require explicit storage of high-resolution 3D meshes, point clouds, or voxels, resulting in massive data volumes and difficulty in transmission and real-time processing. The neural implicit scene model uses a neural network to learn a continuous function representation of the scene, encoding complex scene information as neural network weights. This significantly compresses the scene data while capturing continuous geometry and appearance information. This allows for super-resolution reconstruction, enabling outputs of arbitrary resolutions, rather than being limited to the discrete resolution of the original data. The coordinate encoding layer maps low-dimensional input coordinates (e.g., 3D spatial coordinates and time) to a high-dimensional feature space, making it easier for the neural network to learn and distinguish between different points in space. The multi-scale fusion layer understands scene features at different scales. For example, in a vehicle scene, it is necessary to understand both the overall shape of the vehicle and fine structures such as license plates and lights. The gradient transformation layer learns the geometric gradient information of the scene, thereby improving the geometric accuracy and visual quality of the reconstruction results.

[0036] In an optional manner, the mixed density network module adopts a variational autoencoder, and the lower bound of the variational autoencoder is: in, To infer the model expectations; is the probability distribution of the decoder; is the prior distribution of the latent variable z; To infer the model exist KL divergence of conditional probability; is the conditional distribution of the latent variable z, ; is the mean of the encoder network output; is the variance of the encoder network output; is the standard normal distribution function; The Jacobian of is calculated using the chain rule, which is: in, Indicated by the parameter A reversible transformation function is defined to map the latent variable z to another space; is the transpose of z, which means to find the partial derivative of each component of z; Reversible change It is composed of K continuous flow layers; represents the kth reversible flow layer; represents the input variable of the k-th transformation; is the identity matrix.

[0037] In this embodiment, through reversible transformations (flows), the variational autoencoder learns a more complex and flexible latent space distribution, thereby better capturing the intrinsic structure and multimodal characteristics of the data. By using flow layers to model the distribution of latent variables, the true probability density of data points is more accurately estimated, thereby better understanding and reproducing the statistical characteristics of the scene. In addition, the variational autoencoder itself has a certain degree of interpretability, allowing the characteristics of the generated results to be controlled by manipulating the latent variables. Combined with reversible transformations, more precise control of transformations in the latent space facilitates more precise control and adjustment of specific scene elements (such as vehicle type and motion trajectory) during super-resolution reconstruction.

[0038] In an optional manner, the cross-modal attention fusion network model further includes an adaptive feature pyramid pooling layer; The adaptive feature pyramid pooling layer uses parallel dilated convolution kernels with dilated rates of {1, 2, 4, 8} to extract multi-scale feature maps, and dynamically generates a spatial attention map based on the input noise level according to the feature response prediction network.

[0039] In this embodiment, Figure 3As shown in the figure, dilated convolution kernels with parallel dilation rates {1, 2, 4, 8} are used to capture multi-scale information in the image with different receptive fields. Furthermore, they can simultaneously obtain local details and global context, thereby improving feature richness and expressiveness. A dilation rate of 1 is equivalent to a standard convolution, capturing the most local features. Dilation rates of 2, 4, and 8 expand the receptive field without increasing the number of parameters or losing resolution, capturing medium- and wider-scale contextual information, respectively.

[0040] Step S104: input the implicit scene features into the spatiotemporal attention enhancement model and output enhanced implicit scene features; wherein the spatiotemporal attention enhancement model includes a spatiotemporal attention calculation layer, a dynamic attention mask layer and a sub-pixel convolution enhancement layer.

[0041] In this embodiment, the background is usually relatively static, while vehicles, pedestrians, etc. in the foreground are the main moving objects. The dynamic attention mask layer can identify and highlight the moving areas in the scene, focusing attention resources on the areas with the most significant changes and the most urgent super-resolution needs, avoiding unnecessary over-processing of the static background. Sub-pixel convolution (also known as PixelShuffle) is an efficient upsampling method that generates high-resolution images by rearranging and aggregating the channel information of low-resolution feature maps into spatial dimensions, avoiding the blur and artifacts that may be introduced by traditional interpolation methods (such as bilinear interpolation and bicubic interpolation). Unlike simple upsampling methods, sub-pixel convolution can better retain the detailed information in the implicit scene features while expanding the resolution, and distribute the above information to new high-resolution pixels to generate clearer and more natural super-resolution images.

[0042] In an optional manner, the dynamic attention mask layer obtains pixel-level motion vectors based on the U-Ne optical flow estimation network; Calculating the L2 norm of the optical flow vector between adjacent frames according to the pixel-level motion vector to obtain a motion intensity map; Adaptively determining a motion threshold of the motion intensity map according to the Otsu algorithm to generate a binary dynamic area mask; A Hadamard product operation is performed on the binary dynamic area mask and the basic attention weight.

[0043] In this embodiment, Figure 4As shown, the U-Net optical flow estimation network obtains pixel-level motion vectors, rather than simply relying on pre-set static regions or coarse-grained motion information. This allows for precise identification of areas of actual motion within an image. By calculating the L2 norm of the optical flow vectors, a motion intensity map is generated, providing a quantitative indicator of the intensity of motion for each pixel. This allows for distinguishing between mild and intense motion, enabling more nuanced processing of dynamic scenes. The Otsu algorithm automatically determines the threshold of the motion intensity map, adaptively dividing the image into foreground (moving regions) and background (static regions) based on the motion characteristics of the current frame.

[0044] Step S105 , performing super-resolution reconstruction on the dynamic scene according to the enhanced implicit scene features, the binocular disparity consistency algorithm and the central vision enhancement algorithm.

[0045] In this embodiment, the enhanced implicit scene features are high-dimensional, continuous, deblurred representations of scene geometry, color, material, etc. learned by the neural implicit scene model. They contain the intrinsic structural information of the scene and can overcome the discreteness and sampling limitations of traditional pixel- or voxel-based representations. The binocular disparity consistency algorithm uses the principle of multi-view geometry to provide accurate and robust depth information, which can correct the depth ambiguity or inaccuracy that may exist in certain areas of the implicit model, especially in areas with missing or occluded textures, and improve depth accuracy through consistency constraints across perspectives. The central vision enhancement algorithm simulates the visual characteristics of the human eye, emphasizing the details of the central area while maintaining global consistency. In intelligent driving scenarios, information about the vehicle's direction of travel (central area) is often the most critical. By focusing on the above-mentioned areas, the vehicle's recognition accuracy and perception quality of key targets such as the road ahead, obstacles, and pedestrians are improved, thereby improving driving safety.

[0046] In an optional manner, performing super-resolution reconstruction on a dynamic scene based on the enhanced implicit scene features, the binocular disparity consistency algorithm, and the center vision enhancement algorithm further includes: generating a preliminary high-resolution color image and a first depth image of the enhanced implicit scene features by a ray tracing method; Fusing the preliminary high-resolution depth image with a second depth image calculated using a binocular disparity consistency algorithm to obtain scene depth information; A dynamic scene super-resolution reconstruction result is generated according to the first depth image and the scene depth information.

[0047] In this embodiment, for example, the enhanced implicit scene features contain the 3D geometry, color, material, and dynamic information of the road, vehicles, pedestrians, buildings, and other objects in the scene, but in the form of continuous functions rather than discrete pixels or points. The scene is reconstructed from the driver's perspective (or a high-resolution virtual perspective). Ray tracing techniques are used to traverse the continuous 3D space represented by the enhanced implicit scene features by emitting numerous rays from this perspective. This results in a preliminary 1920x1080 pixel high-resolution color image (which may contain minor artifacts in some undertrained areas) and a corresponding first depth image (derived from the 3D geometric representation within the implicit model). Binocular disparity is calculated between the left and right images captured by the binocular cameras to generate a second depth image. This depth image is highly accurate in texture-rich areas such as vehicle and pedestrian outlines and road edges, but may exhibit large gaps or jumps in weakly textured areas such as the sky and smooth walls. The generated first and second depth images are fused. Binocular depth is preferentially used in areas where confidence in binocular depth is high (e.g., areas with high texture and low matching cost). In areas where binocular depth is missing or confidence is low (such as areas with weak texture or occlusion), depth generated by ray tracing is used to fill in and correct for missing depth. A weighted averaging or voting mechanism is used at object edges to ensure smooth transitions in depth values ​​while maintaining edge sharpness. This results in a more complete and accurate scene depth map than a single depth source (for example, combining the accurate binocular outlines of close-up objects with the continuity of ray tracing in distant and weakly textured areas). If a pedestrian is walking down the road in the scene, the central vision enhancement algorithm applies additional sharpening and detail restoration to the pedestrian area to identify this key dynamic target. This ensures that details such as the pedestrian's clothing and posture are captured with the highest accuracy in both the final reconstructed image and depth map. This helps the autonomous driving system more accurately identify and predict pedestrian behavior, enabling precise environmental perception (for example, accurately measuring the distance to the vehicle ahead, identifying the 3D position of lane lines, and detecting potential obstacles), providing intelligent connected vehicles with higher-precision dynamic scene perception capabilities.

[0048] According to the solution provided by the present invention, low-resolution dynamic scene data is formed based on the point cloud data, RGB-D image and speed / distance information of the vehicle and its surrounding vehicles; the low-resolution dynamic scene data is input into a cross-modal attention fusion network model to output a multi-view and multi-time spatiotemporal alignment feature dataset; wherein the cross-modal attention fusion network model includes a modal alignment module and a spatiotemporal alignment module based on binocular disparity estimation; the spatiotemporal alignment feature dataset is input into a neural implicit scene model to output implicit scene features; wherein the neural implicit scene model includes a multi-scale perception network module and a mixed density network module; the multi-scale perception network module includes a coordinate encoding layer, a multi-scale fusion layer and a gradient conversion layer; the implicit scene features are input into a spatiotemporal attention enhancement model to output enhanced implicit scene features; wherein the spatiotemporal attention enhancement model includes a spatiotemporal attention calculation layer, a dynamic attention mask layer and a sub-pixel convolution enhancement layer; and the dynamic scene is super-resolution reconstructed based on the enhanced implicit scene features, the binocular disparity consistency algorithm and the central vision enhancement algorithm. The present invention overcomes the problem of detail loss caused by traditional pixel interpolation or point cloud downsampling, enhances the spatiotemporal consistency of dynamic scenes, and provides reliable environmental scene information for intelligent connected vehicles.

[0049] Figure 5 The schematic diagram of the framework of the super-resolution reconstruction device for dynamic scenes of intelligent connected vehicles based on neural implicit expression according to an embodiment of the present invention is shown. The super-resolution reconstruction device for dynamic scenes of intelligent connected vehicles based on neural implicit expression includes: a low-resolution data generation module 510 for generating low-resolution dynamic scene data based on point cloud data, RGB-D images, and speed / distance information of the vehicle and its surrounding vehicles; A cross-modal spatiotemporal feature fusion module 520 is configured to input the low-resolution dynamic scene data into a cross-modal attention fusion network model and output a multi-view and multi-time spatiotemporal alignment feature dataset; wherein the cross-modal attention fusion network model includes a modality alignment module and a spatiotemporal alignment module based on binocular disparity estimation; An implicit feature generation module 530 is configured to input the spatiotemporal alignment feature dataset into a neural implicit scene model and output implicit scene features; wherein the neural implicit scene model includes a multi-scale perception network module and a mixture density network module; the multi-scale perception network module includes a coordinate encoding layer, a multi-scale fusion layer, and a gradient conversion layer; An implicit feature enhancement module 540 is configured to input the implicit scene features into a spatiotemporal attention enhancement model and output enhanced implicit scene features; wherein the spatiotemporal attention enhancement model includes a spatiotemporal attention calculation layer, a dynamic attention mask layer, and a sub-pixel convolution enhancement layer; The high-resolution scene generation module 550 is configured to perform super-resolution reconstruction on the dynamic scene according to the enhanced implicit scene feature, the binocular disparity consistency algorithm, and the central vision enhancement algorithm.

[0050] Figure 6 The structural diagram of the computing device embodiment of the present application is shown, and the specific implementation of the computing device is not limited by the specific embodiment of the present application.

[0051] As shown in Figure 6 The computing device can include a processor 602, a communications interface 604, a memory 606, and a communications bus 608.

[0052] The processor 602, the communications interface 604, and the memory 606 can communicate with each other through the communications bus 608. The communications interface 604 is configured to communicate with network elements such as clients or other servers. The processor 602 is configured to execute the program 610, and specifically can execute the related steps in the neural implicit representation intelligent connected vehicle dynamic scene super-resolution reconstruction method embodiment described above.

[0053] Specifically, the program 610 can include program code including computer operation instructions.

[0054] The processor 602 can be a central processing unit CPU, or an application specific integrated circuit ASIC, or one or more integrated circuits configured to implement embodiments of the present application. The one or more processors included in the computing device can be the same type of processor, such as one or more CPUs; or can be different types of processors, such as one or more CPUs and one or more ASICs.

[0055] The memory 606 is configured to store the program 610. The memory 606 can include a high-speed RAM memory, and can also include a non-volatile memory such as at least one disk memory.

[0056] According to the solution provided by the present invention, low-resolution dynamic scene data is formed based on the point cloud data, RGB-D image and speed / distance information of the vehicle and its surrounding vehicles; the low-resolution dynamic scene data is input into a cross-modal attention fusion network model to output a multi-view and multi-time spatiotemporal alignment feature dataset; wherein the cross-modal attention fusion network model includes a modal alignment module and a spatiotemporal alignment module based on binocular disparity estimation; the spatiotemporal alignment feature dataset is input into a neural implicit scene model to output implicit scene features; wherein the neural implicit scene model includes a multi-scale perception network module and a mixed density network module; the multi-scale perception network module includes a coordinate encoding layer, a multi-scale fusion layer and a gradient conversion layer; the implicit scene features are input into a spatiotemporal attention enhancement model to output enhanced implicit scene features; wherein the spatiotemporal attention enhancement model includes a spatiotemporal attention calculation layer, a dynamic attention mask layer and a sub-pixel convolution enhancement layer; and the dynamic scene is super-resolution reconstructed based on the enhanced implicit scene features, the binocular disparity consistency algorithm and the central vision enhancement algorithm. The present invention overcomes the problem of detail loss caused by traditional pixel interpolation or point cloud downsampling, enhances the spatiotemporal consistency of dynamic scenes, and provides reliable environmental scene information for intelligent connected vehicles.

[0057] Those skilled in the art will appreciate that the modules in the devices of the embodiments can be adaptively modified and deployed in one or more devices different from the embodiments. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and furthermore, they can be divided into multiple sub-modules, sub-units, or sub-components. All features disclosed in this specification (including the accompanying claims, abstract, and drawings), as well as all processes or units of any method or device disclosed therein, can be combined in any combination, unless at least some of such features and / or processes or units are mutually exclusive. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature serving the same, equivalent, or similar purpose. Furthermore, those skilled in the art will appreciate that although some embodiments herein include certain features and not others included in other embodiments, combinations of features from different embodiments are intended to fall within the scope of the present invention and form different embodiments. For example, in the claims below, any of the claimed embodiments may be used in any combination. The present invention may be implemented by means of hardware comprising several distinct elements and by means of a suitably programmed computer. In a unit claim that lists several means, several of these means may be embodied by the same hardware item. Unless otherwise specified, the steps in the above embodiments should not be understood as limiting the order of execution.

Claims

1. A neural implicit representation method for super-resolution reconstruction of dynamic scenes of intelligent connected vehicles, characterized by: include: Generate low-resolution dynamic scene data based on point cloud data, RGB-D images, and speed / distance information of the vehicle and its surrounding vehicles; Inputting the low-resolution dynamic scene data into a cross-modal attention fusion network model to output a multi-view and multi-time spatiotemporal alignment feature dataset; wherein the cross-modal attention fusion network model includes a modality alignment module and a spatiotemporal alignment module based on binocular disparity estimation; Inputting the spatiotemporal alignment feature dataset into a neural implicit scene model to output implicit scene features; wherein the neural implicit scene model includes a multi-scale perception network module and a hybrid density network module; the multi-scale perception network module includes a coordinate encoding layer, a multi-scale fusion layer, and a gradient conversion layer; Inputting the implicit scene features into a spatiotemporal attention enhancement model and outputting enhanced implicit scene features; wherein the spatiotemporal attention enhancement model includes a spatiotemporal attention calculation layer, a dynamic attention mask layer, and a sub-pixel convolution enhancement layer; The dynamic scene is super-reconstructed based on the enhanced implicit scene features, binocular disparity consistency algorithm and central vision enhancement algorithm.

2. The neural implicit representation method for super-resolution reconstruction of dynamic scenes of intelligent connected vehicles according to claim 1 is characterized in that: The modality alignment module of the cross-modal attention fusion network model performs dynamic feature calibration through feature mapping. The feature mapping formula is: in, is the calibrated feature matrix; is the total number of modes; , is the original feature matrix of the mth mode; C is the number of feature channels; H is the brightness of the feature map; W is the width of the feature map; for The singular value decomposition result of ; , is the modal importance weight; , is the global eigenvector; , is the learnable calibration matrix; for The left singular vector matrix of ; for The singular value matrix of The inverse matrix of for The right singular vector matrix of The transpose of is the bias vector after calibration.

3. The neural implicit representation method for super-resolution reconstruction of dynamic scenes of intelligent connected vehicles according to claim 1 is characterized in that: The spatiotemporal alignment module is optimized using a continuous optical flow field, and the energy function of the continuous optical flow field is: in, is the weight of the smoothing term; Optical flow field spatial gradients; Optical flow field The L2 norm square of the spatial gradient; A coefficient that controls the fidelity of the optical flow field data; is the brightness value of pixel x in the current image I moving along the optical flow field u; is the brightness value of the corresponding pixel x in the next frame image J; is the rate at which the optical flow field u changes with time t; is the two-dimensional pixel area of ​​image I; optical flow field The Euler-Lagrange equations satisfied are: in, is the Laplace operator; is the brightness gradient of image I; is the rate of change of the brightness gradient at position x+u in the current frame image I.

4. The neural implicit representation method for super-resolution reconstruction of dynamic scenes of intelligent connected vehicles according to claim 1 is characterized in that: The mixed density network module adopts a variational autoencoder, and the lower bound of the variational autoencoder is: in, To infer the model expectations; is the probability distribution of the decoder; is the prior distribution of the latent variable z; To infer the model exist KL divergence of conditional probability; is the conditional distribution of the latent variable z, ; is the mean of the encoder network output; is the variance of the encoder network output; is the standard normal distribution function; The Jacobian of is calculated using the chain rule, which is: in, Indicated by the parameter A reversible transformation function is defined to map the latent variable z to another space; is the transpose of z, which means to find the partial derivative of each component of z; Reversible change It is composed of K continuous flow layers; represents the kth reversible flow layer; represents the input variable of the kth transformation; is the identity matrix.

5. The neural implicit representation method for super-resolution reconstruction of dynamic scenes of intelligent connected vehicles according to claim 1 is characterized in that: The cross-modal attention fusion network model also includes an adaptive feature pyramid pooling layer; The adaptive feature pyramid pooling layer uses parallel dilated convolution kernels with dilated rates of {1, 2, 4, 8} to extract multi-scale feature maps, and dynamically generates a spatial attention map based on the input noise level according to the feature response prediction network.

6. The neural implicit expression method for super-resolution reconstruction of dynamic scenes of intelligent connected vehicles according to claim 1 is characterized in that: The dynamic attention mask layer obtains pixel-level motion vectors based on the U-Ne optical flow estimation network; Calculating the L2 norm of the optical flow vector between adjacent frames according to the pixel-level motion vector to obtain a motion intensity map; Adaptively determining a motion threshold of the motion intensity map according to the Otsu algorithm to generate a binary dynamic area mask; A Hadamard product operation is performed on the binary dynamic area mask and the basic attention weight.

7. The neural implicit representation method for super-resolution reconstruction of dynamic scenes of intelligent connected vehicles according to claim 1 is characterized in that: Performing super-resolution reconstruction of a dynamic scene based on the enhanced implicit scene features, binocular disparity consistency algorithm, and center vision enhancement algorithm further includes: generating a preliminary high-resolution color image and a first depth image of the enhanced implicit scene features by a ray tracing method; Fusing the preliminary high-resolution depth image with a second depth image calculated using a binocular disparity consistency algorithm to obtain scene depth information; A dynamic scene super-resolution reconstruction result is generated according to the first depth image and the scene depth information.

8. The method for super-resolution reconstruction of dynamic scenes of intelligent connected vehicles based on neural implicit expression according to claim 1 is characterized in that: Forming low-resolution dynamic scene data based on point cloud data, RGB-D images, and speed / distance information of the vehicle and its surrounding vehicles further includes: Adaptively downsampling the original high-density point cloud data of the vehicle and its surrounding vehicles based on voxel grid or farthest point sampling, extracting key geometric features through a normal estimation method, and forming a low-density point cloud representation; Performing resolution reduction on the RGB-D image using a multi-scale Gaussian pyramid method, and extracting color, depth, and high-level semantic information using a depth estimation algorithm, wherein the high-level semantic information includes vehicle type, road area, and pedestrian information; The speed / distance information is timestamped and spatially aligned with the downsampled point cloud data and the reduced RGB-D image, and integrated into a context-sensitive multimodal low-resolution feature vector through a deep learning model as the low-resolution dynamic scene data; wherein the speed / distance information includes the relative speed, absolute speed, relative distance, and absolute position of the vehicle and surrounding vehicles.

9. A super-resolution reconstruction device for dynamic scenes of intelligent connected vehicles based on neural implicit expression, characterized in that: A method for super-resolution reconstruction of dynamic scenes of intelligent connected vehicles according to any one of claims 1 to 8 is implemented, comprising: A low-resolution data generation module is used to generate low-resolution dynamic scene data based on the point cloud data, RGB-D images, and speed / distance information of the vehicle and its surrounding vehicles; A cross-modal spatiotemporal feature fusion module is used to input the low-resolution dynamic scene data into a cross-modal attention fusion network model and output a multi-view and multi-time spatiotemporal alignment feature dataset; wherein the cross-modal attention fusion network model includes a modality alignment module and a spatiotemporal alignment module based on binocular disparity estimation; An implicit feature generation module, configured to input the spatiotemporal alignment feature dataset into a neural implicit scene model and output implicit scene features; wherein the neural implicit scene model includes a multi-scale perception network module and a mixture density network module; the multi-scale perception network module includes a coordinate encoding layer, a multi-scale fusion layer, and a gradient conversion layer; An implicit feature enhancement module, configured to input the implicit scene features into a spatiotemporal attention enhancement model and output enhanced implicit scene features; wherein the spatiotemporal attention enhancement model comprises a spatiotemporal attention calculation layer, a dynamic attention mask layer, and a sub-pixel convolution enhancement layer; The high-resolution scene generation module is used to perform super-resolution reconstruction of dynamic scenes based on the enhanced implicit scene features, binocular disparity consistency algorithm and central vision enhancement algorithm.

10. A computing device comprising: A processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the neural implicit expression method for super-resolution reconstruction of dynamic scenes of intelligent connected vehicles described in any one of claims 1-8.

Images