Cross-sensor adaptive 3D object detection method and system based on neural rendering

Abstract

The invention relates to the technical field of data processing, and provides a cross-sensor adaptive 3D object detection method and system based on neural rendering, and the method comprises the steps: preprocessing a pre-obtained data sample into source sensor configuration data; training a pre-constructed neural rendering model based on the source sensor configuration data to obtain a cross-sensor adaptive 3D object detection model; performing 3D object detection on an input to-be-detected image based on the cross-sensor adaptive 3D object detection model; wherein the pre-constructed neural rendering model adopts an explicit dynamic-static decomposition neural radiation field architecture. According to the method, cross-sensor self-adaption can be driven based on real data, a scene is effectively separated into a static background and a dynamic object, different representation modes are adopted for processing, the problem of difference between simulation data and a real environment is solved, and geometric consistency of the dynamic object at a new view angle is guaranteed.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and more specifically, to a cross-sensor adaptive 3D object detection method, system, electronic device, and computer storage medium based on neural rendering. Background Technology

[0002] In autonomous driving systems, visual perception is a key technology for achieving environmental perception and decision-making control. In recent years, camera-based 3D object detection methods have been widely used due to their low cost and high performance, leading to the emergence of various advanced algorithm architectures such as BEVFormer and DETR3D. These methods typically rely on multi-view image input, achieving 3D detection through bird's-eye view (BEV) representation or query-based mechanisms. However, due to differences in vehicle body structure and installation space between different models (such as SUVs and small cars), the installation position, height, and spacing of their cameras vary, resulting in performance degradation when the same model is applied to different sensor configurations. This problem is known as cross-sensor domain variability.

[0003] Currently, resolving this discrepancy issue mainly faces the following technical challenges: First, while existing public datasets (such as nuScenes, Waymo, etc.) provide multi-camera surround view data, they lack synchronously acquired data with multiple sensor configurations in the same scene, making it difficult to isolate and study sensor domain differences. Secondly, most existing domain adaptation methods focus on changes in environmental factors (such as weather and lighting) while ignoring the differences in sensor configuration itself. Furthermore, while synthetic datasets (such as CARLA, SHIFT, etc.) can provide controlled environmental variations, there is still a lack of systematic evaluation benchmarks for differences in sensor configurations.

[0004] Therefore, there is an urgent need to provide a new visual perception solution that can effectively address the problem of cross-sensor domain differences in the field of autonomous driving. Summary of the Invention

[0005] This invention provides a cross-sensor adaptive 3D object detection method and system based on neural rendering, which solves the problem of decreased visual perception performance caused by differences in sensor configuration in the field of autonomous driving through innovative data generation and processing processes.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, embodiments of the present invention provide a cross-sensor adaptive 3D object detection method based on neural rendering, applied to electronic devices, comprising: The pre-acquired data samples are preprocessed, and the data samples are structured and stored in conjunction with the acquisition association information of each data sample as source sensor configuration data; wherein, the data samples are raw images acquired through a preset camera; The pre-built neural rendering model is trained based on the source sensor configuration data to obtain a cross-sensor adaptive 3D object detection model. The cross-sensor adaptive 3D object detection model is used to perform 3D object detection on the input image to be detected. The pre-built neural rendering model adopts a neural radiation field architecture, including static scene modeling and dynamic scene modeling. In the static scene modeling process, all objects in the image are trained under the assumption that the entire scene is absolutely static, and a static 3D scene is learned based on the neural network. In the dynamic scene modeling process, time is introduced as an additional dimension to train the objects in the image, and a static 3D space combined with 1D time 4D spatiotemporal scene is learned based on the neural network.

[0007] Alternatively, an alternative approach is to train a pre-built neural rendering model based on the source sensor configuration data, including: The neural rendering model is trained using source sensor configuration data to learn scene geometry and appearance features from the source sensor configuration data; Based on the scene geometry and appearance features in the source sensor configuration data, a synthetic image of the target sensor configuration is generated using differentiable rendering technology; The trained neural rendering model is fine-tuned using the synthesized images to obtain a cross-sensor adaptive 3D object detection model that adapts to the characteristics of the new sensor.

[0008] Alternatively, during the data sample acquisition process, a synchronization clock signal is generated by a multi-channel synchronization controller. The six cameras receive the synchronization clock signal to trigger the acquisition of synchronized data from different sensor configurations of multiple vehicle models as data samples. Each image frame acquired by each camera is appended with a timestamp. After image acquisition, the transmission delay of the images acquired by the six cameras is corrected by a preset timestamp alignment algorithm.

[0009] In addition, an optional approach is to preprocess the pre-acquired data samples, including: correcting camera parameter deviations in the data samples using a hybrid calibration strategy of offline calibration and online correction; and processing the pre-acquired data samples respectively using image sharpness detection based on the Laplacian operator, exposure consistency check based on histogram analysis, and color equalization processing based on the color correction matrix.

[0010] Alternatively, the acquisition association information for each data sample may include the acquisition timestamp, GPS location, IMU data, weather conditions, light intensity, and sensor operating status information for each data sample.

[0011] Alternatively, during the training of the neural rendering model using source sensor configuration data, 3D bounding box annotations with timestamps can be used to identify whether a point (x, y, z) belongs to a static environment or a dynamic object at time t. The static environment is densely represented using multi-resolution hash encoding, where time t is used only to model illumination changes; and... For each dynamic object instance, learn the density and color differences corresponding to the 3D coordinates (x, y, z) at different times t.

[0012] Alternatively, the step of generating a synthetic image of the target sensor configuration based on the scene geometry and appearance features in the source sensor configuration data using differentiable rendering technology includes: For each pixel in the target image, a ray is emitted from the center of the target camera toward the pixel. 3D point sampling is performed along the ray within a preset depth range; The color and density of each sampling point are determined based on the 3D coordinates of each sampling point and the observation direction of the entire ray. Along the ray, the color and density of the sampling points are accumulated to determine the final color of the pixel; wherein the accumulated color is calculated as follows: Pixel color = summation (transmittance × density × color), where transmittance represents the cumulative probability that light reaches the corresponding sampling point without being blocked, and sampling points that are closer to the target camera and have higher density contribute more to pixel color.

[0013] Secondly, embodiments of the present invention provide a cross-sensor adaptive 3D object detection system based on neural rendering, comprising: A sample preprocessing unit is used to preprocess the pre-acquired data samples and, in conjunction with the acquisition association information of each data sample, to perform structured storage of the data samples as source sensor configuration data; wherein, the data samples are raw images acquired through a preset camera; The model training unit is used to train the pre-built neural rendering model based on the source sensor configuration data to obtain a cross-sensor adaptive 3D object detection model. A 3D object detection unit is used to perform 3D object detection on the input image to be detected based on the cross-sensor adaptive 3D object detection model. The pre-built neural rendering model adopts a neural radiation field architecture, including static scene modeling and dynamic scene modeling. In the static scene modeling process, all objects in the image are trained under the assumption that the entire scene is absolutely static, and a static 3D scene is learned based on the neural network. In the dynamic scene modeling process, time is introduced as an additional dimension to train the objects in the image, and a static 3D space combined with 1D time 4D spatiotemporal scene is learned based on the neural network.

[0014] Thirdly, embodiments of the present invention provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the steps of the above-described cross-sensor adaptive 3D object detection method based on neural rendering.

[0015] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing instructions that, when executed on an electronic device, cause the electronic device to perform a cross-sensor adaptive 3D object detection method based on neural rendering as described in any of the first aspects.

[0016] According to the present invention, the cross-sensor adaptive 3D object detection method and system based on neural rendering is based on real data-driven cross-sensor adaptation. It accurately captures the imaging characteristics under different sensor configurations through real scene data, overcoming the problem of differences between simulation data and real environment. Furthermore, the present invention adopts a neural radiation field model with an explicit dynamic-static decomposition neural rendering architecture, which can effectively separate the scene into static background and dynamic objects, and process them with different representation methods, ensuring the geometric consistency of dynamic objects under new perspectives.

[0017] Understandably, the system described in the second aspect, the electronic device described in the third aspect, and the computer storage medium described in the fourth aspect are all used to execute the corresponding method in the first aspect provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding method provided above, and will not be repeated here. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1This is a flowchart of a cross-sensor adaptive 3D object detection method based on neural rendering according to an embodiment of the present invention. Figure 2 This is a schematic diagram of a data processing architecture for cross-sensor adaptive 3D object detection based on neural rendering according to an embodiment of the present invention. Figure 3 This is a data processing logic structure diagram of a neural rendering model according to an embodiment of the present invention; Figure 4 This is a block diagram of the logical structure of a cross-sensor adaptive 3D object detection system based on neural rendering according to an embodiment of the present invention. Figure 5 This is a schematic diagram of the logic structure of an electronic device according to an embodiment of the present invention.

[0020] In all the accompanying drawings, the same reference numerals indicate similar or corresponding features or functions. Detailed Implementation

[0021] In the following description, numerous specific details are set forth for illustrative purposes and to provide a thorough understanding of one or more embodiments. However, it will be apparent that these embodiments may also be implemented without these specific details.

[0022] The embodiments of this invention can be applied to electronic devices such as computer systems / servers, which can operate with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations suitable for use with electronic devices such as computer systems / servers include, but are not limited to: personal computer systems, server computer systems, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, networked personal computers, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.

[0023] Computer systems / servers and other electronic devices can be described in the general context of computer-executable instructions executed by the computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers and other electronic devices can be implemented in distributed cloud computing environments, where tasks are executed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0024] To better illustrate the technical solution of the present invention, some of the technical terms involved in the present invention will be briefly explained below.

[0025] Neural Radiance Field (NeRF) is a technique that uses deep learning to reconstruct high-quality 3D scenes from multi-view 2D images. It models a continuous 3D radiance field using multi-layer perceptrons (MLP), uses volume rendering with implicit neural scene representation to predict the color and volume density of any point in the scene in a specific direction, and combines it with volume rendering to generate realistic new perspective images.

[0026] In the following description, numerous specific details are set forth for illustrative purposes and to provide a thorough understanding of one or more embodiments. However, it will be apparent that these embodiments may also be implemented without these specific details.

[0027] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present invention.

[0028] This invention provides a cross-sensor adaptive 3D object detection method and system based on neural rendering. In practical applications, it adopts a modular design, mainly comprising four core components: a data preprocessing module, a neural rendering modeling module, an image synthesis and adaptation module, and a model optimization module. First, synchronous data for multi-sensor configurations is generated based on pre-acquired raw image data. Then, neural rendering technology is used to achieve sensor configuration conversion. Finally, the synthesized data is used to train and evaluate the cross-sensor adaptive 3D object detection model.

[0029] The modules communicate with each other through standardized data interfaces, ensuring the continuity and efficiency of the data processing flow. The system adopts a distributed computing architecture, supporting multi-node parallel processing, which can effectively improve data processing efficiency.

[0030] Figure 1 The flowchart of the cross-sensor adaptive 3D object detection method based on neural rendering provided in this embodiment of the invention is shown. Figure 2 This paper illustrates a data processing architecture for cross-sensor adaptive 3D object detection based on neural rendering, according to an embodiment of the present invention. Figure 3 The data processing logic structure of a neural rendering model according to an embodiment of the present invention is illustrated. (Refer to...) Figures 1 to 3 As shown in the embodiments of the present invention, the cross-sensor adaptive 3D object detection method based on neural rendering provided includes the following steps: S110: Preprocess the pre-acquired data samples and combine them with the acquisition association information of each data sample to store the data samples in a structured manner as source sensor configuration data.

[0031] This step is primarily implemented based on a data preprocessing module, where the collected data samples are raw images acquired through preset cameras. Specifically, as an example, a synchronization clock signal can be generated by a multi-channel synchronization controller. Multiple cameras (e.g., six) receive this synchronization clock signal to trigger image acquisition, using synchronized data from different sensor configurations across multiple vehicle models as data samples, effectively controlling the deviation in the acquisition start time. This hardware synchronization technology ensures millisecond-level time alignment accuracy for the data from the six surround-view cameras and supports a data acquisition frequency of 20Hz. Each image frame acquired by one camera is timestamped, and after image acquisition, a preset timestamp alignment algorithm corrects the transmission delay of the images acquired by the six cameras.

[0032] In one specific embodiment of the present invention, each image frame acquired by the camera is appended with a UTC timestamp based on the GPS-BeiDou dual-mode positioning module (positioning accuracy ≤1m). After acquisition, the transmission delay is corrected by the timestamp alignment algorithm, and finally the time alignment accuracy of the six data channels is ≤1ms, which meets the spatiotemporal consistency requirements of each frame of data at a 20Hz acquisition frequency.

[0033] Then, the preprocessing of the collected data samples can include automatic calibration and optimization processing and data quality control processing.

[0034] The automatic calibration optimization process employs a hybrid calibration strategy that combines offline calibration with online correction to design a hybrid calibration process. By combining the accuracy advantages of the checkerboard calibration method with the adaptability of natural feature point matching, it achieves high-precision estimation of camera intrinsic and extrinsic parameters, thereby correcting camera parameter deviations in the data samples.

[0035] Specifically, as an example, in the offline calibration stage, a 12×12 checkerboard calibration board (square size 50mm×50mm) can be used to acquire 200+ calibration images at 15° intervals within a 3m×3m×2m space. Then, the camera's intrinsic parameters (focal length f_x, f_y, principal point coordinates c_x, c_y, distortion coefficients k1 / k2 / p1 / p2) and extrinsic parameters (rotation matrix R, translation vector T) are calculated using a preset calibration algorithm, achieving a calibration accuracy of 0.1 pixels. In the online correction stage, natural feature points (such as road markings and street light outlines) are extracted from the images in real time, and the SIFT feature matching algorithm is used to compare them with a pre-stored calibration scene feature library. If the matching error is >0.5 pixels, the distortion coefficients in the camera's intrinsic parameters and the rotation matrix in the extrinsic parameters are automatically adjusted to correct calibration deviations caused by vehicle vibration and temperature changes. The correction frequency is synchronized with the acquisition frequency (20Hz).

[0036] Data quality control processing is achieved by establishing a multi-level quality inspection mechanism, including image sharpness detection based on the Laplacian operator, exposure consistency check based on histogram analysis, and color balance processing based on the color correction matrix.

[0037] Specifically, as an example, when performing image sharpness detection, if the variance of the Laplacian operator is less than 1000, the image is determined to be a blurry image, the frame data is automatically discarded, and the camera is triggered to re-acquire the image; when performing exposure consistency checks, if the cosine similarity of the histograms of adjacent frames is less than 0.8, it is determined to be an exposure anomaly, and a histogram equalization algorithm is required for correction. After correction, the similarity must be ≥0.95 to be retained; when performing color equalization processing, if any channel deviates from the standard color chart by more than 5ΔE, the 3A automatic white balance algorithm needs to be called for adjustment, and the color space is normalized based on the color correction matrix (such as the sRGB standard).

[0038] In order to build a complete metadata recording system, after preprocessing the pre-acquired data samples, it is also necessary to combine the acquisition association information of each data sample (such as acquisition timestamp, GPS location, IMU data, weather conditions, light intensity and sensor working status) to store the preprocessed data samples in a structured manner.

[0039] In one specific embodiment of the present invention, a hybrid storage architecture of "time-series database + relational database" is adopted to realize metadata record management. Specifically, the time-series database (InfluxDB) stores high-frequency acquired time-series metadata, including acquisition timestamps, IMU data (acceleration ±16g, angular velocity ±2000° / s), and light intensity (0-100000 lux), indexed by "sensor ID-acquisition time", supporting millisecond-level query response. The relational database (PostgreSQL) stores static metadata, including sensor model, GPS location (latitude and longitude, altitude), weather conditions (sunny / rainy / snowy), and device operating status (temperature, voltage), using JSONB format to store complex structured data, supporting multi-condition combined queries (such as "filtering sensor data of a sunny plain area on May 1, 2024").

[0040] After determining the source sensor configuration data, step S120 can be performed to train the pre-built neural rendering model based on the source sensor configuration data to obtain a cross-sensor adaptive 3D object detection model.

[0041] In one specific embodiment of the present invention, training a pre-constructed neural rendering model based on the source sensor configuration data mainly includes the following three stages: S121: Train the neural rendering model using source sensor configuration data to learn scene geometry and appearance features in the source sensor configuration data; S122: Based on the scene geometry and appearance features in the source sensor configuration data, generate a synthetic image of the target sensor configuration using differentiable rendering technology; S123: The trained neural rendering model is fine-tuned using the synthesized image to obtain a cross-sensor adaptive 3D object detection model that adapts to the characteristics of the new sensor.

[0042] The three stages described above are implemented based on the aforementioned neural rendering modeling module, image synthesis and adaptation module, and model optimization module, respectively. In this invention, based on the neural rendering modeling module, the neural rendering model is trained using source sensor configuration data to learn the scene's geometric and appearance features from the source sensor configuration data. The pre-built neural rendering model employs a Neural Radiation Field (NeRF) architecture. A Neural Radiation Field is a neural network that represents a continuous 3D scene using a simple fully connected neural network (typically an MLP). The input to this neural network is the 3D coordinates (x, y, z) of a sampling point and the 2D viewpoint's viewing direction (θ, φ). The output is the color and density (RGB, σ) of that sampling point, thus forming a "radiation field," that is, inputting any point in 3D space and outputting the physical properties (geometric density and visual appearance) of that point.

[0043] The Neural Radiation Field (NeRF) architecture in this invention employs an "explicit dynamic-static decomposition" architecture that combines static and dynamic scene modeling. In the NeRF static scene modeling, the input is (x, y, z, θ, φ). It assumes the entire scene is absolutely static; all objects in the training image, from buildings to trees, must remain completely still. Any minute changes (such as swaying leaves or pedestrian movement) will cause the model to fail, resulting in "ghosting" blur. To handle the prevalent moving objects (vehicles, pedestrians) in autonomous driving video sequences, time must be introduced as an additional dimension into the model. Therefore, the input for the NeRF dynamic scene modeling is (x, y, z, t, θ, φ). Since the density and color of the same 3D coordinate (x, y, z) can be completely different at different times t, the neural network not only learns a static 3D scene but also a 4D spatiotemporal scene (3D space + 1D time).

[0044] For example, at t=1, the coordinates (10,2,1) might correspond to the front of a car, with high density and a red color. At t=5, the same coordinates (10,2,1) might be an empty road after a car has passed, with zero density and a gray road surface color. By inputting (x,y,z,t) into the network, the network can learn that this location was occupied by a moving car at a certain time period and then became empty again.

[0045] In one embodiment of the invention, a more advanced and structured dynamic modeling method is employed, utilizing time t, i.e., explicit dynamic-static decomposition, with inputs also including (x, y, z, t, θ, φ). Specifically, firstly, 3D bounding box annotations (which themselves carry timestamps, i.e., t) are used to identify whether a point (x, y, z) belongs to a static environment or a dynamic object at time t. Then, for the static background, a dense representation is created using multi-resolution hash encoding, where the density and color are primarily related to the 3D coordinates (x, y, z), with a weak relationship to t (possibly only used to model lighting changes). For dynamic objects, since their density and color strongly depend on the 3D coordinates (x, y, z), the network learns an independent, time-varying representation for each dynamic object instance (such as a specific car).

[0046] After determining the scene geometry and appearance features in the source sensor configuration data, the image synthesis and adaptation module generates a synthetic image of the target sensor configuration using differentiable rendering technology.

[0047] In this embodiment, the goal of NeRF is to synthesize a 2D image viewed from any new perspective. This process is called "volume rendering." Based on the scene geometry and appearance features in the source sensor configuration data, the steps for generating a synthesized image of the target sensor configuration using differentiable rendering technology are as follows: a. Ray casting: For each pixel in the target image, a ray is emitted from the camera center towards that pixel. Specifically, as an example, given the target camera's intrinsic parameters (f_x, f_y, c_x, c_y) and extrinsic parameters (R, T), for each pixel (i, j) of the target image (resolution W × H), calculate its ray direction vector in the camera coordinate system. Then, convert the pixel coordinates (i, j) to normalized device coordinates: (u, v) = ((i - c_x) / f_x, (j - c_y) / f_y), then the ray direction vector: d = R -1 ×(u,v,1) T (R) -1 (The extrinsic rotation matrix is ​​the inverse of the ray origin, with the camera optical center as the starting point: o = -R) -1×T (T is the extrinsic parameter translation vector). The target image here is a learning process. Initially, it is a randomly initialized value, and then it gradually learns the specific pixel values ​​of the target sensor parameter distribution through a loss function.

[0048] b. Sampling along a ray: Along this ray, a series of 3D points are densely collected within a certain depth range.

[0049] c. Query the neural network: Input the (x,y,z) of each sampling point and the 2D observation direction (θ,φ) of the entire ray into the neural network to obtain the color and density (RGB,σ) of each sampling point.

[0050] d. Accumulated Color (Volume Rendering Integral): Along the ray, the colors and densities of these points are summed to calculate the final color of the pixel. Its physical meaning is similar to passing a ray of light through a dense cloud: points with higher density (σ) will block more light. Color (c) is the light emitted by that point itself. The final pixel color is a weighted sum of the colors of all points along this ray, with the weights determined by the density of that point and the cumulative transparency of all points preceding it.

[0051] A simplified mathematical understanding is: pixel color = summation (transmittance * density * color), where "transmittance" represents the cumulative probability that light successfully reaches that point without being completely blocked along the way. Higher density results in lower transmittance and correspondingly higher opacity. Therefore, points closer to the camera and with higher density contribute the most to the final color.

[0052] It should be noted that this invention mainly discusses using NeRF as the core of neural rendering to build a data-driven sensor adaptation pipeline, realizing the conversion of datasets from one camera configuration (such as an SUV) to another configuration (such as a compact car). The core objective is to mitigate the impact of cross-sensor domain on 3D object detection. Therefore, the above calculation method does not describe the sampling point spacing in more detail; a general setting is sufficient here.

[0053] This image synthesis and adaptation module, based on a learned neural network, generates new perspective images under the target sensor configuration by modifying the extrinsic parameters (rotation and translation matrices) and intrinsic parameters (focal length and optical center coordinates) of the rendering camera. The algorithm employs differentiable rendering technology, supports end-to-end optimization training, and achieves high-quality image conversion from the source sensor configuration to the target sensor configuration.

[0054] Finally, the trained neural rendering model is fine-tuned using the generated synthetic images to obtain a cross-sensor adaptive 3D object detection model that adapts to the characteristics of the new sensor. It should be noted that fine-tuning in this step refers to training the 3D object detection model by adding newly generated target sensor data to the model trained on the original sensor data, allowing the 3D object detection model to learn the feature distribution of the target sensor dataset. The purpose of fine-tuning is to optimize the parameters of the entire network; no new loss function needs to be added, and only a small learning rate is required for the model to learn the features of the target sensor data.

[0055] During sensor adaptation, a closed-loop adaptation process of "parameter configuration - image synthesis - quality verification" can be adopted. Specifically, in the parameter configuration stage, a sensor parameter database is established to store camera configuration parameters for different vehicle models (SUVs / sedans / trucks). Users can select the target vehicle model through a web interface, and the system automatically calls the corresponding parameters. In the image synthesis stage, the intrinsic and extrinsic parameters of the rendering camera are modified according to the target sensor parameters to generate an image under the target configuration. Then, differentiable rendering technology is used to backpropagate the difference (L1 loss) between the synthesized image and the source sensor image to the neural rendering model, iteratively optimizing the model parameters (learning rate 1e-4, 1000 iterations) to match the imaging characteristics of the synthesized image with those of the target sensor (such as distortion patterns and viewing angle range). In the quality verification stage, the adapted target sensor can be quality verified using preset quality verification indicators (such as structural similarity indicators, peak signal-to-noise ratio, and similarity of learned perceptual image patches). If any indicator fails to meet the standard, the number of sampling points can be automatically adjusted (e.g., increased to 128) or the model parameters can be optimized, and the image can be resynthesized until it passes the test, ensuring that only images that pass all quality checks are used for subsequent model training.

[0056] The method described above by the present invention can be performed by an apparatus, which can be implemented by software and / or hardware.

[0057] In summary, the cross-sensor adaptive 3D object detection method based on neural rendering provided by this invention accurately captures the imaging characteristics under different sensor configurations using real-world scene data, overcoming the discrepancy between simulation data and the real environment. It achieves sensor configuration adaptation through neural rendering technology, designs a neural radiation field model with an explicit dynamic-static decomposition mechanism, effectively separating the scene into static background and dynamic objects, and processes them using different representation methods to ensure the geometric consistency of dynamic objects under new perspectives. Furthermore, it employs multi-resolution hash encoding technology to improve the training and rendering efficiency of the neural radiation field, significantly reducing computational overhead while maintaining rendering quality. Applying this invention can bring the following beneficial effects to the field of autonomous driving perception: 1. Significantly improves detection accuracy: It greatly improves the performance of 3D detection models in cross-sensor scenarios and effectively solves the problem of performance degradation of models under new sensor configurations; 2. Establishment of dedicated evaluation benchmarks: The developed dedicated dataset is the first to achieve isolation research on cross-sensor domain gaps, providing a reliable experimental platform for related research; 3. Reduce data costs: By using neural rendering technology to achieve efficient reuse of data across vehicle models, data preparation costs are significantly reduced and the system deployment process is accelerated; 4. Enhanced model adaptability: It exhibits good adaptability to different types of detectors, significantly improving the model's generalization ability; 5. Guaranteed rendering quality: Outperforms existing methods on multiple quality evaluation metrics, ensuring the reliability of synthetic data in practical applications; 6. Achieve system compatibility: Supports flexible sensor parameter adjustment and can be seamlessly integrated with existing autonomous driving perception systems.

[0058] Figure 4 This is a block diagram of the cross-sensor adaptive 3D object detection system based on neural rendering according to the present invention. The cross-sensor adaptive 3D object detection system 400 based on neural rendering provided by the present invention can be installed in an electronic device. Depending on the functions implemented, such as... Figure 4 As shown, the cross-sensor adaptive 3D object detection system 400 based on neural rendering may include: a sample preprocessing unit 410, a model training unit 420, and a 3D object detection unit 430. The unit described herein may also be referred to as a module, which mainly refers to a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, and are stored in the memory of the electronic device.

[0059] In this embodiment, the functions of each module / unit are as follows: The sample preprocessing unit 410 is used to preprocess the pre-acquired data samples and combine the acquisition association information of each data sample to perform structured storage of the data samples as source sensor configuration data; wherein the data samples are the original images acquired by a preset camera; The model training unit 420 is used to train the pre-built neural rendering model based on the source sensor configuration data to obtain a cross-sensor adaptive 3D object detection model. The 3D object detection unit 430 is used to perform 3D object detection on the input image to be detected based on the cross-sensor adaptive 3D object detection model. The pre-built neural rendering model adopts a neural radiation field architecture, including static scene modeling and dynamic scene modeling. In the static scene modeling process, all objects in the image are trained under the assumption that the entire scene is absolutely static, and a static 3D scene is learned based on the neural network. In the dynamic scene modeling process, time is introduced as an additional dimension to train the objects in the image, and a static 3D space combined with 1D time 4D spatiotemporal scene is learned based on the neural network.

[0060] It should be noted that the embodiments of the cross-sensor adaptive 3D object detection system based on neural rendering and the embodiments of the cross-sensor adaptive 3D object detection method based on neural rendering can be used for reference, and will not be elaborated here.

[0061] Figure 5 This is a schematic diagram of the electronic device that implements the cross-sensor adaptive 3D object detection system based on neural rendering according to the present invention.

[0062] like Figure 5 As shown, the electronic device 1 may include a processor 10, a memory 11 and a bus, and may also include a computer program stored in the memory 11 and executable on the processor 10, such as a neural rendering-based cross-sensor adaptive 3D object detection program 12.

[0063] The memory 11 includes at least one type of readable storage medium, including flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of the electronic device 1, such as the portable hard drive of the electronic device 1. In other embodiments, the memory 11 can also be an external storage device of the electronic device 1. Furthermore, the memory 11 can include both internal and external storage units of the electronic device 1. The memory 11 can be used not only to store application software and various types of data installed on the electronic device 1, such as program code, but also to temporarily store data that has been output or will be output.

[0064] In some embodiments, the processor 10 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules stored in the memory 11 (e.g., a cross-sensor adaptive 3D object detection program based on neural rendering) and calls data stored in the memory 11 to perform various functions of the electronic device 1 and process data.

[0065] The bus can be a Peripheral Component Interconnect Standard (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. The bus is configured to enable communication between the memory 11 and at least one processor 10, etc.

[0066] Figure 5 Only electronic devices with components are shown; it will be understood by those skilled in the art that... Figure 5 The structure shown does not constitute a limitation on the electronic device 1, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0067] For example, although not shown, the electronic device 1 may also include a power supply (such as a battery) to power the various components. Preferably, the power supply can be logically connected to the at least one processor 10 through a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device 1 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0068] Furthermore, the electronic device 1 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, a Bluetooth interface, etc.), which is typically used to establish communication connections between the electronic device 1 and other electronic devices.

[0069] Optionally, the electronic device 1 may further include a user interface, which may be a display, an input unit (such as a keyboard), or, optionally, a standard wired interface or a wireless interface. In some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an organic light-emitting diode touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the electronic device 1 and to display a visual user interface.

[0070] It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in the scope of the patent application.

[0071] The memory 11 in the electronic device 1 stores a combination of multiple instructions for a cross-sensor adaptive 3D object detection program 12 based on neural rendering. When run in the processor 10, it can realize a cross-sensor adaptive 3D object detection method based on neural rendering.

[0072] Furthermore, if the modules / units integrated in the electronic device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory.

[0073] In the several embodiments provided by this invention, it should be understood that the disclosed devices, systems, and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0074] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A cross-sensor adaptive 3D object detection method based on neural rendering, applied to electronic devices, characterized in that, include: The pre-acquired data samples are preprocessed, and the data samples are structured and stored in conjunction with the acquisition association information of each data sample as source sensor configuration data; wherein, the data samples are raw images acquired through a preset camera; The pre-built neural rendering model is trained based on the source sensor configuration data to obtain a cross-sensor adaptive 3D object detection model. The cross-sensor adaptive 3D object detection model is used to perform 3D object detection on the input image to be detected. The pre-built neural rendering model adopts a neural radiation field architecture, including static scene modeling and dynamic scene modeling. In the static scene modeling process, all objects in the image are trained under the assumption that the entire scene is absolutely static, and a static 3D scene is learned based on the neural network. In the dynamic scene modeling process, time is introduced as an additional dimension to train the objects in the image, and a static 3D space combined with 1D time 4D spatiotemporal scene is learned based on the neural network.

2. The cross-sensor adaptive 3D object detection method based on neural rendering according to claim 1, characterized in that, Training a pre-built neural rendering model based on the source sensor configuration data includes: The neural rendering model is trained using source sensor configuration data to learn scene geometry and appearance features from the source sensor configuration data; Based on the scene geometry and appearance features in the source sensor configuration data, a synthetic image of the target sensor configuration is generated using differentiable rendering technology; The trained neural rendering model is fine-tuned using the synthesized images to obtain a cross-sensor adaptive 3D object detection model that adapts to the characteristics of the new sensor.

3. The cross-sensor adaptive 3D object detection method based on neural rendering according to claim 2, characterized in that, During the data sample acquisition process, a synchronization clock signal is generated by a multi-channel synchronization controller. Six cameras receive this synchronization clock signal, triggering image acquisition of synchronized data from different sensor configurations across multiple vehicle models, which are then used as data samples. Each image frame captured by the camera is timestamped, and the transmission delay of the images captured by the six cameras is corrected by a preset timestamp alignment algorithm after image acquisition.

4. The cross-sensor adaptive 3D object detection method based on neural rendering according to any one of claims 1-3, characterized in that, The preprocessing of the pre-acquired data samples includes: The camera parameter deviations in the data samples are corrected using a hybrid calibration strategy that combines offline calibration and online correction. The pre-acquired data samples are processed using image sharpness detection based on the Laplacian operator, exposure consistency check based on histogram analysis, and color balance processing based on the color correction matrix.

5. The cross-sensor adaptive 3D object detection method based on neural rendering according to any one of claims 1-3, characterized in that, The acquisition association information for each data sample includes the acquisition timestamp, GPS location, IMU data, weather conditions, light intensity, and sensor operating status information for each data sample.

6. The cross-sensor adaptive 3D object detection method based on neural rendering according to claim 3, characterized in that, During the training of the neural rendering model using source sensor configuration data, 3D bounding box annotations with timestamps are used to identify whether a point (x, y, z) belongs to a static environment or a dynamic object at time t. The static environment is densely represented using multi-resolution hashing, where the density and color are related to 3D coordinates (x, y, z), and the multi-resolution hashing of the static environment uses time t only to model illumination variations; and, For each dynamic object instance, learn the density and color differences corresponding to the 3D coordinates (x, y, z) at different times t.

7. The cross-sensor adaptive 3D object detection method based on neural rendering according to claim 6, characterized in that, The step of generating a synthetic image of the target sensor configuration based on scene geometry and appearance features in the source sensor configuration data using differentiable rendering technology includes: For each pixel in the target image, a ray is emitted from the center of the target camera toward the pixel; 3D point sampling is performed on the target image along the ray within a preset depth range; The color and density of each sampling point are determined based on the 3D coordinates of each sampling point and the observation direction of the entire ray. Along the ray, the color and density of the sampling points are accumulated to determine the final color of the pixel; wherein the accumulated color is calculated as follows: Pixel color = summation (transmittance × density × color), where transmittance represents the cumulative probability that light reaches the corresponding sampling point without being blocked, and sampling points that are closer to the target camera and have higher density contribute more to pixel color.

8. A cross-sensor adaptive 3D object detection system based on neural rendering, characterized in that, include: A sample preprocessing unit is used to preprocess the pre-acquired data samples and, in conjunction with the acquisition association information of each data sample, to perform structured storage of the data samples as source sensor configuration data; wherein, the data samples are raw images acquired through a preset camera; The model training unit is used to train the pre-built neural rendering model based on the source sensor configuration data to obtain a cross-sensor adaptive 3D object detection model. A 3D object detection unit is used to perform 3D object detection on the input image to be detected based on the cross-sensor adaptive 3D object detection model. The pre-built neural rendering model adopts a neural radiation field architecture, including static scene modeling and dynamic scene modeling. In the static scene modeling process, all objects in the image are trained under the assumption that the entire scene is absolutely static, and a static 3D scene is learned based on the neural network. In the dynamic scene modeling process, time is introduced as an additional dimension to train the objects in the image, and a static 3D space combined with 1D time 4D spatiotemporal scene is learned based on the neural network.

9. An electronic device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, when the processor executes the computer program, it implements the steps of the cross-sensor adaptive 3D object detection method based on neural rendering as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing instructions, characterized in that, When the instructions are executed on an electronic device, the electronic device performs the cross-sensor adaptive 3D object detection method based on neural rendering as described in any one of claims 1-7.

Images