Three-dimensional driving recording system and method, electronic equipment and storage medium

Abstract

The invention relates to a three-dimensional driving recording system and method, electronic equipment and a storage medium, the system comprises a data acquisition module and at least one group of NeRF networks, and the NeRF networks comprise a TensoRF1 network, a TensoRF2 network and a Hexplane network; the data acquisition module is used for acquiring and storing a video image of a forward view angle with the vehicle as the center when the vehicle runs; the TensoRF1 network and the TensoRF2 network perform joint training on the input static streetscape images based on the NeRF thought to obtain camera pose and internal reference data corresponding to each static streetscape image; and inputting the video images and the corresponding camera poses and internal reference data into a HexPlane network for training to obtain dynamic implicit three-dimensional reconstruction representation of the whole scene, thereby solving the problem that the driving recording system is high in calculated amount and needs more images in a three-dimensional reconstruction mode.

Description

Technical Field

[0001] The present invention relates to the field of vehicle driving record, and in particular to a three-dimensional vehicle driving record system, method, electronic equipment and storage medium. Background Art

[0002] The car driving recorder, also known as the car "black box", is a digital, fully automatic, intelligent on-board monitoring device controlled by a microcomputer, which applies the design concept of the aircraft black box to the car and transfers aviation military technology to civilian use. It integrates mechanics, electronics, and microcomputers, and is suitable for preventing accidents, monitoring violations, and scientific management, and provides a fair, accurate, and scientific basis for accident analysis.

[0003] Car driving recorders have multiple functions of recording timely driving data, and can play an irreplaceable role in road traffic management. Record traffic violations and prevent accidents. In view of the main causes of traffic accidents such as speeding and fatigue driving, car driving recorders automatically, continuously and truthfully record the driver's driving speed and continuous driving conditions for hundreds of days, which is equivalent to having a traffic policeman on the car for a long time, and conduct comprehensive, long-term and uninterrupted timely monitoring of the driver's safe driving conditions. These data can be used to establish driver safety driving files, provide scientific basis for assessment, evaluation, rewards and penalties, and are also conducive to cultivating drivers' good habits of driving safe and medium-speed cars, so as to reduce violations and prevent accidents. Scientifically analyze and handle road traffic accidents. Car driving recorders can record the speed, time, deceleration and braking conditions and driving conditions of vehicles before the accident, and can provide quantitative and accurate data for the handling of traffic accidents. The data is used to cooperate with the on-site investigation and evidence collection of traffic accidents, and use scientific and quantitative data to fairly judge the cause and responsibility of the accident, which provides a scientific basis for improving the work efficiency of traffic police and effectively protecting the legitimate rights and interests of car owners and drivers.

[0004] At present, the most popular form of car dashcams is to record the situation in front of the car in real time, but the popular three-dimensional reconstruction method requires a large number of images while having a high amount of calculation, and it is difficult to obtain a reconstruction result with photo-level clarity. Summary of the invention

[0005] The present invention aims at the technical problems existing in the prior art and provides a three-dimensional driving recorder system, method, electronic device and storage medium to solve the problem that the three-dimensional driving recorder system in the prior art has high calculation amount and requires more images in the three-dimensional reconstruction method.

[0006] According to a first aspect of the present invention, a three-dimensional driving recorder system is provided, comprising:

[0007] A data acquisition module and at least one set of NeRF networks, wherein the NeRF networks include: a TensoRF1 network, a TensoRF2 network, and a HexPlane network;

[0008] The data acquisition module is used to acquire and save video images from a forward perspective centered on the vehicle when the vehicle is traveling;

[0009] The TensoRF1 network and the TensoRF2 network jointly train the input static street view images based on the idea of ​​NeRF to obtain the camera pose and internal reference data corresponding to each static street view image;

[0010] The video image and its corresponding camera pose and internal reference data are input into the HexPlane network for training to obtain a dynamic implicit three-dimensional reconstruction representation of the entire scene.

[0011] Based on the above technical solution, the present invention can also make the following improvements.

[0012] Optionally, the data acquisition module includes: a data storage terminal and a camera directly in front of the vehicle;

[0013] The camera collects the video from the perspective directly in front of the vehicle in real time and saves it in the data storage terminal.

[0014] Optionally, the TensoRF1 network is used to reconstruct the pure street view in the static street view image.

[0015] The TensoRF2 network is used to reconstruct a static scene including a static vehicle in the static street view image.

[0016] Optionally, the process of training the NeRF network includes: performing block training on the entire scene according to relative postures to obtain a three-dimensional reconstruction model of a static street scene:

[0017] The distance from the pose of each static street view image to the current NeRF network 3D reconstruction center is calculated. If the distance exceeds a set threshold, a new NeRF network is created in real time, and the image is divided into the training data corresponding to the new NeRF network.

[0018] Optionally, after the NeRF network is trained, a Marching Cube algorithm is used on the NeRF blocks to obtain a three-dimensional texture file of a static scene.

[0019] Optionally, the system further includes a reconstruction module, which reconstructs a corresponding image according to the viewing angle requirement information input by the user and the output of the improved NeRF network module.

[0020] According to a second aspect of the present invention, there is provided a three-dimensional driving recorder system, comprising:

[0021] Step 1, collecting and saving a video image of a forward perspective centered on the vehicle when the vehicle is traveling;

[0022] Step 2, using the TensoRF1 network and the TensoRF2 network to jointly train the input static street view image based on the idea of ​​NeRF to obtain the camera pose and internal reference data corresponding to each static street view image;

[0023] Step 3: Input the video image and its corresponding camera pose and internal reference data into the HexPlane network for training to obtain a dynamic implicit 3D reconstruction representation of the entire scene.

[0024] According to a third aspect of the present invention, there is provided an electronic device, comprising a memory and a processor, wherein the processor is configured to implement the steps of a three-dimensional vehicle driving recording method when executing a computer management program stored in the memory.

[0025] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium on which a computer management program is stored, and when the computer management program is executed by a processor, the steps of the three-dimensional driving recording method are implemented.

[0026] The present invention provides a three-dimensional driving record system, method, electronic device and storage medium, in which the neural network involved is simpler (the system includes three NeRF networks) and has lower requirements for input data. It only needs to input image data information of a single perspective (directly in front of the vehicle). The camera posture and internal parameters can be obtained by training two TensoRF networks in the static street scene model construction unit and are passed as input to the dynamic object model construction unit. The time corresponding to each image data is automatically generated according to the sequence number of the image frame under the premise that the time interval between taking the pictures is equal. Implicit three-dimensional reconstruction based on NeRF is adopted, which can maximize the use of the clarity of the original data. It can make the clarity of the rendering result close to the original data through continuous iteration. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A schematic structural diagram of an embodiment of a three-dimensional vehicle driving record system provided by the present invention;

[0028] Figure 2 A schematic diagram of the hardware structure of a possible electronic device provided by the present invention;

[0029] Figure 3 A schematic diagram of the hardware structure of a possible computer-readable storage medium provided by the present invention. DETAILED DESCRIPTION

[0030] The principles and features of the present invention are described below in conjunction with the accompanying drawings. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0031] Figure 1 A schematic diagram of a three-dimensional vehicle driving recorder system according to an embodiment of the present invention is shown in FIG. Figure 1 As shown, the system includes: a data acquisition module and at least one group of NeRF networks, and the NeRF networks include: TensoRF1 network, TensoRF2 network and HexPlane network.

[0032] The data acquisition module is used to collect and save video images from a forward perspective centered on the vehicle when the vehicle is traveling.

[0033] Based on the idea of ​​NeRF, the TensoRF1 network and the TensoRF2 network jointly train the input static street scene images to obtain the camera pose and intrinsic parameter data corresponding to each static street scene image.

[0034] The video image and its corresponding camera pose and intrinsic parameter data are input into the HexPlane network for training to obtain a dynamic implicit 3D reconstruction representation of the entire scene.

[0035] The present invention provides a three-dimensional vehicle driving recorder system, which solves the problem that the three-dimensional vehicle driving recorder system in the prior art has a high computational complexity and requires a large number of images in a three-dimensional reconstruction method.

[0036] Example 1

[0037] Embodiment 1 of the present invention is an embodiment of a three-dimensional vehicle driving recorder system provided by the present invention, Figure 1 It can be seen that an embodiment of the system includes: and at least one group of NeRF networks, the NeRF networks include: TensoRF1 network, TensoRF2 network and HexPlane network.

[0038] The data acquisition module is used to collect and save video images from a forward perspective centered on the vehicle when the vehicle is traveling.

[0039] In a possible implementation manner, the data acquisition module includes: a data storage terminal and a camera directly in front of the vehicle.

[0040] The camera collects the video from the front of the vehicle in real time and saves it to the data storage terminal.

[0041] Based on the idea of ​​NeRF, the TensoRF1 network and the TensoRF2 network jointly train the input static street scene images to obtain the camera pose and intrinsic parameter data corresponding to each static street scene image.

[0042] In a possible implementation manner, the TensoRF1 network is used to reconstruct pure street scenes in static street view images, and the TensoRF2 network is used to reconstruct static scenes containing static vehicles in static street view images.

[0043] In the specific implementation, the TensoRF1 network and the TensoRF2 network use the semantic segmentation algorithm to process the video images and remove the objects in motion in the scene of the video images. The masks used by the TensoRF1 network and the TensoRF2 network for error calculation are different. The TensoRF1 network directly uses the semantic segmentation network to remove all objects that may be dynamic (corresponding to the purely static street scene reconstruction), while the TensoRF2 network uses optical flow estimation based on the mask of the TensoRF1 network, which will include objects such as vehicles that were originally static.

[0044] In a possible implementation manner, the process of training the NeRF network includes: performing block training on the entire scene according to relative posture to obtain a three-dimensional reconstruction model of a static street scene:

[0045] The distance from the pose of each static street view image to the current NeRF network 3D reconstruction center is calculated. If the distance exceeds the set threshold, a new NeRF network is created in real time, and the image is divided into the training data corresponding to the new NeRF network.

[0046] After training, the 3D texture file of the static scene can be obtained by using the Marching Cube algorithm on a specific NeRF block.

[0047] In a possible implementation manner, after the NeRF network is trained, the three-dimensional texture file of the static scene is obtained by using the MarchingCube algorithm on the NeRF blocks.

[0048] In the specific implementation, after the static network (TensoRF1, TensoRF2) is trained, the saved weight file is used to perform the Marching Cube algorithm to obtain the texture. If Marching Cube is used for TensoRF1, a three-dimensional file of a purely static street scene is obtained. If it is used for TensoRF2, a three-dimensional file containing static vehicles and other objects is obtained.

[0049] The video image and its corresponding camera pose and intrinsic parameter data are input into the HexPlane network for training to obtain a dynamic implicit 3D reconstruction representation of the entire scene.

[0050] In a possible implementation manner, the system further includes a reconstruction module, which reconstructs a corresponding image according to the viewing angle requirement information input by the user and the output of the improved NeRF network module.

[0051] Example 2

[0052] Embodiment 2 of the present invention is an embodiment of a three-dimensional driving recording method provided by the present invention, combined with Figure 1 It can be seen that the embodiment of the recording method includes:

[0053] Step 1, collecting and saving a video image of a forward perspective centered on the vehicle when the vehicle is traveling;

[0054] Step 2: Process the video image to obtain a static street view image after removing moving objects. Use the TensoRF1 network and the TensoRF2 network based on the idea of ​​NeRF to jointly train the input static street view image to obtain the camera pose and internal reference data corresponding to each static street view image.

[0055] Step 3: Input the video image and its corresponding camera pose and internal parameter data into the HexPlane network for training to obtain a dynamic implicit 3D reconstruction representation of the entire scene.

[0056] It can be understood that the three-dimensional driving recording method provided by the present invention corresponds to the three-dimensional driving recording system provided by the aforementioned embodiments. The relevant technical features of the three-dimensional driving recording method can refer to the relevant technical features of the three-dimensional driving recording system, which will not be repeated here.

[0057] See also Figure 2 , Figure 2 Schematic diagram of an electronic device provided by an embodiment of the present invention. Figure 2 As shown, an embodiment of the present invention provides an electronic device, including a memory 1310, a processor 1320, and a computer program 1311 stored in the memory 1310 and executable on the processor 1320. When the processor 1320 executes the computer program 1311, the following steps are implemented: collecting and saving video images of a forward perspective centered on the vehicle when the vehicle is traveling; using the TensoRF1 network and the TensoRF2 network to jointly train the input static street scene images based on the idea of ​​NeRF to obtain the camera pose and internal parameter data corresponding to each static street scene image; inputting the video image and its corresponding camera pose and internal parameter data into the HexPlane network for training to obtain a dynamic implicit three-dimensional reconstruction representation of the overall scene.

[0058] See also Figure 3 , Figure 3 Schematic diagram of an embodiment of a computer-readable storage medium provided by the present invention. Figure 3As shown, this embodiment provides a computer-readable storage medium 1400, on which a computer program 1411 is stored. When the computer program 1411 is executed by a processor, the following steps are implemented: collecting and saving video images of a forward perspective centered on the vehicle when the vehicle is traveling; using the TensoRF1 network and the TensoRF2 network to jointly train the input static street scene images based on the idea of ​​NeRF to obtain the camera pose and intrinsic parameter data corresponding to each static street scene image; inputting the video image and its corresponding camera pose and intrinsic parameter data into the HexPlane network for training to obtain a dynamic implicit three-dimensional reconstruction representation of the overall scene.

[0059] The embodiments of the present invention provide a three-dimensional driving record system, method, electronic device and storage medium. The neural network involved is simpler (the system includes three NeRF networks) and has lower requirements for input data. It only needs to input image data information of a single perspective (directly in front of the vehicle). The camera pose and internal parameters can be obtained by training two TensoRF networks in the static street scene model construction unit and passed as input to the dynamic object model construction unit. The time corresponding to each image data is automatically generated according to the sequence number of the image frame under the premise that the time interval between taking pictures is equal. Implicit three-dimensional reconstruction based on NeRF is adopted, which can maximize the use of the clarity of the original data. It can make the clarity of the rendering result close to the original data through continuous iteration.

[0060] It should be noted that in the above embodiments, the description of each embodiment has its own emphasis, and for parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0061] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0062] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0063] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0064] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0065] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0066] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A three-dimensional driving recorder system, characterized in that: The system includes: a data acquisition module and at least one set of NeRF networks, wherein the NeRF networks include: a TensoRF1 network, a TensoRF2 network and a HexPlane network; The data acquisition module is used to acquire and save video images from a forward perspective centered on the vehicle when the vehicle is traveling; The TensoRF1 network and the TensoRF2 network jointly train the input static street view images based on the idea of ​​NeRF to obtain the camera pose and internal reference data corresponding to each static street view image; The video image and its corresponding camera pose and internal reference data are input into the HexPlane network for training to obtain a dynamic implicit three-dimensional reconstruction representation of the entire scene.

2. The system according to claim 1, characterized in that The data acquisition module includes: a data storage terminal and a camera directly in front of the vehicle; The camera collects the video from the perspective directly in front of the vehicle in real time and saves it in the data storage terminal.

3. The system according to claim 1, characterized in that The TensoRF1 network is used to reconstruct the pure street scene in the static street scene image; The TensoRF2 network is used to reconstruct a static scene including a static vehicle in the static street view image.

4. The system according to claim 1, characterized in that The training process of the NeRF network includes: training the whole scene in blocks according to relative positions to obtain a 3D reconstruction model of a static street scene: The distance from the pose of each static street view image to the current NeRF network 3D reconstruction center is calculated. If the distance exceeds a set threshold, a new NeRF network is created in real time, and the image is divided into the training data corresponding to the new NeRF network.

5. The system according to claim 1, characterized in that After the NeRF network is trained, the three-dimensional texture file of the static scene is obtained by using the Marching Cube algorithm on the NeRF blocks.

6. The system according to claim 1, characterized in that The system also includes a reconstruction module, which reconstructs a corresponding image according to the viewing angle requirement information input by the user and the output of the improved NeRF network module.

7. A three-dimensional driving recording method, characterized in that: The method comprises: Step 1, collecting and saving a video image of a forward perspective centered on the vehicle when the vehicle is traveling; Step 2, using the TensoRF1 network and the TensoRF2 network to jointly train the input static street view image based on the idea of ​​NeRF to obtain the camera pose and internal reference data corresponding to each static street view image; Step 3: Input the video image and its corresponding camera pose and internal reference data into the HexPlane network for training to obtain a dynamic implicit 3D reconstruction representation of the entire scene.

8. An electronic device, characterized in that: It comprises a memory and a processor, wherein the processor is used to implement the steps of the three-dimensional driving recording method as claimed in claim 7 when executing the computer management program stored in the memory.

9. A computer-readable storage medium, characterized in that: A computer management program is stored thereon, and when the computer management program is executed by the processor, the steps of the three-dimensional driving recording method as claimed in claim 7 are implemented.

Images