Method and apparatus for motion object reconstruction based on event data augmentation

Abstract

The application relates to a moving object reconstruction method and device based on event data enhancement, wherein the method comprises the following steps: collecting moving video data containing a moving object; generating a high-frame-rate reconstruction image sequence meeting a first preset frame rate condition based on the moving video data; inputting a single-view depth map into a preset point cloud generation network to generate a single-view dense point cloud; inputting the single-view dense point cloud under each view into a preset multi-view point cloud fusion network to obtain a fused dense point cloud; processing the fused dense point cloud obtained at each moment to render a three-dimensional motion trajectory of the object. Thus, the problems in the prior art that it is difficult to obtain an accurate motion trajectory of a high-speed moving object through a detection or segmentation algorithm, that the tracking capability for the high-speed moving object is hindered due to the fact that a traditional camera collects data based on a fixed frame rate, and that the fine level of the moving object reconstruction is reduced are solved.

Description

Technical Field

[0001] This application relates to the fields of computer vision and moving object reconstruction technology, and in particular to a moving object reconstruction method and apparatus based on event data augmentation. Background Technology

[0002] 3D reconstruction is a research area in computer vision and graphics, which involves reconstructing 3D scenes from images or videos captured by monocular or binocular cameras. It has broad application potential in metaverse, virtual reality, and human-computer interaction. Although there are many existing results on the reconstruction of objects or scenes, there is still a lack of effective methods for the reconstruction of moving objects.

[0003] In related technologies, traditional cameras can be used to reconstruct moving objects at low frame rates through detection or segmentation algorithms, or different sensors can be combined for data fusion to obtain high-quality perception information, including object detection and tracking.

[0004] However, in related technologies, it is difficult to obtain accurate motion trajectories of high-speed moving objects through detection or segmentation algorithms. Since traditional cameras are based on a fixed frame rate when collecting data, this hinders the ability to track high-speed moving objects and reduces the level of refinement in the reconstruction of moving objects, which urgently needs to be improved. Summary of the Invention

[0005] This application provides a method and apparatus for reconstructing moving objects based on event data augmentation, in order to solve the problems in related technologies, such as the difficulty in obtaining accurate motion trajectories of high-speed moving objects through detection or segmentation algorithms, and the fact that traditional cameras are based on a fixed frame rate when collecting data, which hinders their ability to track high-speed motion and reduces the level of precision in moving object reconstruction.

[0006] The first aspect of this application provides a method for reconstructing moving objects based on event data augmentation, comprising the following steps: acquiring motion video data containing moving objects; generating a high frame rate reconstructed image sequence that satisfies a first preset frame rate condition based on the motion video data; generating a single-view depth map based on the corresponding event intermediate representation for the high frame rate reconstructed image sequence under each viewpoint; inputting the single-view depth map into a preset point cloud generation network to generate a single-view dense point cloud; inputting the single-view dense point cloud under each viewpoint into a preset multi-view point cloud fusion network to obtain a fused dense point cloud; and processing the fused dense point cloud obtained at each moment to render the three-dimensional motion trajectory of the object.

[0007] Optionally, in one embodiment of this application, generating a high frame rate reconstructed image sequence that satisfies a preset frame rate condition based on the motion video data includes: preprocessing the motion video data to obtain a low frame rate image sequence and an event stream that satisfy a second preset frame rate condition; inputting the low frame rate image sequence and the event stream into a preset video interpolation network to generate a high frame rate reconstructed image sequence that satisfies the preset frame rate condition.

[0008] Optionally, in one embodiment of this application, the event point set of the event stream includes event point coordinates, event point timestamps, and event point polarities.

[0009] Optionally, in one embodiment of this application, the step of inputting the low frame rate image sequence and the event stream into a preset video frame interpolation network to generate a high frame rate reconstructed image sequence that satisfies the preset frame rate condition includes: accumulating the event stream forward and backward according to each adjacent image frame in the low frame rate image sequence to obtain a forward event intermediate representation and a backward event intermediate representation; feeding the forward event intermediate representation and the backward event intermediate representation into a preset optical flow prediction network to obtain a forward optical flow and a backward optical flow; for each adjacent image frame in the low frame rate image sequence, splitting the forward optical flow and the backward optical flow according to a linear ratio, adding the previous moment's image frame to the split forward optical flow to obtain a forward reconstructed image frame, and subtracting the next moment's image frame from the split backward optical flow to obtain a backward reconstructed image frame; inputting the forward reconstructed image frame and the backward reconstructed image frame into a preset bidirectional image synthesis layer to obtain a reconstructed image, thereby generating the high frame rate reconstructed image sequence.

[0010] Optionally, in one embodiment of this application, generating a single-view depth map based on the corresponding event intermediate representation for the high frame rate reconstructed image sequence under each viewpoint includes: traversing the high frame rate reconstructed image sequence under each viewpoint, accumulating the event stream according to the frame rate to obtain the event intermediate representation under each frame rate; traversing each image frame in the high frame rate reconstructed image sequence, inputting the image frame and the corresponding event intermediate representation into a depth map prediction network to generate the single-view depth map.

[0011] Optionally, in one embodiment of this application, the step of inputting the single-view depth map into a preset point cloud generation network to generate a single-view dense point cloud includes: traversing each viewpoint, inputting the single-view depth map and the corresponding event intermediate representation into the point cloud generation network to obtain a single-view sparse point cloud; traversing each viewpoint, performing upsampling and fine-tuning operations on the single-view sparse point cloud to obtain the single-view dense point cloud.

[0012] Optionally, in one embodiment of this application, the step of processing the fused dense point cloud obtained at each moment to render the three-dimensional motion trajectory of the object includes: stitching the fused dense point cloud obtained at each moment to generate a continuous object point cloud motion trajectory; and converting the object point cloud motion trajectory into a mesh model to obtain the three-dimensional motion trajectory of the object.

[0013] A second aspect of this application provides a motion object reconstruction device based on event data augmentation, comprising: an acquisition module for acquiring motion video data containing a moving object; a first generation module for generating a high frame rate reconstructed image sequence that satisfies a first preset frame rate condition based on the motion video data; a second generation module for generating a single-view depth map based on the corresponding event intermediate representation for the high frame rate reconstructed image sequence at each viewpoint; a third generation module for inputting the single-view depth map into a preset point cloud generation network to generate a single-view dense point cloud; a fusion module for inputting the single-view dense point cloud at each viewpoint into a preset multi-view point cloud fusion network to obtain a fused dense point cloud; and a reconstruction module for processing the fused dense point cloud obtained at each moment to render a three-dimensional motion trajectory of the object.

[0014] Optionally, in one embodiment of this application, the first generation module includes: a first acquisition unit, configured to preprocess the motion video data to obtain a low frame rate image sequence and an event stream that satisfy a second preset frame rate condition; and a first generation unit, configured to input the low frame rate image sequence and the event stream into a preset video interpolation network to generate a high frame rate reconstructed image sequence that satisfies the preset frame rate condition.

[0015] Optionally, in one embodiment of this application, the event point set of the event stream includes event point coordinates, event point timestamps, and event point polarities.

[0016] Optionally, in one embodiment of this application, the first generation unit includes: a first acquisition subunit, configured to accumulate the event stream in a forward and reverse manner according to each adjacent image frame in the low frame rate image sequence, respectively, to obtain a forward event intermediate representation and a reverse event intermediate representation; a second acquisition subunit, configured to send the forward event intermediate representation and the reverse event intermediate representation into a preset optical flow prediction network, respectively, to obtain a forward optical flow and a reverse optical flow; a third acquisition subunit, configured to split the forward optical flow and the reverse optical flow according to a linear ratio for each adjacent image frame in the low frame rate image sequence, add the previous moment image frame to the split forward optical flow to obtain a forward reconstructed image frame, and subtract the next moment image frame from the split reverse optical flow to obtain a reverse reconstructed image frame; and a generation subunit, configured to input the forward reconstructed image frame and the reverse reconstructed image frame into a preset bidirectional image synthesis layer to obtain a reconstructed image, thereby generating the high frame rate reconstructed image sequence.

[0017] Optionally, in one embodiment of this application, the third generation module includes: a first traversal unit, configured to traverse the high frame rate reconstructed image sequence under each viewpoint, accumulate the event stream according to the frame rate, and obtain the event intermediate representation under each frame rate; and a second generation unit, configured to traverse each image frame in the high frame rate reconstructed image sequence, input the image frame and the corresponding event intermediate representation into the depth map prediction network, and generate the single-view depth map.

[0018] Optionally, in one embodiment of this application, the third generation module further includes: a second acquisition unit, configured to traverse each viewpoint and input the single-view depth map and the corresponding event intermediate representation into the point cloud generation network to obtain a single-view sparse point cloud; and a second traversal unit, configured to traverse each viewpoint and perform upsampling and fine-tuning operations on the single-view sparse point cloud to obtain the single-view dense point cloud.

[0019] Optionally, in one embodiment of this application, the reconstruction module includes: a stitching unit, used to stitch together the fused dense point cloud obtained at each moment to generate a continuous object point cloud motion trajectory; and a conversion unit, used to convert the object point cloud motion trajectory into a mesh model to obtain the object's three-dimensional motion trajectory.

[0020] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the event data augmentation-based moving object reconstruction method as described in the above embodiments.

[0021] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for reconstructing moving objects based on event data augmentation.

[0022] This application's embodiments can acquire motion video data containing moving objects, reconstruct image sequences, generate single-view depth maps, and ultimately obtain the object's three-dimensional motion trajectory. This fully utilizes the high temporal resolution of event data, overcoming the limitations of traditional cameras in tracking high-speed moving objects, and achieving more refined motion object reconstruction. Therefore, it solves the problems in related technologies, such as the difficulty in obtaining accurate motion trajectories of high-speed moving objects through detection or segmentation algorithms, and the limitations of traditional cameras, which rely on a fixed frame rate during data acquisition, hindering their ability to track high-speed moving objects and reducing the level of detail in motion object reconstruction.

[0023] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0024] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0025] Figure 1 This is a flowchart of a moving object reconstruction method based on event data augmentation provided in an embodiment of this application;

[0026] Figure 2 This is a flowchart of a moving object reconstruction method based on event data augmentation according to an embodiment of this application;

[0027] Figure 3 This is a schematic diagram illustrating the principle of a motion object reconstruction method based on event data augmentation according to an embodiment of this application;

[0028] Figure 4 This is a schematic diagram of a moving object reconstruction device based on event data augmentation provided in an embodiment of this application;

[0029] Figure 5 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of this application. Detailed Implementation

[0030] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0031] The following description, with reference to the accompanying drawings, outlines a method and apparatus for reconstructing moving objects based on event data augmentation, according to embodiments of this application. Addressing the aforementioned issues in the background art, where detection or segmentation algorithms struggle to obtain accurate trajectories of high-speed moving objects, and where traditional cameras, relying on a fixed frame rate during data acquisition, hinder tracking high-speed objects and reduce the precision of moving object reconstruction, this application provides a method for reconstructing moving objects based on event data augmentation. This method acquires motion video data containing moving objects, reconstructs image sequences, generates single-view depth maps, and ultimately obtains the object's three-dimensional motion trajectory. This fully leverages the high temporal resolution of event data, overcoming the limitations of traditional cameras in tracking high-speed moving objects, and achieving more refined moving object reconstruction. Thus, it solves the problems in the related art where detection or segmentation algorithms struggle to obtain accurate trajectories of high-speed moving objects, and where traditional cameras, relying on a fixed frame rate during data acquisition, hinder tracking high-speed objects and reduce the precision of moving object reconstruction.

[0032] Specifically, Figure 1 This is a flowchart illustrating a method for reconstructing moving objects based on event data augmentation, provided in an embodiment of this application.

[0033] like Figure 1 As shown, the moving object reconstruction method based on event data augmentation includes the following steps:

[0034] In step S101, motion video data containing moving objects is acquired.

[0035] It is understood that the embodiments of this application can collect motion video data containing moving objects, such as through conventional cameras and event cameras. The conventional camera is a common camera that collects three-channel color information and is based on equal-interval exposure, while the event camera is a novel dynamic visual sensor that operates asynchronously with pixels and only records changes in light intensity.

[0036] In actual implementation, the embodiments of this application can collect motion video data containing moving objects using traditional cameras and event cameras, providing a basis for obtaining high frame rate reconstructed image sequences based on the motion video data, thereby breaking through the limitations of traditional cameras in high-speed object tracking and obtaining more refined moving object reconstruction.

[0037] In step S102, a high frame rate reconstructed image sequence that meets the first preset frame rate condition is generated based on the motion video data.

[0038] It is understood that satisfying the first preset frame rate condition in the embodiments of this application can mean that the reconstructed image sequence can satisfy a certain high frame rate.

[0039] In actual implementation, the embodiments of this application can generate high frame rate reconstructed image sequences. Based on the collected motion video data, a high frame rate reconstructed image sequence that meets the first preset frame rate condition is generated, thereby obtaining a more refined reconstruction of moving objects.

[0040] It should be noted that the first preset frame rate can be set by those skilled in the art according to the actual situation, and no specific restrictions are imposed here.

[0041] Optionally, in one embodiment of this application, generating a high frame rate reconstructed image sequence that meets a preset frame rate condition based on motion video data includes: preprocessing the motion video data to obtain a low frame rate image sequence and an event stream that meet a second preset frame rate condition; inputting the low frame rate image sequence and the event stream into a preset video interpolation network to generate a high frame rate reconstructed image sequence that meets the preset frame rate condition.

[0042] It is understood that the second preset frame rate condition in the embodiments of this application can be that the image sequence meets a certain low frame rate, the low frame rate image sequence is the image sequence captured by a traditional camera, which is represented by a fixed number of images per second, and the event stream is the set of event points captured by the event camera.

[0043] In actual implementation, the embodiments of this application can preprocess the acquired motion video data to obtain low frame rate image sequences and event streams, that is, to obtain image sequences acquired by traditional cameras and event point sets acquired by event cameras. Furthermore, the embodiments of this application can input the low frame rate image sequences and event streams into a certain video interpolation network to generate high frame rate reconstructed image sequences that meet certain frame rate conditions, thereby making full use of the high temporal resolution of event data to enhance the reconstruction of moving objects.

[0044] It should be noted that the second preset frame rate condition can be set by those skilled in the art according to the actual situation, and no specific restrictions are imposed here.

[0045] Optionally, in one embodiment of this application, the event point set of the event stream includes event point coordinates, event point timestamps, and event point polarities.

[0046] In some embodiments, the event point set of the event stream includes, but is not limited to, event point coordinates, event point timestamps, and event point polarities. Specifically, the event point coordinates include the x-coordinate and y-coordinate of the event point, and the event point polarities include positive and negative polarities. The event point set of the event stream provides a basis for the subsequent input video frame interpolation network, thus facilitating the reconstruction of moving objects.

[0047] Optionally, in one embodiment of this application, a low frame rate image sequence and an event stream are input into a preset video frame interpolation network to generate a high frame rate reconstructed image sequence that meets a preset frame rate condition. This includes: accumulating the event stream forward and backward according to each adjacent image frame in the low frame rate image sequence to obtain a forward event intermediate representation and a backward event intermediate representation; feeding the forward event intermediate representation and the backward event intermediate representation into a preset optical flow prediction network to obtain a forward optical flow and a backward optical flow; for each adjacent image frame in the low frame rate image sequence, splitting the forward optical flow and the backward optical flow according to a linear ratio, adding the previous moment's image frame to the split forward optical flow to obtain a forward reconstructed image frame, and subtracting the next moment's image frame from the split backward optical flow to obtain a backward reconstructed image frame; inputting the forward reconstructed image frame and the backward reconstructed image frame into a preset bidirectional image synthesis layer to obtain a reconstructed image, thereby generating a high frame rate reconstructed image sequence.

[0048] For example, in this embodiment, the number of pixel-level event points between each adjacent image frame can be counted in both forward and reverse directions to form a matrix consistent with the original event camera resolution, serving as the forward and reverse event intermediate representations. The optical flow prediction network in this embodiment can consist of convolutional and deconvolutional layers, respectively feeding the forward and reverse event intermediate representations into a specific optical flow prediction network to obtain the forward and reverse optical flows. For each adjacent image frame in a low frame rate image sequence, this embodiment can split the forward and reverse optical flows according to a linear ratio, adding the previous moment's image frame to the split forward optical flow to obtain a forward reconstructed image frame, and subtracting the next moment's image frame from the split reverse optical flow to obtain a reverse reconstructed image frame. The bidirectional image synthesis layer in this embodiment can use convolutional layers to fine-tune the forward and reverse reconstructed image frames, and then perform pixel-level averaging to obtain a single reconstructed image, thereby achieving the reconstruction of moving objects using event data as enhancement.

[0049] In step S103, for each viewpoint, a single-view depth map is generated based on the corresponding event intermediate representation for the high frame rate reconstructed image sequence.

[0050] As one possible implementation, embodiments of this application can generate single-view depth maps. For high-frame-rate reconstructed image sequences under each viewpoint, single-view depth maps are generated with the help of corresponding event intermediate representations, thereby providing a basis for subsequent input point cloud generation networks and further ensuring more refined reconstruction of moving objects.

[0051] Optionally, in one embodiment of this application, for each high frame rate reconstructed image sequence under each viewpoint, a single-view depth map is generated based on the corresponding intermediate event representation, including: traversing the high frame rate reconstructed image sequence under each viewpoint, accumulating the event stream according to the frame rate to obtain the intermediate event representation under each frame rate; traversing each image frame in the high frame rate reconstructed image sequence, inputting the image frame and the corresponding intermediate event representation into the depth map prediction network to generate a single-view depth map.

[0052] For example, embodiments of this application can reconstruct the frame rate of an image sequence at a high frame rate, and for any two adjacent images, count the number of pixel-level event points between them to form a matrix consistent with the original event camera resolution, which serves as an intermediate representation of events in the next image at that frame rate.

[0053] Furthermore, the depth map prediction network in the application embodiment can be composed of convolutional layers and deconvolutional layers. It traverses each image frame in the high frame rate reconstructed image sequence, and uses the image frame and the corresponding event intermediate representation as input to the depth map prediction network to generate a single-view depth map. For example, after performing several convolutional layer operations on the image frame and the event intermediate representation, they are stitched together, and then restored through several deconvolutional layers to obtain a single-view depth map with the same resolution as the original image sequence.

[0054] In step S104, the single-view depth map is input into a preset point cloud generation network to generate a single-view dense point cloud.

[0055] Specifically, embodiments of this application can generate single-view point clouds by inputting a single-view depth map into a certain point cloud generation network, thereby generating a single-view dense point cloud to ensure the three-dimensional reconstruction of moving objects.

[0056] Optionally, in one embodiment of this application, the single-view depth map is input into a preset point cloud generation network to generate a single-view dense point cloud, including: traversing each viewpoint, inputting the single-view depth map and the corresponding event intermediate representation into the point cloud generation network to obtain a single-view sparse point cloud; traversing each viewpoint, performing upsampling and fine-tuning operations on the single-view sparse point cloud to obtain a single-view dense point cloud.

[0057] As one possible implementation, embodiments of this application can traverse each viewpoint, inputting the single-view depth map and the corresponding intermediate event representation into the point cloud generation network. The point cloud generation network can consist of transformation operations based on camera intrinsic parameters. Through the transformation matrix composed of camera intrinsic parameters, the image coordinate system under the single-view depth map is converted into the point cloud coordinate system, generating a single-view sparse point cloud.

[0058] Furthermore, in the embodiments of this application, each viewpoint can be traversed to perform upsampling and fine-tuning operations on the sparse point cloud of a single view. The upsampling and fine-tuning operations can be implemented by a multilayer perceptual network based on point clouds. For example, sampling can be performed first and then fine-tuning can be performed to obtain a dense point cloud of a single view.

[0059] The embodiments of this application can obtain single-view sparse point clouds and single-view dense point clouds by traversing each viewpoint, thereby utilizing the point cloud features to obtain the motion trajectory of the object and improving the planar reconstruction effect of single-view sparse point clouds and the three-dimensional reconstruction effect of single-view dense point clouds.

[0060] In step S105, the single-view dense point cloud under each perspective is input into a preset multi-view point cloud fusion network to obtain the fused dense point cloud.

[0061] In some embodiments, multi-view point cloud fusion can be performed by inputting the single-view dense point cloud from each perspective into a certain multi-view point cloud fusion network to obtain the fused dense point cloud. The multi-view point cloud fusion network includes regional point cloud feature extraction operation, multi-view point cloud registration operation and multi-view point cloud fusion operation.

[0062] Furthermore, the region-level point cloud feature extraction operation in this embodiment can be implemented through a multi-scale grouped multilayer perceptron to divide the point cloud into several regions, and extract fine-grained structural information of the point cloud within each region using a multilayer perceptron to obtain high-dimensional point cloud region-level features. The multi-view point cloud registration operation in this embodiment can perform feature matching based on the point cloud region-level features of different views, minimizing geometric projection errors through iterative correspondence search and transformation estimation, ultimately obtaining the correspondence between point cloud region-level features under different views. The multi-view point cloud fusion operation in this embodiment can fuse point clouds based on the correspondence between point cloud region-level features under different views, stitching together the point cloud region-level features from different views according to the correspondence to generate a fused dense point cloud, further ensuring the 3D reconstruction of moving objects.

[0063] In step S106, the fused dense point cloud obtained at each time step is processed to render the three-dimensional motion trajectory of the object.

[0064] In actual implementation, the embodiments of this application can reconstruct moving objects by processing the fused dense point cloud obtained at each moment and rendering the three-dimensional motion trajectory of the object, thereby obtaining a more refined reconstruction of the moving object based on the three-dimensional motion trajectory of the object.

[0065] Optionally, in one embodiment of this application, the fused dense point cloud obtained at each time moment is processed to render the three-dimensional motion trajectory of the object, including: stitching the fused dense point cloud obtained at each time moment to generate a continuous object point cloud motion trajectory; converting the object point cloud motion trajectory into a mesh model to obtain the three-dimensional motion trajectory of the object.

[0066] In other embodiments, the fused dense point cloud obtained at each moment can be stitched together to form a continuous object point cloud motion trajectory, and the object point cloud motion trajectory can be converted into a mesh model to obtain the object's three-dimensional motion trajectory.

[0067] For example, embodiments of this application can convert the motion trajectory of an object's point cloud into a mesh model through point cloud smoothing, normal calculation, and mesh generation operations. Specifically, point cloud smoothing can use statistical analysis methods to filter irregular data in the point cloud, normal calculation can use approximate estimation methods to estimate normals, and mesh generation can use a greedy triangulation algorithm to generate a mesh model, thereby enhancing the reconstruction of the moving object.

[0068] Specifically, in combination Figure 2 and Figure 3 As shown, the working principle of the motion object reconstruction method based on event data augmentation of this application is explained in detail with a specific embodiment.

[0069] like Figure 2 As shown, embodiments of this application may include the following steps:

[0070] Step S201: Video and Event Data Acquisition: Video containing moving objects is acquired using a conventional camera and an event camera, respectively. In this embodiment, a conventional camera and an event camera can be used to acquire video containing moving objects, respectively. The conventional camera is a common camera that acquires three-channel color information and is based on equal-interval exposures, while the event camera is a novel dynamic vision sensor that operates asynchronously with pixels and only records changes in light intensity.

[0071] Step S202: High frame rate reconstructed image sequence generation: A high frame rate reconstructed image sequence is generated using data acquired by a traditional camera and an event camera. In this embodiment, the data acquired by the traditional camera and the event camera can be preprocessed to obtain a low frame rate image sequence and an event stream, respectively. The low frame rate image sequence is an image sequence acquired by the traditional camera, characterized by a fixed number of images per second. The event stream is a set of event points acquired by the event camera. Each event point includes event point coordinates, event point timestamps, and event point polarity. Event point coordinates include the x-coordinate and y-coordinate of the event point; event point polarity includes positive and negative polarities.

[0072] Step S203: Single-view depth map generation: For each viewpoint, a single-view depth map is generated using the corresponding event intermediate representation. In this embodiment, a low-frame-rate image sequence and event stream can be input into a video interpolation network to generate a high-frame-rate reconstructed image sequence. For each viewpoint, a single-view depth map is generated using the corresponding event intermediate representation.

[0073] Step S204: Single-view point cloud generation: Input the single-view depth map into the point cloud generation network to generate a single-view dense point cloud. In this embodiment, each viewpoint can be traversed, and the single-view depth map and corresponding event intermediate representations can be input into the point cloud generation network to obtain a single-view sparse point cloud. Then, by traversing each viewpoint, the single-view sparse point cloud is upsampled and fine-tuned to obtain a single-view dense point cloud.

[0074] Step S205: Multi-view point cloud fusion: Input the single-view dense point cloud from each viewpoint into the multi-view point cloud fusion network to obtain the fused dense point cloud. In this embodiment, the single-view dense point cloud from each viewpoint can be input into the multi-view point cloud fusion network to obtain the fused dense point cloud. The multi-view point cloud fusion network includes region-level point cloud feature extraction, multi-view point cloud registration, and multi-view point cloud fusion.

[0075] Step S206: Moving Object Reconstruction: The fused dense point cloud obtained at each moment is processed to render the three-dimensional motion trajectory of the object. In this embodiment, the fused dense point cloud obtained at each moment can be stitched together to form a continuous object point cloud motion trajectory. The object point cloud motion trajectory is then converted into a mesh model to obtain the three-dimensional motion trajectory of the object.

[0076] like Figure 3 As shown, the embodiments of this application include: a video and event data acquisition module, a high frame rate reconstructed image sequence generation module, a single-view depth map generation module, a single-view point cloud generation module, a multi-view point cloud fusion module, and a moving object reconstruction module.

[0077] Specifically, the video and event data acquisition module uses traditional cameras and event cameras to acquire videos containing moving objects, respectively; the high frame rate reconstructed image sequence generation module uses the data acquired by traditional cameras and event cameras to generate high frame rate reconstructed image sequences; the single-view depth map generation module generates single-view depth maps for each viewpoint's high frame rate reconstructed image sequence by using the corresponding event intermediate representation; the single-view point cloud generation module inputs the single-view depth map into the point cloud generation network to generate a single-view dense point cloud; the multi-view point cloud fusion module inputs the single-view dense point cloud from each viewpoint into the multi-view point cloud fusion network to obtain a fused dense point cloud; and the moving object reconstruction module processes the fused dense point cloud obtained at each moment to render the three-dimensional motion trajectory of the object.

[0078] The moving object reconstruction method based on event data augmentation proposed in this application can acquire motion video data containing moving objects, reconstruct image sequences, generate single-view depth maps, and finally obtain the three-dimensional motion trajectory of the object. This fully utilizes the high temporal resolution of event data, overcoming the limitations of traditional cameras in tracking high-speed moving objects, and achieving more refined moving object reconstruction. Therefore, it solves the problem in related technologies where detection or segmentation algorithms struggle to obtain accurate motion trajectories of high-speed moving objects. Furthermore, the fixed frame rate of traditional cameras during data acquisition hinders their ability to track high-speed motion, reducing the level of detail in moving object reconstruction.

[0079] Next, with reference to the accompanying drawings, a motion object reconstruction apparatus based on event data enhancement proposed according to an embodiment of this application is described.

[0080] Figure 4 This is a schematic diagram of the structure of the motion object reconstruction device based on event data augmentation according to an embodiment of this application.

[0081] like Figure 4 As shown, the motion object reconstruction device 10 based on event data enhancement includes: an acquisition module 100, a first generation module 200, a second generation module 300, a third generation module 400, a fusion module 500, and a reconstruction module 600.

[0082] Specifically, the acquisition module 100 is used to acquire motion video data containing moving objects.

[0083] The first generation module 200 is used to generate a high frame rate reconstructed image sequence that meets the first preset frame rate condition based on motion video data.

[0084] The second generation module 300 is used to reconstruct the image sequence at a high frame rate for each viewpoint and generate a single-view depth map based on the corresponding event intermediate representation.

[0085] The third generation module 400 is used to input a single-view depth map into a preset point cloud generation network to generate a single-view dense point cloud.

[0086] The fusion module 500 is used to input the single-view dense point cloud from each perspective into a preset multi-view point cloud fusion network to obtain the fused dense point cloud.

[0087] The reconstruction module 600 is used to process the fused dense point cloud obtained at each moment and render the three-dimensional motion trajectory of the object.

[0088] Optionally, in one embodiment of this application, the first generation module 200 includes: a first acquisition unit and a first generation unit.

[0089] The first acquisition unit is used to preprocess the motion video data to obtain a low frame rate image sequence and an event stream that meet the second preset frame rate condition.

[0090] The first generation unit is used to input low frame rate image sequences and event streams into a preset video frame interpolation network to generate high frame rate reconstructed image sequences that meet preset frame rate conditions.

[0091] Optionally, in one embodiment of this application, the event point set of the event stream includes event point coordinates, event point timestamps, and event point polarities.

[0092] Optionally, in one embodiment of this application, the first generation unit includes: a first acquisition subunit, a second acquisition subunit, a third acquisition subunit, and a generation subunit.

[0093] The first acquisition subunit is used to obtain the intermediate representation of the forward event and the intermediate representation of the reverse event by accumulating the event stream in the forward and reverse directions according to each adjacent image frame in the low frame rate image sequence.

[0094] The second acquisition subunit is used to feed the intermediate representations of the forward event and the intermediate representations of the reverse event into a preset optical flow prediction network to obtain the forward optical flow and the reverse optical flow.

[0095] The third acquisition subunit is used to split the forward optical flow and the reverse optical flow in a linear ratio for each adjacent image frame in a low frame rate image sequence, add the image frame of the previous moment to the split forward optical flow to obtain the forward reconstructed image frame, and subtract the image frame of the next moment from the split reverse optical flow to obtain the reverse reconstructed image frame.

[0096] The generation sub-unit is used to input the forward reconstructed image frame and the reverse reconstructed image frame into a preset bidirectional image synthesis layer to obtain the reconstructed image and generate a high frame rate reconstructed image sequence.

[0097] Optionally, in one embodiment of this application, the third generation module 400 includes: a first traversal unit and a second generation unit.

[0098] The first traversal unit is used to traverse the high frame rate reconstructed image sequence under each viewpoint, accumulate the event stream according to the frame rate, and obtain the intermediate representation of the event under each frame rate.

[0099] The second generation unit is used to traverse each image frame in the high frame rate reconstructed image sequence, input the image frame and the corresponding event intermediate representation into the depth map prediction network, and generate a single-view depth map.

[0100] Optionally, in one embodiment of this application, the third generation module 400 further includes: a second acquisition unit and a second traversal unit.

[0101] The second acquisition unit is used to traverse each viewpoint, input the single-view depth map and the corresponding event intermediate representation into the point cloud generation network, and obtain the single-view sparse point cloud.

[0102] The second traversal unit is used to traverse each viewpoint, upsample and fine-tune the sparse point cloud of a single view to obtain a dense point cloud of a single view.

[0103] Optionally, in one embodiment of this application, the reconstruction module 600 includes a splicing unit and a conversion unit.

[0104] The stitching unit is used to stitch together the fused dense point cloud obtained at each moment to generate a continuous object point cloud motion trajectory.

[0105] The conversion unit is used to convert the motion trajectory of an object's point cloud into a mesh model to obtain the object's three-dimensional motion trajectory.

[0106] It should be noted that the foregoing explanation of the embodiment of the motion object reconstruction method based on event data augmentation also applies to the motion object reconstruction device based on event data augmentation in this embodiment, and will not be repeated here.

[0107] The motion object reconstruction apparatus based on event data augmentation proposed in this application can acquire motion video data containing moving objects, reconstruct image sequences, generate single-view depth maps, and ultimately obtain the three-dimensional motion trajectory of the object. This fully utilizes the high temporal resolution of event data, overcoming the limitations of traditional cameras in tracking high-speed moving objects, and achieving more refined motion object reconstruction. Therefore, it solves the problem in related technologies where detection or segmentation algorithms struggle to obtain accurate motion trajectories of high-speed moving objects. Furthermore, the fixed frame rate of traditional cameras during data acquisition hinders their ability to track high-speed motion, reducing the level of detail in motion object reconstruction.

[0108] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include:

[0109] The memory 501, the processor 502, and the computer program stored on the memory 501 and capable of running on the processor 502.

[0110] When the processor 502 executes the program, it implements the motion object reconstruction method based on event data enhancement provided in the above embodiments.

[0111] Furthermore, electronic devices also include:

[0112] Communication interface 503 is used for communication between memory 501 and processor 502.

[0113] The memory 501 is used to store computer programs that can run on the processor 502.

[0114] The memory 501 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0115] If the memory 501, processor 502, and communication interface 503 are implemented independently, then the communication interface 503, memory 501, and processor 502 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 5 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0116] Optionally, in a specific implementation, if the memory 501, processor 502, and communication interface 503 are integrated on a single chip, then the memory 501, processor 502, and communication interface 503 can communicate with each other through an internal interface.

[0117] Processor 502 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0118] This embodiment also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method for reconstructing moving objects based on event data augmentation.

[0119] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0120] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0121] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0122] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0123] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0124] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0125] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0126] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A method for reconstructing moving objects based on event data augmentation, characterized in that, Includes the following steps: The system collects motion video data and event stream data containing moving objects, wherein the motion video data is collected using a traditional camera and the event stream data is collected using an event camera. The motion video data and the event stream data are preprocessed to obtain a low frame rate image sequence and an event stream that satisfy the second preset frame rate condition, respectively; a high frame rate reconstructed image sequence that satisfies the first preset frame rate condition is generated based on the low frame rate image sequence and the event stream. The high frame rate reconstructed image sequence under each viewpoint is traversed, the event stream is accumulated according to the frame rate, the intermediate representation of the event under each frame rate is obtained, and a single-view depth map is generated based on the corresponding intermediate representation of the event. The single-view depth map is input into a preset point cloud generation network to generate a single-view dense point cloud. The single-view dense point cloud from each perspective is input into a preset multi-view point cloud fusion network to obtain a fused dense point cloud; and The fused dense point cloud obtained at each moment is processed to render the three-dimensional motion trajectory of the object.

2. The method according to claim 1, characterized in that, The process of generating a high frame rate reconstructed image sequence that meets preset frame rate conditions based on the low frame rate image sequence and event stream includes: The low frame rate image sequence and the event stream are input into a preset video frame interpolation network to generate a high frame rate reconstructed image sequence that meets the preset frame rate conditions.

3. The method according to claim 2, characterized in that, The event point set of the event stream includes event point coordinates, event point timestamps, and event point polarities.

4. The method according to claim 2, characterized in that, The step of inputting the low frame rate image sequence and the event stream into a preset video frame interpolation network to generate a high frame rate reconstructed image sequence that meets the preset frame rate condition includes: Based on each adjacent image frame in the low frame rate image sequence, the event stream is accumulated in both forward and reverse directions to obtain the forward event intermediate representation and the reverse event intermediate representation. The intermediate representations of the forward event and the intermediate representations of the reverse event are respectively fed into a preset optical flow prediction network to obtain the forward optical flow and the reverse optical flow; For each adjacent image frame in the low frame rate image sequence, the forward optical flow and the reverse optical flow are split in a linear ratio. The image frame at the previous moment is added to the split forward optical flow to obtain a forward reconstructed image frame, and the image frame at the next moment is subtracted from the split reverse optical flow to obtain a reverse reconstructed image frame. The forward reconstructed image frame and the reverse reconstructed image frame are input into a preset bidirectional image synthesis layer to obtain a reconstructed image, thereby generating the high frame rate reconstructed image sequence.

5. The method according to claim 1, characterized in that, A single-view depth map is generated based on the corresponding intermediate event representations, including: Each image frame in the high frame rate reconstructed image sequence is traversed, and the image frame and the corresponding intermediate representation of the event are input into the depth map prediction network to generate the single-view depth map.

6. The method according to claim 1, characterized in that, The step of inputting the single-view depth map into a preset point cloud generation network to generate a single-view dense point cloud includes: By traversing each viewpoint, the single-view depth map and the corresponding intermediate representation of the event are input into the point cloud generation network to obtain a single-view sparse point cloud. By traversing each viewpoint, the sparse point cloud of the single view is upsampled and fine-tuned to obtain the dense point cloud of the single view.

7. The method according to claim 1, characterized in that, The process of processing the fused dense point cloud obtained at each moment to render the three-dimensional motion trajectory of the object includes: The fused dense point cloud obtained at each moment is stitched together to generate a continuous object point cloud motion trajectory. The motion trajectory of the object's point cloud is converted into a mesh model to obtain the object's three-dimensional motion trajectory.

8. A motion object reconstruction device based on event data augmentation, characterized in that, include: The acquisition module is used to acquire motion video data and event stream data containing moving objects, wherein the motion video data is acquired through a traditional camera and the event stream data is acquired through an event camera; The first generation module is used to preprocess the motion video data and the event stream data to obtain a low frame rate image sequence and an event stream that satisfy a second preset frame rate condition, respectively; and to generate a high frame rate reconstructed image sequence that satisfies the first preset frame rate condition based on the low frame rate image sequence and the event stream. The second generation module is used to traverse the high frame rate reconstructed image sequence under each viewpoint, accumulate the event stream according to the frame rate, obtain the intermediate representation of the event under each frame rate, and generate a single-view depth map based on the corresponding intermediate representation of the event. The third generation module is used to input the single-view depth map into a preset point cloud generation network to generate a single-view dense point cloud. The fusion module is used to input the single-view dense point cloud from each perspective into a preset multi-view point cloud fusion network to obtain a fused dense point cloud; and The reconstruction module is used to process the fused dense point cloud obtained at each moment and render the three-dimensional motion trajectory of the object.

9. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to implement the event data augmentation-based moving object reconstruction method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the event data augmentation-based moving object reconstruction method as described in any one of claims 1-7.

Images