Depth image video generation method and device based on data fusion

Abstract

This application relates to a method and apparatus for generating depth image video based on data fusion. The method includes: acquiring video and event stream data at a preset frame rate from an event camera; combining adjacent frames in the video and dividing the event stream data within corresponding time intervals to obtain event data packets; constructing a convolutional neural network; concatenating data with the same sequence number and inputting it into the encoder part of the convolutional neural network model for forward propagation; concatenating feature vectors to generate a depth map; and calculating the image frames of the depth map corresponding to each video frame sequence number to form a depth map sequence to generate a depth image video. This solves the problems in related technologies, such as the difficulty in accurately obtaining depth maps synchronized with video frames from video data and event stream data, the difficulty in achieving depth map video generation, the high dependence on hyperparameters and optimization tools in problem solving, poor stability, and the difficulty in effectively denoising event stream data.

Description

Technical Field

[0001] This application relates to the fields of computer vision and neuromorphic computing technology, and in particular to a method and apparatus for generating depth images and videos based on data fusion. Background Technology

[0002] An event camera is a bio-inspired sensor that operates on a significantly different principle from traditional cameras. Unlike traditional cameras that capture absolute light intensity at a fixed frame rate, an event camera outputs data only when the scene's light intensity changes; this output data is called an event stream. Compared to traditional cameras, event cameras offer advantages such as high dynamic range, high temporal resolution, and no motion blur.

[0003] In the field of event-based camera depth estimation, existing methods based on handcrafted models establish a matching relationship between pose signals and pixel depths, and solve the problem using hyperparameters and optimization tools.

[0004] However, in related technologies, it is impossible to obtain depth maps synchronized with video frames from video data and event stream data, making it difficult to achieve the effect of depth map video generation. The problem-solving process is highly dependent on hyperparameters and optimization tools, resulting in poor stability and difficulty in achieving better denoising effects for event stream data, which needs to be improved. Summary of the Invention

[0005] This application provides a method and apparatus for generating depth images and videos based on data fusion, in order to solve the problems in related technologies, such as the inability to obtain depth maps synchronized with video frames from video data and event stream data, the difficulty in achieving the effect of depth map video generation, the high dependence on hyperparameters and optimization tools in problem solving, poor stability, and the difficulty in achieving better denoising effects on event stream data.

[0006] The first aspect of this application provides a method for generating depth image video based on data fusion, comprising the following steps: acquiring video and event stream data at a preset frame rate from an event camera; combining adjacent frames in the video to obtain image frame groups, wherein each group of video frames in the image frame group includes a target frame for depth estimation and an adjacent frame before and after it; for the timestamp corresponding to the intermediate frame of each group of video frames, dividing the event stream data into data packets with the same time interval according to the timestamp; constructing a convolutional neural network; based on the image frame groups and the data packets, concatenating data with the same sequence number and inputting it into the model encoder part of the convolutional neural network for forward propagation to obtain preprocessed frames and event stream data to obtain feature vectors to complete the encoding calculation of fused data; concatenating the feature vectors and using a multimodal fusion network decoder for forward propagation decoding calculation to generate a depth map; and calculating the image frames of the depth map corresponding to each video frame sequence number, and combining all depth maps to form a depth map sequence to generate a depth image video.

[0007] Optionally, in one embodiment of this application, the step of combining adjacent frames in the video to obtain an image frame group, wherein each group of video frames in the image frame group includes a target frame for depth estimation and a preceding and following frame, and for the timestamp corresponding to the intermediate frame of each group of video frames, the event stream data is divided into data packets with the same time interval according to the timestamp, including: combining adjacent frames in the video, wherein each group of frames includes a target frame for depth estimation and a preceding and following frame, and Rk represents the original video frame with sequence number k, I k I represents the result after video frame preprocessing. k This represents a group of video frames corresponding to the original video frame with sequence number k, where prep() refers to the preprocessing process:

[0008] I k =prep(R k ),

[0009] I k ={I k-1 I k I k+1},

[0010] For the timestamp corresponding to the intermediate frame of each group of image frames, the event stream data is divided into data packets with the same time interval according to the timestamp. Each event e in the event stream data has corresponding x, y, p, and t attributes. Let video frame I... k The timestamp is t k , use e i This indicates that the sequence number is i and the timestamp is t in the event stream. iA single event, E k Indicates video frame group I k The corresponding event packet, recorded

[0011] E k ={e i |wheret k ≤t i ≤t k+1},

[0012] To perform event stream tensor quantization calculations and preprocessing, for each event packet E k Accumulate it in the spatial domain to generate the corresponding event tensor T. k This allows it to be adapted to neural network inputs.

[0013] Optionally, in one embodiment of this application, the formula for obtaining the event stream tensor is:

[0014]

[0015] k b (a) = max(0, 1-|a|),

[0016]

[0017] Where N is the total number of frames in the input video, x i y i t i p i The event package E is respectively k Event e i The frame's two-dimensional spatial coordinates, timestamp, and polarity; B is the discretization hyperparameter; k b (a) Used for bilinear sampling of spatiotemporal neighborhoods

[0018] Furthermore, the formula for obtaining the result after video frame preprocessing is as follows:

[0019] I k =sobel(R) k ),

[0020] Among them, R k The input is the original video frame, and sobel() is the sobel operator.

[0021] Optionally, in one embodiment of this application, the encoder portion of the convolutional neural network includes an input convolutional layer, a hidden convolutional layer, and an output convolutional layer. The input convolutional layer has 2*B + 3*k input channels, corresponding to the preprocessed event tensor and video frame combination, where k is the number of channels of the input video image frames. The input convolutional layer has a kernel size of 3*3, a stride of 1, and 16 output channels. The hidden convolutional layer has 16 input channels, a kernel size of 3*3, a stride of 1, and 16 output channels. The output convolutional layer has 16 input channels, a kernel size of 3*3, a stride of 1, and 64 output channels.

[0022] Optionally, in one embodiment of this application, the step of concatenating the feature vectors and performing forward propagation decoding calculations using a multimodal fusion network decoder to generate a depth map includes: inputting the concatenated feature vectors into a coarse synthesis sub-network to obtain a coarse output result, wherein the coarse output result is concatenated with the input preprocessed frame and input into a fine-tuning sub-network to obtain a depth prediction result.

[0023] A second aspect of this application provides a depth image and video generation apparatus based on data fusion, comprising: an acquisition module for acquiring video and event stream data at a preset frame rate from an event camera; a combination module for combining adjacent frames in the video to obtain image frame groups, wherein each group of video frames in the image frame group includes a target frame for depth estimation and an adjacent frame before and after it, and for the timestamp corresponding to the intermediate frame of each group of video frames, the event stream data is divided into data packets with the same time interval according to the timestamp; a construction module for constructing a convolutional neural network; an input module for concatenating data with the same sequence number based on the image frame groups and the data packets and inputting it into the model encoder part of the convolutional neural network for forward propagation to obtain preprocessed frames and event stream data to obtain feature vectors to complete the encoding calculation of fused data; a first generation module for concatenating the feature vectors and performing forward propagation decoding calculation using a multimodal fusion network decoder to generate a depth map; and a second generation module for calculating the image frame of the depth map corresponding to each video frame sequence number and combining all depth maps to form a depth map sequence to generate a depth image and video.

[0024] Optionally, in one embodiment of this application, the combining module includes: a combining unit, configured to combine adjacent frames in the video, wherein each group of frames includes a target frame for depth estimation and one adjacent frame before and after it, and Rk represents the original video frame with sequence number k. k I represents the result after video frame preprocessing. k This represents a group of video frames corresponding to the original video frame with sequence number k, where prep() refers to the preprocessing process:

[0025] I k =prep(R k ),

[0026] I k ={I k-1 I k I k+1},

[0027] For the timestamp corresponding to the intermediate frame of each group of image frames, the event stream data is divided into data packets with the same time interval according to the timestamp. Each event e in the event stream data has corresponding x, y, p, and t attributes. Let video frame I... k The timestamp is t k , use e i This indicates that the sequence number is i and the timestamp is t in the event stream. i A single event, E k Indicates video frame group I k The corresponding event packet, denoted as:

[0028] E k ={e i |where t k ≤t i ≤t k+1},

[0029] To perform event stream tensor quantization calculations and preprocessing, for each event packet E k Accumulate it in the spatial domain to generate the corresponding event tensor T. k This allows it to be adapted to neural network inputs.

[0030] Optionally, in one embodiment of this application, the formula for obtaining the event stream tensor is:

[0031]

[0032] k b (a) = max(0, 1-|a|),

[0033]

[0034] Where N is the total number of frames in the input video, x i y i t i p i The event package E is respectively k Event e i The frame's two-dimensional spatial coordinates, timestamp, and polarity; B is the discretization hyperparameter; k b (a) Used for bilinear sampling of spatiotemporal neighborhoods

[0035] Furthermore, the formula for obtaining the result after video frame preprocessing is as follows:

[0036] I k =sobel(R) k ),

[0037] Among them, R k The input is the original video frame, and sobel() is the sobel operator.

[0038] Optionally, in one embodiment of this application, the encoder portion of the convolutional neural network includes an input convolutional layer, a hidden convolutional layer, and an output convolutional layer. The input convolutional layer has 2*B + 3*k input channels, corresponding to the preprocessed event tensor and video frame combination, where k is the number of channels of the input video image frames. The input convolutional layer has a kernel size of 3*3, a stride of 1, and 16 output channels. The hidden convolutional layer has 16 input channels, a kernel size of 3*3, a stride of 1, and 16 output channels. The output convolutional layer has 16 input channels, a kernel size of 3*3, a stride of 1, and 64 output channels.

[0039] Optionally, in one embodiment of this application, the first generation module includes: an input / output unit, used to input the concatenated feature vector into a coarse synthesis sub-network to obtain a coarse output result, wherein the coarse output result is concatenated with the input preprocessed frame and input into a fine-tuning sub-network to obtain a depth prediction result.

[0040] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the depth image and video generation method based on data fusion as described in the above embodiments.

[0041] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described data fusion-based depth image and video generation method.

[0042] This application embodiment can use event data to provide inter-frame motion information, obtain a depth map synchronized with video frames from video data and event stream data, and use a convolutional neural network to encode the event stream, thereby achieving the effect of depth map video generation. This ensures more effective observation of high-speed scenes and improves the denoising effect of event stream data. Therefore, it solves the problems in related technologies, such as the inability to obtain a depth map synchronized with video frames from video data and event stream data, the difficulty in achieving the effect of depth map video generation, the high dependence on hyperparameters and optimization tools in problem solving, poor stability, and the inability to effectively denoise event stream data.

[0043] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0044] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0045] Figure 1 This is a flowchart of a depth image and video generation method based on data fusion according to an embodiment of this application;

[0046] Figure 2 This is a streaming data visualization diagram of a data fusion-based depth image and video generation method according to an embodiment of this application;

[0047] Figure 3 This is a flowchart illustrating a data fusion-based depth image and video generation method according to an embodiment of this application.

[0048] Figure 4 This is a visualization diagram of a depth map of a depth image and video generation method based on data fusion according to an embodiment of this application.

[0049] Figure 5 This is a schematic diagram of a depth image and video generation device based on data fusion according to an embodiment of this application;

[0050] Figure 6 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of this application. Detailed Implementation

[0051] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0052] The following description, with reference to the accompanying drawings, describes a data fusion-based method and apparatus for generating depth images and videos according to embodiments of this application. Addressing the issues raised in the background art, such as the inability to obtain depth maps synchronized with video frames from video data and event stream data, the difficulty in achieving depth map video generation, high dependence on hyperparameters and optimization tools in problem-solving, poor stability, and difficulty in effectively denoising event stream data, this application provides a data fusion-based method for generating depth images and videos. In this method, event data can provide inter-frame motion information, a depth map synchronized with video frames can be obtained from video data and event stream data, and a convolutional neural network is used to encode the event stream, thereby achieving the effect of depth map video generation. This ensures more effective observation of high-speed scenes and improves the denoising effect of event stream data. Therefore, this method solves the problems in the related art, such as the inability to obtain depth maps synchronized with video frames from video data and event stream data, the difficulty in achieving depth map video generation, the high dependence on hyperparameters and optimization tools in problem-solving, poor stability, and difficulty in effectively denoising event stream data.

[0053] Specifically, Figure 1 This is a flowchart illustrating a depth image and video generation method based on data fusion, provided in an embodiment of this application.

[0054] like Figure 1 As shown, the depth image and video generation method based on data fusion includes the following steps:

[0055] In step S101, video and event stream data at a preset frame rate are acquired from the event camera.

[0056] It is understood that, in the embodiments of this application, compared with traditional cameras, event cameras have extremely high temporal resolution and dynamic range, and can completely record motion information in the scene. Traditional cameras are limited by their working mechanisms such as exposure, and the cost of professional high-speed cameras required to shoot high frame rate and high dynamic range videos in dynamic scenes is extremely high. The signals generated by event cameras and traditional cameras can complement each other's deficiencies in spatial brightness information and temporal resolution.

[0057] In actual implementation, the embodiments of this application can acquire raw data and obtain video and event stream data at a certain frame rate through a visual sensor, such as acquiring video and event stream data at a preset frame rate through an event camera. This facilitates the subsequent acquisition of a depth map synchronized with the video frame from the video data and event stream data, achieving the effect of depth map video generation, thereby enabling more effective observation of high-speed scenes.

[0058] In step S102, adjacent frames in the video are combined to obtain image frame groups. Each group of video frames in the image frame group contains the target frame for depth estimation and the frame that is adjacent to it. For the timestamp corresponding to the middle frame of each group of video frames, the event stream data is divided into data packets with the same time interval according to the timestamp.

[0059] As one possible implementation, the embodiments of this application can perform data structuring and preprocessing. First, adjacent frames in the video are combined to obtain image frame groups. Each group of video frames in the image frame group contains the target frame for depth estimation and the frame that is adjacent to the previous and next frames. At the same time, event data packets are aggregated according to the corresponding time interval, and event stream tensor quantization calculation and preprocessing are performed. Each event packet is accumulated in the spatial domain to generate the corresponding event tensor, thereby adapting to the neural network input.

[0060] Optionally, in one embodiment of this application, adjacent frames in the video are combined to obtain an image frame group, wherein each group of video frames in the image frame group includes a target frame for depth estimation and a preceding and following frame. For the timestamp corresponding to the intermediate frame of each group of video frames, the event stream data is divided into data packets with the same time interval according to the timestamp, including: combining adjacent frames in the video, each group of frames including a target frame for depth estimation and a preceding and following frame, and using Rk to represent the original video frame with sequence number k, I k I represents the result after video frame preprocessing. k This represents a set of video frames corresponding to the original video frame with sequence number k, where prep() refers to the preprocessing process.

[0061] I k =prep(R k ),

[0062] I k ={I k-1 I k I k+1},

[0063] For the timestamp corresponding to the intermediate frame of each group of image frames, the event stream data is divided into data packets with the same time interval according to the timestamp. Each event e in the event stream data has corresponding x, y, p, and t attributes. Let video frame I be... k The timestamp is t k , use e i This indicates that the sequence number is i and the timestamp is t in the event stream. i A single event, E k Indicates video frame group I k The corresponding event packet, denoted as:

[0064] E k ={ei |wheret k ≤t i ≤t k+1},

[0065] To perform event stream tensor quantization calculations and preprocessing, for each event packet E k Accumulate it in the spatial domain to generate the corresponding event tensor T. k This allows it to be adapted to neural network inputs.

[0066] In some embodiments, data structuring and preprocessing can be performed, combining adjacent frames in the video. Each group of frames includes the target frame for depth estimation and the preceding and following frame for subsequent preprocessing, as shown in the following formula:

[0067] I k =prep(R k ),

[0068] I k ={I k-1 I k I k+1},

[0069] Among them, R k I represents the original video frame with sequence number k. k Indicates the result after video frame preprocessing, bold I k This represents a set of video frames corresponding to the video frame with sequence number k. prep() refers to the preprocessing process.

[0070] For the timestamp corresponding to the intermediate frame of each group of image frames, the event stream data is divided into data packets with the same time interval according to the timestamp, as shown in the following formula:

[0071] E k ={e i |where t k ≤t i ≤t k+1},

[0072] Each event e in the event stream has corresponding x, y, p, and t attributes, denoted as video frame I. k The timestamp is t k , use e i This indicates that the sequence number is i and the timestamp is t in the event stream. i A single event, E k Indicates video frame group I k The corresponding event packet.

[0073] The embodiments of this application can perform event stream tensor quantization calculation and preprocessing for each event packet E. kAccumulate it in the spatial domain to generate the corresponding event tensor T. k This allows it to adapt to neural network inputs, making it easier to use event streams and video image frames as inputs, better utilizing multimodal data information, and greatly improving performance.

[0074] Optionally, in one embodiment of this application, the formula for obtaining the event stream tensor is:

[0075]

[0076] k b (a) = max(0, 1-|a|),

[0077]

[0078] Where N is the total number of frames in the input video, x i y i t i p i Event package E k Event e i The frame's two-dimensional spatial coordinates, timestamp, and polarity; B is the discretization hyperparameter; k b (a) Used for bilinear sampling of the spatiotemporal neighborhood, and the formula for obtaining the result after video frame preprocessing is:

[0079] I k =sobel(R) k ),

[0080] Among them, R k The input is the original video frame, and sobel() is the sobel operator.

[0081] In some embodiments, data structuring and preprocessing can be performed. First, adjacent image frames in the video sequence are combined, and event data packets are aggregated according to the corresponding time intervals. Both are preprocessed. Data structuring and preprocessing are performed by using the formula for obtaining the event stream tensor and the formula for obtaining the result of video frame preprocessing. Thus, the event stream and video image frames are used as inputs to better utilize multimodal data information and greatly improve the effect.

[0082] In step S103, a convolutional neural network is constructed; based on image frame groups and data packets, data with the same sequence number are concatenated and input into the model encoder part of the convolutional neural network for forward propagation to obtain preprocessed frames and event stream data to obtain feature vectors, thereby completing the encoding calculation of fused data.

[0083] In actual implementation, the embodiments of this application can construct a convolutional neural network, fuse data encoding, and, based on the image frame groups and data packets obtained from data structuring and preprocessing and fused data encoding, concatenate data with the same sequence number, such as concatenating data with the same sequence number I. k and T k T k-1 The data is concatenated and fed into the encoder part of the convolutional neural network model in the depth estimation convolutional neural network for forward propagation, resulting in preprocessed frames and event stream data to obtain feature vectors, such as F. It and F Et The encoding calculation of the fused data is completed, and then the event stream is encoded using a convolutional neural network to ensure a better effect on the denoising of the event stream data.

[0084] Optionally, in one embodiment of this application, the encoder portion of the convolutional neural network includes an input convolutional layer, a hidden convolutional layer, and an output convolutional layer. The input convolutional layer has 2*B+3*k input channels, corresponding to the combination of the preprocessed event tensor and video frames, where k is the number of channels of the image frames of the input video. The kernel size of the input convolutional layer is 3*3, the stride is 1, and the number of output channels is 16. The hidden convolutional layer has 16 input channels, a kernel size of 3*3, a stride of 1, and 16 output channels. The output convolutional layer has 16 input channels, a kernel size of 3*3, a stride of 1, and 64 output channels.

[0085] In actual implementation, embodiments of this application can construct a convolutional neural network, such as through a convolutional neural network encoder. The convolutional neural network encoder includes an input convolutional layer, a hidden convolutional layer, and an output convolutional layer. The number of input channels in the input convolutional layer is 2*B + 3*k, corresponding to the combination of preprocessed event tensors and video frames. Here, k is the number of channels of the image frames input from the video sequence and event stream data obtained from the event camera when acquiring video data and event stream data. When the video image frame is a grayscale image, k=1. When the video data and event stream data are acquired, and the image frame of the video input from the event camera is an RGB image, k=3. In this embodiment, the size of the convolutional kernel is 3×3, the stride is 1, and the number of output channels is 16. In this embodiment, the number of input channels of the hidden convolutional layer is 16, the size of the convolutional kernel is 3×3, the stride is 1, and the number of output channels is 16. In this embodiment, the number of input channels of the output convolutional layer is 16, the size of the convolutional kernel is 3×3, the stride is 1, and the number of output channels is 64.

[0086] Furthermore, the network decoder portion in this embodiment includes a coarse synthesis subnetwork and a fine-tuning subnetwork. The coarse synthesis subnetwork uses a U-Net structure with 64 input channels in the input layer and 1 output channel in the output layer. The fine-tuning subnetwork also uses a U-Net structure with 1+k input channels in the input layer and 1 output channel in the output layer. Here, k is the number of channels for acquiring video data and event stream data, specifically the number of image frames from the video sequence and event stream data acquired from the event camera. Specifically, when the image frames from the video sequence and event stream data acquired from the event camera are grayscale images, k = 1; when the image frames from the video sequence and event stream data acquired from the event camera are RGB images, k = 3.

[0087] The embodiments of this application can use convolutional neural networks to encode the event stream, ensuring a better effect on denoising the event stream data.

[0088] In step S104, the feature vectors are concatenated and forward propagation decoding calculation is performed using a multimodal fusion network decoder to generate a depth map.

[0089] In actual implementation, the embodiments of this application can integrate data decoding and depth map calculation corresponding to a single video frame, and concatenate the feature vectors, such as by combining the calculated feature vector F. It and F Et The data is stitched together, and a forward propagation decoding calculation is performed using the modal fusion network decoder obtained from the fused data encoding to generate a depth map.

[0090] For example, in this embodiment of the application, video data and event stream data can be input and acquired. The image frames of the video input from the video sequence and event stream data acquired from the event camera are RGB images, k=3. The number of input channels of the sub-network input layer is 64, the number of output channels of the output layer is 1, the number of input channels of the fine-tuning sub-network input layer is 4, and the number of output channels of the output layer is 1. By splicing feature vectors, a depth map is generated, ensuring that a relatively accurate depth map is obtained based on the video and event stream acquired by the existing event camera.

[0091] Optionally, in one embodiment of this application, concatenating feature vectors and using a multimodal fusion network decoder to perform forward propagation decoding calculations to generate a depth map includes: inputting the concatenated feature vectors into a coarse synthesis sub-network to obtain a coarse output result, wherein the coarse output result is concatenated with the input preprocessed frame and input into a fine-tuning sub-network to obtain a depth prediction result.

[0092] In some cases, embodiments of this application can use the feature vector F obtained from preprocessed video adjacent image frames and event stream tensors.It and F Et The concatenated feature vectors are input into the coarse synthesis sub-network to obtain a coarse output result. In this embodiment, the concatenated feature vectors input into the coarse synthesis sub-network to obtain a coarse output result can be combined and concatenated with the input preprocessed frame and input into the fine-tuning sub-network to obtain a depth prediction result, which facilitates the subsequent generation of a depth map and achieves the effect of depth map video generation.

[0093] In step S105, the image frame of the depth map corresponding to each video frame sequence number is calculated, and all depth maps are combined to form a depth map sequence to generate a depth image video.

[0094] In actual execution, the embodiments of this application can complete all depth map calculations. For all data combinations of data structuring and preprocessing, the calculation of all intermediate frames is completed. The image frame of the depth map corresponding to each video frame sequence number is calculated. If the input video contains N=1000 frames, it is necessary to repeatedly construct the depth estimation convolutional neural network, fuse data encoding and fusion data decoding, and calculate the depth map corresponding to a single video frame a total of 1000 times.

[0095] Furthermore, the embodiments of this application can perform data post-processing, combining all intermediate frames obtained in the completion of all depth map calculations, combining all depth maps to form a depth map sequence, realizing depth map video generation, thereby achieving the effect of depth map video generation, improving the stability of problem solving, and shortening the time spent generating depth images.

[0096] Specifically, in combination Figures 2 to 4 As shown, the working principle of the depth image and video generation method based on data fusion in this application is explained in detail with a specific embodiment.

[0097] like Figure 2As shown, the embodiments of this application can include the visualization effect of image frames before preprocessing and event packets after preprocessing. The event camera in this embodiment, as a novel visual sensor, outputs data in a completely different form than traditional cameras, making it impossible to directly apply various algorithms of traditional cameras and images. Traditional cameras acquire scene light intensity values ​​at a fixed rate (i.e., frame rate) and output image data at the same rate. The event camera, however, does not have a frame rate concept; each pixel works asynchronously. When a change in light intensity is detected, it outputs an event. Each event is a quadruple (x, y, t, p), containing the pixel's horizontal and vertical coordinates (x, y), a timestamp t, and the event polarity p (where p = -1 indicates a decrease in light intensity at the pixel, and p = 1 indicates an increase in light intensity). The event data output by all pixels are summarized to form an event list composed of individual events, serving as the event stream data output by the camera. In this embodiment, a 20-second video data acquired by a traditional camera and the corresponding event stream data output by the event camera can be displayed. Therefore, the various algorithms and methods applicable to traditional cameras and traditional image processing cannot be directly used for event cameras and event data.

[0098] Next, as Figure 3 As shown, embodiments of this application may include the following steps:

[0099] Step S301: Acquisition of video data and event stream data. This embodiment of the application can acquire raw data by obtaining video and event stream data at a certain frame rate from the event camera.

[0100] Step S302: Data structuring and preprocessing. This embodiment of the application can perform data structuring and preprocessing, combining adjacent frames in the video. Each group of frames includes the target frame for depth estimation and the preceding and following frame. For the timestamp corresponding to the middle frame of each group of video frames, the event stream data is divided into data packets with the same time interval based on the timestamp.

[0101] Step S303: Convolutional Neural Network Construction. Embodiments of this application can construct a depth estimation convolutional neural network.

[0102] Step S304: Data Encoding Calculation Fusion. This embodiment of the application can fuse data encoding. Based on data structuring and preprocessing, the image frame groups and event packets obtained from the data encoding calculation are fused, and data I with the same sequence number are... k and T k T k-1 The encoder part of the neural network model constructed by concatenating the input convolutional neural network is propagated forward to obtain the feature vector F of the preprocessed frame and event stream data. It and F Et Complete the encoding calculation of the fused data.

[0103] Step S305: Fusion of data decoding and single-depth image calculation. In this embodiment, the calculated feature vector F... It and F Et The data is stitched together and then forward-propagated using a multimodal fusion network decoder obtained from the fused data encoding to generate a depth map.

[0104] Step S306: Calculate all depth images. In this embodiment, for each input video frame sequence number, the process of repeatedly fusing data encoding, fusing data decoding, and calculating the depth map corresponding to a single video frame is performed to complete the calculation of all depth maps.

[0105] Step S307: Data post-processing. This embodiment of the application can perform data post-processing, combining all the depth maps obtained in the post-processing to form a depth map sequence, thereby generating a depth image video.

[0106] Furthermore, such as Figure 4 As shown, embodiments of this application can integrate data decoding and depth map calculation corresponding to a single video frame, and calculate the feature vector F. It and F Et The data is stitched together and then forward-propagated using a multimodal fusion network decoder obtained from the fused data encoding to generate a depth map.

[0107] In this embodiment, the input video image frames from the video sequence and event stream data acquired from the event camera can be RGB (color system) images, k=3. The number of input channels in the combined sub-network input layer is 64, the number of output channels in the output layer is 1, the number of input channels in the fine-tuning sub-network input layer is 4, and the number of output channels in the output layer is 1. The depth map generated from the video frame and event stream packet at a certain moment in this embodiment is as follows. Figure 4 As shown.

[0108] The depth image and video generation method based on data fusion proposed in this application can use event data to provide inter-frame motion information, obtain a depth map synchronized with the video frames from video data and event stream data, and use a convolutional neural network to encode the event stream, thereby achieving the effect of depth map video generation. This ensures more effective observation of high-speed scenes and improves the denoising effect of event stream data. Therefore, it solves the problems in related technologies, such as the inability to obtain a depth map synchronized with the video frames from video data and event stream data, the difficulty in achieving the effect of depth map video generation, the high dependence on hyperparameters and optimization tools in problem solving, poor stability, and the inability to effectively denoise event stream data.

[0109] Next, referring to the accompanying drawings, a depth image and video generation apparatus based on data fusion according to an embodiment of this application is described.

[0110] Figure 5 This is a schematic diagram of the structure of a depth image and video generation device based on data fusion according to an embodiment of this application.

[0111] like Figure 5 As shown, the data fusion-based depth image and video generation device 10 includes: an acquisition module 100, a combination module 200, a construction module 300, an input module 400, a first generation module 500, and a second generation module 600.

[0112] Specifically, the acquisition module 100 is used to acquire video and event stream data at a preset frame rate from the event camera.

[0113] The combination module 200 is used to combine adjacent frames in the video to obtain image frame groups. Each group of video frames in the image frame group contains the target frame for depth estimation and the frame that is adjacent to the previous and next frames. For the timestamp corresponding to the middle frame of each group of video frames, the event stream data is divided into data packets with the same time interval according to the timestamp.

[0114] Module 300 is used to build convolutional neural networks.

[0115] The input module 400 is used to concatenate data with the same sequence number based on image frame groups and data packets and input them to the model encoder part of the convolutional neural network for forward propagation to obtain preprocessed frame and event stream data to obtain feature vectors, so as to complete the encoding calculation of fused data.

[0116] The first generation module 500 is used to concatenate feature vectors and perform forward propagation decoding calculations using a multimodal fusion network decoder to generate a depth map.

[0117] The second generation module 600 is used to calculate the image frame of the depth map corresponding to each video frame sequence number, and combine all depth maps to form a depth map sequence to generate a depth image video.

[0118] Optionally, in one embodiment of this application, the combination module 200 includes: a combination unit.

[0119] The combination unit is used to combine adjacent frames in the video. Each group of frames contains the target frame for depth estimation and the frame immediately preceding and following it, denoted by R. k I represents the original video frame with sequence number k. k I represents the result after video frame preprocessing. k This represents a set of video frames corresponding to the original video frame with sequence number k, where prep() refers to the preprocessing process.

[0120] I k =prep(R k ),

[0121] I k ={I k-1 I k I k+1},

[0122] For the timestamp corresponding to the intermediate frame of each group of image frames, the event stream data is divided into data packets with the same time interval according to the timestamp. Each event e in the event stream data has corresponding x, y, p, and t attributes. Let video frame I be... k The timestamp is t k , use e i This indicates that the sequence number is i and the timestamp is t in the event stream. i A single event, E k Indicates video frame group I k The corresponding event packet, denoted as:

[0123] E k ={e i |wheret k ≤t i ≤t k+1},

[0124] To perform event stream tensor quantization calculations and preprocessing, for each event packet E k Accumulate it in the spatial domain to generate the corresponding event tensor T. k This allows it to be adapted to neural network inputs.

[0125] Optionally, in one embodiment of this application, the formula for obtaining the event stream tensor is:

[0126]

[0127] k b (a) = max(0, 1-|a|),

[0128]

[0129] Where N is the total number of frames in the input video, x i y i t i p i Event package E k Event e i The frame's two-dimensional spatial coordinates, timestamp, and polarity; B is the discretization hyperparameter; k b (a) Used for bilinear sampling of the spatiotemporal neighborhood, and the formula for obtaining the result after video frame preprocessing is:

[0130] I k =sobel(R) k ),

[0131] Among them, R k The input is the original video frame, and sobel() is the sobel operator.

[0132] Optionally, in one embodiment of this application, the encoder portion of the convolutional neural network includes an input convolutional layer, a hidden convolutional layer, and an output convolutional layer. The input convolutional layer has 2*B+3*k input channels, corresponding to the combination of the preprocessed event tensor and video frames, where k is the number of channels of the image frames of the input video. The kernel size of the input convolutional layer is 3*3, the stride is 1, and the number of output channels is 16. The hidden convolutional layer has 16 input channels, a kernel size of 3*3, a stride of 1, and 16 output channels. The output convolutional layer has 16 input channels, a kernel size of 3*3, a stride of 1, and 64 output channels.

[0133] Optionally, in one embodiment of this application, the first generation module 500 includes an input / output unit.

[0134] The input / output unit is used to input the concatenated feature vector into the coarse synthesis sub-network to obtain the coarse output result. The coarse output result is concatenated with the input preprocessed frame and input into the fine-tuning sub-network to obtain the depth prediction result.

[0135] It should be noted that the foregoing explanation of the embodiment of the depth image and video generation method based on data fusion also applies to the depth image and video generation apparatus based on data fusion in this embodiment, and will not be repeated here.

[0136] The depth image and video generation apparatus based on data fusion proposed in this application can use event data to provide inter-frame motion information, obtain a depth map synchronized with video frames from video data and event stream data, and use a convolutional neural network to encode the event stream, thereby achieving the effect of depth map video generation. This ensures more effective observation of high-speed scenes and improves the denoising effect of event stream data. Therefore, it solves the problems in related technologies, such as the inability to obtain a depth map synchronized with video frames from video data and event stream data, the difficulty in achieving depth map video generation, the high dependence on hyperparameters and optimization tools in problem solving, poor stability, and difficulty in effectively denoising event stream data.

[0137] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include:

[0138] The memory 601, the processor 602, and the computer program stored on the memory 601 and capable of running on the processor 602.

[0139] When the processor 602 executes the program, it implements the depth image and video generation method based on data fusion provided in the above embodiments.

[0140] Furthermore, electronic devices also include:

[0141] Communication interface 603 is used for communication between memory 601 and processor 602.

[0142] The memory 601 is used to store computer programs that can run on the processor 602.

[0143] The memory 601 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0144] If the memory 601, processor 602, and communication interface 603 are implemented independently, then the communication interface 603, memory 601, and processor 602 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 6 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0145] Optionally, in a specific implementation, if the memory 601, processor 602, and communication interface 603 are integrated on a single chip, then the memory 601, processor 602, and communication interface 603 can communicate with each other through an internal interface.

[0146] The processor 602 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0147] This embodiment also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for generating depth images and videos based on data fusion.

[0148] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0149] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0150] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0151] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0152] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0153] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0154] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0155] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A method for generating a depth image video based on data fusion, characterized in that, The method comprises the following steps: obtaining video and event stream data at a preset frame rate from an event camera; combining adjacent frames in the video to obtain an image frame group, wherein each group of video frames of the image frame group comprises a target frame for depth estimation and one frame adjacent before and after, and event stream data is divided into data packets with the same time interval according to time stamps corresponding to middle frames of each group of video frames; constructing a convolutional neural network; based on the image frame group and the data packet, splicing data with the same sequence number into a model encoder part of the convolutional neural network for forward propagation to obtain preprocessed frames and event stream data to obtain a feature vector to complete the encoding calculation of the fused data; splicing the feature vector and performing forward propagation decoding calculation by using a multi-modal fusion network decoder to generate a depth map; and calculating image frames of the depth map corresponding to each video frame sequence number and combining all the depth maps to form a depth map sequence to generate a depth image video.

2. The method of claim 1, wherein, The combining of adjacent frames in the video to obtain an image frame group, wherein each group of video frames of the image frame group comprises a target frame for depth estimation and one frame adjacent before and after, and event stream data is divided into data packets with the same time interval according to time stamps corresponding to middle frames of each group of video frames, comprises: combining adjacent frames in the video, each group of frames comprising a target frame with depth estimation and one frame before and after the target frame, denoted as R k represents the original video frame with sequence number k, I k represents the result of pre-processing of the video frame, I k represents a group of video frames corresponding to the original video frame with sequence number k, wherein prep() denotes the pre-processing process: I k = prep(R k ), I k = {I k-1 , I k , I k+1}, For the timestamp corresponding to the intermediate frame of each group of image frames, the event stream data is divided into data packets with the same time interval according to the timestamp. Each event e in the event stream data has corresponding x, y, p, and t attributes. Let video frame I... k The timestamp is t k , use e i This indicates that the sequence number is i and the timestamp is t in the event stream. i A single event, E k Indicates video frame group I k The corresponding event packet, denoted as: E k = {e i | where t k ≤ t i ≤ t k+1}, To perform event stream tensorization computation and preprocessing, for each event packet E k Cumulate it in the spatial domain, generating the corresponding event tensor T k Make it fit the neural network input.

3. The method of claim 2, wherein, The formula for obtaining the event stream tensor is: k b (a) = max(0, 1 - |a|), where N is the total number of frames of the input video, x i , y i , t i , p i are the two-dimensional spatial coordinates, timestamp and polarity of event e k in the event bag E i , respectively, B is a discretization hyperparameter, k b (a) for bilinear sampling of spatio-temporal neighborhoods, and the formula for obtaining the result after preprocessing the video frame is: I k = sobel(R k ), wherein R k is the input raw video frame, sobel() is a sobel operator.

4. The method of claim 1, wherein, The encoder part of the convolutional neural network comprises an input convolutional layer, a hidden convolutional layer and an output convolutional layer, wherein the input channel number of the input convolutional layer is 2*B+3*k, corresponding to the preprocessed event tensor and the video frame combination, wherein k is the channel number of the image frame of the input video, the size of the convolution kernel of the input convolutional layer is 3*3, the step is 1, and the output channel number is 16, and the input channel number of the hidden convolutional layer is 16, the size of the convolution kernel is 3*3, the step is 1, and the output channel number is 16, and the input channel number of the output convolutional layer is 16, the size of the convolution kernel is 3*3, the step is 1, and the output channel number is 64.

5. The method of claim 1, wherein, The splicing of the feature vector and the forward propagation decoding calculation by using the multi-modal fusion network decoder to generate the depth map comprises: inputting the spliced feature vector into a coarse synthesis subnetwork to obtain a coarse output result, wherein the coarse output result is combined and spliced with the input preprocessed frame to be input into a fine tuning subnetwork to obtain a depth prediction result.

6. A data fusion based depth image video generation apparatus, characterized by, It comprises: an acquisition module for obtaining video and event stream data at a preset frame rate from an event camera; a combination module for combining adjacent frames in the video to obtain an image frame group, wherein each group of video frames of the image frame group comprises a target frame for depth estimation and one frame adjacent before and after, and event stream data is divided into data packets with the same time interval according to time stamps corresponding to middle frames of each group of video frames; a construction module for constructing a convolutional neural network; The input module is configured to input data with the same sequence number to a model encoder part of the convolutional neural network based on the image frame group and the data packet for forward propagation to obtain a preprocessed frame and event stream data to obtain a feature vector to complete encoding calculation of the fused data. The first generation module is configured to splice the feature vector and perform forward propagation decoding calculation by using a multi-modal fusion network decoder to generate a depth map. The second generation module is configured to calculate image frames of the depth map corresponding to each video frame sequence number and combine all the depth maps to form a depth map sequence to generate a depth image video.

7. The apparatus of claim 6, wherein, The combination module includes: a combining unit for combining adjacent frames in the video, each group of frames comprising a target frame with depth estimation and one frame before and after the target frame, denoted as R k represents the original video frame with sequence number k, I k represents the result of pre-processing of the video frame, I k represents a group of video frames corresponding to the original video frame with sequence number k, wherein prep() denotes the pre-processing process: I k = prep(R k ), I k = {I k-1 , I k , I k+1}, For the timestamp corresponding to the intermediate frame of each group of image frames, the event stream data is divided into data packets with the same time interval according to the timestamp. Each event e in the event stream data has corresponding x, y, p, and t attributes. Let video frame I... k The timestamp is t k , use e i This indicates that the sequence number is i and the timestamp is t in the event stream. i A single event, E k Indicates video frame group I k The corresponding event packet, denoted as: E k = {e i | where t k ≤ t i ≤ t k+1}, To perform event stream tensorization computation and pre-processing, for each event packet E k Cumulate it in the spatial domain, generating the corresponding event tensor T k Make it fit the neural network input.

8. The apparatus of claim 7, wherein, An acquisition formula of the event stream tensor is: k b (a) = max(0, 1 - |a|), where N is the total number of frames of the input video, x i , y i , t i , p i are the two-dimensional spatial coordinates, timestamp and polarity of event e k frame in the event bag E i , respectively, B is the discretization hyperparameter, k b (a) for bilinear sampling of spatio-temporal neighborhoods, An acquisition formula of a result of the preprocessed video frame is: I k = sobel(R k ), wherein R k is the input raw video frame, sobel() is a sobel operator.

9. An electronic device, comprising: includes: A memory, a processor, and a computer program stored on the memory and executable on the processor, the processor executing the program to implement the depth image video generation method based on data fusion according to any one of claims 1-5.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the depth image video generation method based on data fusion according to any one of claims 1-5.

Images