Depth image video generation method and device, electronic equipment and storage medium

Abstract

The application relates to a depth image video generation method and device, electronic equipment and a storage medium, wherein the method comprises: acquiring a video with a preset frame rate and event stream data; dividing the event stream data according to an exposure interval of the video to obtain event data packets; in a training branch, inputting the event data packets into a model encoder part of a pre-constructed convolutional neural network for forward propagation; and in an inference branch, inputting the event data packets into a model encoder part of an optimized convolutional neural network for forward propagation. Thus, the problems in the prior art that it is difficult to collect depth image annotation data corresponding to the event stream from a real scene to train the convolutional neural network, it is difficult to obtain a depth map synchronized with a video frame from the event stream data, it is difficult to realize the effect of depth map video generation, the problem solving highly depends on hyperparameters and optimization solving tools, the stability is poor, and it is difficult to better play the effect of event stream data denoising are solved.

Description

Technical Field

[0001] This application relates to the fields of computer vision and neuromorphic computing technology, and in particular to a method, apparatus, electronic device and storage medium for generating depth images and videos. Background Technology

[0002] An event camera is a bio-inspired sensor that operates on a significantly different principle from traditional cameras. Unlike traditional cameras that acquire absolute light intensity at a fixed frame rate, an event camera outputs data only when the light intensity of a branch changes; this output data is called an event stream. Compared to traditional cameras, event cameras offer advantages such as high dynamic range, high temporal resolution, and no motion blur.

[0003] In the field of depth estimation based on event cameras, existing methods based on handcrafted models establish a matching relationship between event signals and pixel depths, and then solve the problem using hyperparameters and optimization tools. Neural network-based methods, inspired by traditional RGB vision algorithms, can, to some extent, solve some vision problems in the event camera domain in a similar manner. However, due to significant differences in the characteristics of the two types of cameras and their main application branches, neural network-based methods in the event camera field still face various challenges.

[0004] However, in related technologies, neural networks trained in the traditional way require a large amount of data with depth annotations. In the high-speed motion branch adapted by event cameras, the acquisition of depth data is relatively difficult. As a result, in the field of depth estimation of event cameras, neural network-based methods often cannot obtain sufficient real data for training, while models trained based on simulation data are difficult to adapt to real branches. This problem urgently needs to be solved. Summary of the Invention

[0005] This application provides a method, apparatus, electronic device, and storage medium for generating depth images and videos, in order to solve the problems in related technologies, such as the inability to obtain depth maps synchronized with video frames from video data and event stream data, the difficulty in achieving the effect of depth map video generation, the high dependence on hyperparameters and optimization tools in problem solving, poor stability, and the difficulty in achieving better denoising effects on event stream data.

[0006] The first aspect of this application provides a method for generating depth image video, comprising the following steps: acquiring video and event stream data at a preset frame rate; dividing the event stream data according to the exposure interval of the video to obtain event data packets, and combining each video frame in the video with its preceding and following video frames, assigning a number to the frame combination corresponding to each video frame according to the temporal sequence of the video frames, wherein the event data packets within the corresponding interval of the video image of each video frame and the video image of the preceding video frame are given the same sequence number; in the training branch, inputting the event data packets into the encoder part of a pre-built convolutional neural network model for forward propagation to obtain... The first feature vector is used, and it is fed into two decoders for decoding to generate a first depth map and a relative pose estimate. A reprojection loss is calculated using video frame groups with the same sequence number as the event data packet, based on the first depth map and the relative pose. Backpropagation of this loss optimizes the neural network parameters. In the inference branch, the event data packet is input into the encoder part of the optimized convolutional neural network for forward propagation to obtain a second feature vector. This second feature vector is then fed into the depth decoder for decoding to generate a second depth map. Image frames corresponding to each sequence number of the depth map are calculated to form a depth map sequence, thereby generating a depth image video.

[0007] Optionally, in one embodiment of this application, the step of dividing the event stream data according to the exposure interval of the video to obtain event data packets, and combining each video frame in the video with its preceding and following video frames, includes: each group of video frames based on the image frame group contains a target frame for depth estimation and the preceding and following video frames; for the timestamp corresponding to the middle frame of each group of video frames, the event stream data is divided into data packets with the same time interval according to the timestamp.

[0008] Optionally, in one embodiment of this application, the step of dividing the event stream data according to the exposure interval of the video to obtain event data packets, and combining each video frame in the video with its preceding and following video frames, further includes: performing event stream tensor quantization calculation and preprocessing based on the data packets, accumulating each event packet of the data packets in the spatial domain to generate a corresponding event tensor, so as to adapt it to the neural network input.

[0009] Optionally, in one embodiment of this application, the formula for obtaining the event tensor is:

[0010]

[0011] k b (a) = max(0, 1-|a|),

[0012]

[0013] Where N is the total number of frames in the input video, x i y i t i p i Event package E k Event e i The frame's two-dimensional spatial coordinates, timestamp, and polarity; B is the discretization hyperparameter; k b (a) Used for bilinear sampling of spatiotemporal neighborhoods.

[0014] Optionally, in one embodiment of this application, the formula for calculating the reprojection loss is:

[0015] L p =∑ k′ pe(I k I k′→t ),

[0016] I k′→k =I t′ <proj(D k P k→k′ ,K)>,

[0017]

[0018] Where k represents the target frame number, k′ represents the reference frame number, and the target frame corresponds to I. k ={I k-1 I k I k+1 The I in} k Reference frame corresponds to I k+1 with I k-1 D t Represented by the event tensor T k The depth map generated after decoding the encoded features, the proj() operation represents the current index of the depth map D. k Based on the camera intrinsic parameter matrix K and the relative pose P t→t′ SSIM() represents the structural similarity loss when projected onto the image plane of the reference frame.

[0019] A second aspect of this application provides a depth image video generation apparatus, comprising: an acquisition module for acquiring video and event stream data at a preset frame rate; a combination module for dividing the event stream data into event data packets based on the exposure interval of the video, and combining each video frame in the video with its preceding and following video frames, assigning a number to the frame combination corresponding to each video frame according to the temporal sequence of the video frames, wherein the event data packets within the corresponding interval of the video image of each video frame and the video image of the preceding video frame are assigned the same sequence number; and a first generation module for inputting the event data packets into the encoder part of a pre-built convolutional neural network model for forward feeding in a training branch. The system performs forward propagation to obtain a first feature vector, which is then fed into two decoders for decoding to generate a first depth map and a relative pose estimate. A reprojection loss is calculated using video frame groups with the same sequence number as the event data packet, based on the first depth map and the relative pose, and the loss is used for backpropagation to optimize the neural network parameters. A second generation module, in the inference branch, inputs the event data packet into the encoder part of the optimized convolutional neural network model for forward propagation to obtain a second feature vector. This second feature vector is then fed into the depth decoder for decoding to generate a second depth map. Image frames corresponding to each sequence number of the depth map are calculated to form a depth map sequence, thereby generating a depth image video.

[0020] Optionally, in one embodiment of this application, the combining module includes: a partitioning unit, configured to partition the event stream data into data packets with the same time interval based on the timestamp corresponding to the timestamp of the intermediate frame of each group of video frames containing the target frame for depth estimation and the adjacent video frames before and after it, according to the timestamp of the intermediate frame of each group of video frames.

[0021] Optionally, in one embodiment of this application, the combining module further includes: a generation unit, used to perform event stream tensor quantization calculation and preprocessing based on the data packet, accumulate each event packet of the data packet in the spatial domain, generate a corresponding event tensor, and adapt it to the neural network input.

[0022] Optionally, in one embodiment of this application, the formula for obtaining the event tensor is:

[0023]

[0024] k b (a) = max(0, 1-|a|),

[0025]

[0026] Where N is the total number of frames in the input video, x i y it i p i Event package E k Event e i The frame's two-dimensional spatial coordinates, timestamp, and polarity; B is the discretization hyperparameter; k b (a) Used for bilinear sampling of spatiotemporal neighborhoods.

[0027] Optionally, in one embodiment of this application, the formula for calculating the reprojection loss is:

[0028] L p =∑ k′ pe(I k I k′→t ),

[0029] I k′→k =I t′ <proj(D k P k→k′ ,K)>,

[0030]

[0031] Where k represents the target frame number, k′ represents the reference frame number, and the target frame corresponds to I. k ={I k-1 I k I k+1 The I in} k Reference frame corresponds to I k+1 with I k-1 D t Represented by the event tensor T k The depth map generated after decoding the encoded features, the proj() operation represents the current index of the depth map D. k Based on the camera intrinsic parameter matrix K and the relative pose P t→t′ SSIM() represents the structural similarity loss when projected onto the image plane of the reference frame.

[0032] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the depth image video generation method as described in the above embodiments.

[0033] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the depth image and video generation method described above.

[0034] This application's embodiments can obtain depth maps synchronized with video frames from video data and event stream data, achieving the effect of depth map video generation. It exhibits strong stability and effectively denoises event stream data. Therefore, it solves the problems in related technologies, such as the inability to obtain depth maps synchronized with video frames from video data and event stream data, difficulty in achieving depth map video generation, high dependence on hyperparameters and optimization tools in problem solving, poor stability, and difficulty in effectively denoising event stream data.

[0035] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0036] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0037] Figure 1 This is a flowchart of a depth image video generation method provided according to an embodiment of this application;

[0038] Figure 2 This is a streaming data visualization diagram of a depth image video generation method according to an embodiment of this application;

[0039] Figure 3 This is a depth map visualization diagram of a depth image video generation method according to an embodiment of this application;

[0040] Figure 4 This is a flowchart of a depth image video generation method according to an embodiment of this application;

[0041] Figure 5 This is a schematic diagram of the structure of a depth image and video generation apparatus according to an embodiment of this application;

[0042] Figure 6 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of this application. Detailed Implementation

[0043] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0044] The following description, with reference to the accompanying drawings, outlines a depth image and video generation method, apparatus, electronic device, and storage medium according to embodiments of this application. Addressing the issues raised in the background section regarding the inability to obtain a depth map synchronized with video frames from video data and event stream data, hindering the generation of depth map videos, and exhibiting high dependence on hyperparameters and optimization tools in problem-solving, resulting in poor stability and difficulty in effectively denoising event stream data, this application provides a depth image and video generation method. This method can obtain a depth map synchronized with video frames from video data and event stream data, achieving the effect of depth map video generation, exhibiting strong stability, and effectively denoising event stream data. Therefore, it solves the problems in the related technologies, such as the inability to obtain a depth map synchronized with video frames from video data and event stream data, the difficulty in generating depth map videos, the high dependence on hyperparameters and optimization tools in problem-solving, poor stability, and difficulty in effectively denoising event stream data.

[0045] Specifically, Figure 1 This is a schematic flowchart of a depth image video generation method provided in an embodiment of this application.

[0046] like Figure 1 As shown, the depth image video generation method includes the following steps:

[0047] In step S101, video and event stream data at a preset frame rate are acquired.

[0048] In actual implementation, the embodiments of this application can obtain video and event stream data at a preset frame rate from the event camera, which provides support for obtaining a depth map synchronized with the video frame from the event stream data. This helps to achieve the effect of generating depth map video and better achieve the effect of denoising event stream data.

[0049] It should be noted that the preset frame rate can be set by those skilled in the art according to the actual situation, and no specific restrictions are imposed here.

[0050] In step S102, the event stream data is divided according to the exposure interval of the video to obtain event data packets. Each video frame in the video is combined with its preceding and following video frames. Each video frame is assigned a number according to the time sequence of the video frames. The event data packets in the interval corresponding to the video image of each video frame and the video image of the previous video frame are given the same sequence number.

[0051] In actual implementation, the embodiments of this application can perform data preprocessing, divide the event stream data according to the video exposure interval to obtain event data packets, and combine each video frame in the video with its preceding and following video frames. Each video frame is assigned a number according to the temporal sequence of the video frames. In the embodiments of this application, the event data packets in the corresponding interval of each video frame and the video image of the previous video frame are given the same sequence number, thereby ensuring that the depth image annotation data corresponding to the event stream is collected from the real scene, providing a basis for training the convolutional neural network. It does not rely on hyperparameters and optimization tools and has strong stability.

[0052] Optionally, in one embodiment of this application, the event stream data is divided according to the exposure interval of the video to obtain event data packets, and each video frame in the video is combined with its preceding and following video frames, including: each group of video frames based on the image frame group contains the target frame for depth estimation and the preceding and following video frames, and for the timestamp corresponding to the middle frame of each group of video frames, the event stream data is divided into data packets with the same time interval according to the timestamp.

[0053] In some embodiments, adjacent frames in the video can be combined, with each group of video frames containing the target frame for depth estimation and the preceding and following video frames, denoted by I. k I represents the original video frame with sequence number k. k This represents a group of video frames corresponding to the original video frame with sequence number k:

[0054] I k ={I k-1 I k I k+1},

[0055] For the timestamp corresponding to the intermediate frame of each group of video frames, the event stream data is divided into data packets with the same time interval according to the timestamp. Each event e in the event stream data has corresponding x, y, p, and t attributes. Let video frame I... k The timestamp is t k , use e i This indicates that the sequence number is i and the timestamp is t in the event stream. i A single event, E k Indicates video frame group I k The corresponding event packet, denoted as:

[0056] E k ={e i |wheret k-1 ≤t i ≤t k},

[0057] The embodiments of this application can divide event stream data into data packets with the same time interval according to timestamps, thereby ensuring a better effect on denoising event stream data.

[0058] Optionally, in one embodiment of this application, the event stream data is divided according to the exposure interval of the video to obtain event data packets, and each video frame in the video is combined with its preceding and following video frames. The method further includes: performing event stream tensor quantization calculation and preprocessing based on the data packets, accumulating each event packet of the data packets in the spatial domain to generate a corresponding event tensor, and adapting it to the neural network input.

[0059] As one possible implementation, embodiments of this application can perform event stream tensor quantization calculation and preprocessing based on data packets, and perform event packet E on each data packet. k Accumulate it in the spatial domain to generate the corresponding event tensor T. k This allows it to adapt to neural network inputs, thus enabling unsupervised or self-supervised methods that do not rely on labeled data, thereby improving stability.

[0060] Optionally, in one embodiment of this application, the formula for obtaining the event tensor is:

[0061]

[0062] k b (a) = max(0, 1-|a|),

[0063]

[0064] Where N is the total number of frames in the input video, x i y i t i p i Event package E k Event e i The frame's two-dimensional spatial coordinates, timestamp, and polarity; B is the discretization hyperparameter; k b (a) Used for bilinear sampling of spatiotemporal neighborhoods.

[0065] In actual implementation, the embodiments of this application can obtain the event tensor using the following formula:

[0066]

[0067] k b (a) = max(0, 1-|a|),

[0068]

[0069] To improve the accuracy of calculations, the depth map generated from video frames and event stream packets at a certain moment is as follows: Figure 2 and Figure 3 As shown, this can further enhance the noise reduction effect on event stream data.

[0070] In step S103, in the training branch, the event data packet is input to the model encoder part of the pre-built convolutional neural network for forward propagation to obtain the first feature vector. The first feature vector is then sent to two decoders for decoding to generate the first depth map and relative pose estimation. The reprojection loss is calculated based on the first depth map and relative pose using video frame groups with the same sequence number as the event data packet. The neural network parameters are then optimized by backpropagation of the loss.

[0071] In actual implementation, the embodiments of this application can first construct a convolutional neural network. The encoder part of the convolutional neural network includes an input convolutional layer, a hidden convolutional layer, and an output convolutional layer. The input convolutional layer has B input channels, corresponding to the number of channels in the event tensor Tk. The kernel size of the input convolutional layer is 3*3, the stride is 1, and the number of output channels is 16. The hidden convolutional layer has 16 input channels, a kernel size of 3*3, a stride of 1, and 16 output channels. The output convolutional layer has 16 input channels, a kernel size of 3*3, a stride of 1, and 64 output channels. In the network decoder part, the depth decoder includes an input layer, a hidden convolutional layer, an upsampling layer, and an output convolutional layer. The input layer has 64 input channels, and the output layer has 1 output channel. Compared to the depth decoder, the pose decoder adds an additional global average pooling layer before the output layer, with 6 output channels.

[0072] Furthermore, in the training branch, the event data packet is input into the encoder part of the pre-built convolutional neural network model for forward propagation to obtain a first feature vector. The first feature vector is then sent to two decoders for decoding to generate a first depth map and relative pose estimation. Using video frame groups with the same sequence number as the event data packet, the reprojection loss is calculated based on the first depth map and relative pose. The calculated loss is then used to optimize the neural network parameters through a backpropagation algorithm, thereby obtaining a depth map synchronized with the video frames from the video data and event stream data, effectively achieving the effect of depth map video generation.

[0073] Optionally, in one embodiment of this application, the formula for calculating the reprojection loss is:

[0074] L p =∑ k′ pe(I k I k′→t ),

[0075] I k′→k =It′ <proj(D k P k→k′ ,K)>,

[0076]

[0077] Where k represents the target frame number, k′ represents the reference frame number, and the target frame corresponds to I. k ={I k-1 I k I k+1 The I in} k Reference frame corresponds to I k+1 with I k-1 D t Represented by the event tensor T k The depth map generated after decoding the encoded features, the proj() operation represents the current index of the depth map D. k Based on the camera intrinsic parameter matrix K and the relative pose P t→t′ SSIM() represents the structural similarity loss when projected onto the image plane of the reference frame.

[0078] In actual implementation, the embodiments of this application can be calculated using the following formula for reprojection loss:

[0079] L p =∑ k′ pe(I k I k′→t ),

[0080] I k′→k =I t′ <proj(D k P k→k′ ,K)>,

[0081]

[0082] Improve the accuracy of calculations to further enhance the generation of depth map videos.

[0083] In step S104, in the inference branch, the event data packet is input to the model encoder part of the optimized convolutional neural network for forward propagation to obtain the second feature vector, and the second feature vector is sent to the depth decoder for decoding to generate the second depth map. The image frame of the depth map corresponding to each sequence number is calculated to form a depth map sequence to generate a depth image video.

[0084] Specifically, in the inference branch, the event data packet is input into the encoder part of the optimized convolutional neural network model for forward propagation to obtain a second feature vector. The second feature vector is then sent to the depth decoder for decoding to generate a second depth map. The image frames of the depth map corresponding to each sequence number are calculated to form a depth map sequence to generate a depth image video. This ensures that a depth map synchronized with the video frame is obtained from the video data and event stream data, achieving the effect of depth map video generation and better denoising the event stream data.

[0085] Specifically, it can be combined with Figure 4 As shown, the working principle of the depth image and video generation method of this application is explained in detail with a specific embodiment.

[0086] like Figure 4 As shown, embodiments of this application may include the following steps:

[0087] Step S401: Acquisition of video data and event stream data.

[0088] In this embodiment of the application, video and event stream data at a preset frame rate can be obtained from the event camera.

[0089] Step S402: Data structuring and preprocessing.

[0090] In this embodiment, data structuring and preprocessing can be performed. Event stream data is divided according to the exposure interval of the video to obtain event data packets. Each video frame in the video is combined with its preceding and following video frames. Each video frame is assigned a number according to the time sequence of the video frames. The event data packets in the interval corresponding to the video image of each video frame and the video image of the previous video frame are given the same sequence number.

[0091] Step S403: Convolutional Neural Network Construction.

[0092] In this embodiment, a convolutional neural network can be constructed first. The encoder part of the convolutional neural network includes an input convolutional layer, a hidden convolutional layer, and an output convolutional layer.

[0093] Step S404: Event data training encoding calculation.

[0094] In this embodiment of the application, event data training encoding calculation can be performed in the training branch.

[0095] Step S405: Decode the data to calculate the depth map and relative pose.

[0096] In this embodiment of the application, the first feature vector can be fed into two decoders for decoding to generate a first depth map and a relative pose estimate.

[0097] Step S406: Calculate the loss by fusing video frames and backpropagate.

[0098] In this embodiment, a video frame group with the same sequence number as the event data packet can be used to calculate the reprojection loss based on the first depth map and relative pose. The calculated loss is then used to optimize the neural network parameters through a backpropagation algorithm.

[0099] Step S407: Event data reasoning and encoding calculation.

[0100] In this embodiment of the application, event data reasoning and encoding calculations can be performed in the reasoning branch.

[0101] Step S408: Decode the data and calculate the depth map.

[0102] In this embodiment of the application, a depth map can be calculated by data decoding, and the second feature vector can be sent to a depth decoder for decoding to generate a second depth map.

[0103] Step S409: Combine all depth maps to obtain a depth video.

[0104] In this embodiment, all depth maps can be combined to obtain a depth video. By calculating the image frames of the depth map corresponding to each sequence number, a depth map sequence is formed to generate a depth image video.

[0105] The depth image video generation method proposed in this application can obtain a depth map synchronized with video frames from video data and event stream data, achieving the effect of depth map video generation. It exhibits strong stability and can better denoise event stream data. This solves the problems in related technologies, such as the inability to obtain a depth map synchronized with video frames from video data and event stream data, the difficulty in achieving depth map video generation, the high dependence on hyperparameters and optimization tools in problem solving, poor stability, and the inability to effectively denoise event stream data.

[0106] Next, the depth image and video generation apparatus according to embodiments of this application is described with reference to the accompanying drawings.

[0107] Figure 5 This is a schematic diagram of the structure of a depth image and video generation apparatus according to an embodiment of this application.

[0108] like Figure 5 As shown, the depth image and video generation device 10 includes: an acquisition module 100, a combination module 200, a first generation module 300, and a second generation module 400.

[0109] Specifically, the acquisition module 100 is used to acquire video and event stream data at a preset frame rate.

[0110] The combination module 200 is used to divide the event stream data according to the video exposure interval to obtain event data packets, and to combine each video frame in the video with its preceding and following video frames. Each video frame is assigned a number according to the time sequence of the video frames. The event data packets in the interval corresponding to the video image of each video frame and the video image of the previous video frame are given the same sequence number.

[0111] The first generation module 300 is used in the training branch to input the event data packet into the model encoder part of the pre-built convolutional neural network for forward propagation to obtain the first feature vector, and to send the first feature vector into two decoders for decoding to generate the first depth map and relative pose estimation, and to calculate the reprojection loss based on the first depth map and relative pose using video frame groups with the same sequence number as the event data packet, and to optimize the neural network parameters by backpropagation of the loss.

[0112] The second generation module 400 is used in the inference branch to input the event data packet into the model encoder part of the optimized convolutional neural network for forward propagation to obtain the second feature vector, and to send the second feature vector into the depth decoder for decoding to generate the second depth map. It also calculates the image frame of the depth map corresponding to each sequence number to form a depth map sequence to generate a depth image video.

[0113] Optionally, in one embodiment of this application, the combining module 200 includes: a partitioning unit.

[0114] The segmentation unit is used to divide the event stream data into data packets with the same time interval based on the timestamp corresponding to the timestamp of each group of video frames, which includes the target frame for depth estimation and the adjacent video frames before and after it, based on the timestamp of the middle frame of each group of video frames.

[0115] Optionally, in one embodiment of this application, the combination module 200 further includes a generation unit.

[0116] The generation unit is used to perform event stream tensor quantization calculation and preprocessing based on data packets. It accumulates each event packet of the data packet in the spatial domain to generate the corresponding event tensor, which is then adapted to the neural network input.

[0117] Optionally, in one embodiment of this application, the formula for obtaining the event tensor is:

[0118]

[0119] k b (a) = max(0, 1-|a|),

[0120]

[0121] Where N is the total number of frames in the input video, x i y i t i p i Event package E k Event e i The frame's two-dimensional spatial coordinates, timestamp, and polarity; B is the discretization hyperparameter; k b (a) Used for bilinear sampling of spatiotemporal neighborhoods.

[0122] Optionally, in one embodiment of this application, the formula for calculating the reprojection loss is:

[0123] L p =∑ k′ pe(I k I k′→t ),

[0124] I k′→k =I t′ <proj(D k P k→k′ ,K)>,

[0125]

[0126] Where k represents the target frame number, k′ represents the reference frame number, and the target frame corresponds to I. k ={I k-1 I k I k+1 The I in} k Reference frame corresponds to I k+1 with I k-1 D t Represented by the event tensor T k The depth map generated after decoding the encoded features, the proj() operation represents the current index of the depth map D. k Based on the camera intrinsic parameter matrix K and the relative pose P t→t′ SSIM() represents the structural similarity loss when projected onto the image plane of the reference frame.

[0127] It should be noted that the foregoing explanation of the embodiment of the depth image and video generation method also applies to the depth image and video generation apparatus of this embodiment, and will not be repeated here.

[0128] The depth image video generation apparatus proposed in this application can obtain a depth map synchronized with video frames from video data and event stream data, achieving the effect of depth map video generation. It exhibits strong stability and effectively denoises event stream data. This solves the problems in related technologies where it is impossible to obtain a depth map synchronized with video frames from video data and event stream data, making it difficult to achieve the effect of depth map video generation. Furthermore, these technologies heavily rely on hyperparameters and optimization tools for problem-solving, resulting in poor stability and difficulty in effectively denoising event stream data.

[0129] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include:

[0130] The memory 601, the processor 602, and the computer program stored on the memory 601 and capable of running on the processor 602.

[0131] When the processor 602 executes the program, it implements the depth image and video generation method provided in the above embodiments.

[0132] Furthermore, electronic devices also include:

[0133] Communication interface 603 is used for communication between memory 601 and processor 602.

[0134] The memory 601 is used to store computer programs that can run on the processor 602.

[0135] The memory 601 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0136] If the memory 601, processor 602, and communication interface 603 are implemented independently, then the communication interface 603, memory 601, and processor 602 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 6 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0137] Optionally, in a specific implementation, if the memory 601, processor 602, and communication interface 603 are integrated on a single chip, then the memory 601, processor 602, and communication interface 603 can communicate with each other through an internal interface.

[0138] The processor 602 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0139] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the depth image and video generation method described above.

[0140] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0141] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0142] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0143] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0144] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0145] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0146] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0147] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A method for generating depth image video, characterized in that, Includes the following steps: Acquire video and event stream data at a preset frame rate; The event stream data is divided according to the exposure interval of the video to obtain event data packets. Each video frame in the video is combined with its preceding and following video frames. The frame combination corresponding to each video frame is assigned a number according to the time sequence of the video frames. The event data packets within the interval corresponding to the video image of each video frame and the video image of the previous video frame are given the same sequence number. In the training branch, the event data packet is input into the encoder part of the pre-built convolutional neural network for forward propagation to obtain a first feature vector. This first feature vector is then fed into two decoders for decoding, generating a first depth map and a relative pose estimate. Using video frame groups with the same sequence number as the event data packet, a reprojection loss is calculated based on the first depth map and the relative pose. Backpropagation of this loss optimizes the neural network parameters. In the inference branch, the event data packet is input to the model encoder part of the optimized convolutional neural network for forward propagation to obtain the second feature vector. The second feature vector is then sent to the depth decoder for decoding to generate the second depth map. The image frames of the depth map corresponding to each sequence number are calculated to form a depth map sequence to generate a depth image video. The step of dividing the event stream data according to the exposure interval of the video to obtain event data packets, and combining each video frame in the video with its preceding and following video frames, includes: each video frame based on the image frame group contains a target frame for depth estimation and the preceding and following video frames; for the timestamp corresponding to the middle frame of each video frame, the event stream data is divided into event data packets with the same time interval according to the timestamp. The step of dividing the event stream data according to the exposure interval of the video to obtain event data packets, and combining each video frame in the video with its preceding and following video frames, further includes: performing event stream tensor quantization calculation and preprocessing based on the data packets, accumulating each event data packet of the data packets in the spatial domain to generate a corresponding event tensor, and adapting it to the neural network input; The formula for obtaining the event tensor is: ， ， ， in, N The total number of frames in the input video. x i , y i , t i , p i Event data packet E k Event e i The frame's two-dimensional spatial coordinates, timestamp, and polarity. B To discretize hyperparameters, Used for bilinear sampling of spatiotemporal neighborhoods.

2. The method according to claim 1, characterized in that, The formula for calculating the reprojection loss is: ， ， ， in, Represents the target frame number. Represents the reference frame number, corresponding to the target frame. In Reference frame corresponding and , Represented by event tensor The depth map generated after decoding the encoded features The operation represents the depth map of the current index. Based on the camera intrinsic parameter matrix and relative pose Projected onto the image plane of the reference frame This represents the structural similarity loss.

3. A depth image / video generation device, characterized in that, include: The acquisition module is used to acquire video and event stream data at a preset frame rate; The combination module is used to divide the event stream data according to the exposure interval of the video to obtain event data packets, and to combine each video frame in the video with its preceding and following video frames, and to assign a number to the frame combination corresponding to each video frame according to the time sequence of the video frames, wherein the event data packets in the interval corresponding to the video image of each video frame and the video image of the previous video frame are given the same sequence number. A first generation module is configured to, during the training branch, input the event data packet into the encoder part of a pre-built convolutional neural network for forward propagation to obtain a first feature vector, and then feed the first feature vector into two decoders for decoding to generate a first depth map and a relative pose estimate. Additionally, using video frame groups with the same sequence number as the event data packet, a reprojection loss is calculated based on the first depth map and the relative pose, and the neural network parameters are optimized through backpropagation of the loss. The second generation module is used to input the event data packet into the model encoder part of the optimized convolutional neural network for forward propagation in the inference branch to obtain the second feature vector, and send the second feature vector into the depth decoder for decoding to generate the second depth map, and calculate the image frame of the depth map corresponding to each sequence number to form a depth map sequence to generate a depth image video. The combination module includes a partitioning unit, which is used to partition the event stream data into event data packets with the same time interval based on the timestamp corresponding to the timestamp of the intermediate frame of each video frame in the image frame group, which includes a target frame for depth estimation and the adjacent video frames before and after it. The combined module further includes: a generation unit, used for event stream tensor quantization calculation and preprocessing based on data packets, accumulating each event data packet of the data packet in the spatial domain to generate a corresponding event tensor, so as to adapt it to the neural network input; The formula for obtaining the event tensor is: ， ， ， in, N The total number of frames in the input video. x i , y i , t i , p i Event data packet E k Event e i The frame's two-dimensional spatial coordinates, timestamp, and polarity. B To discretize hyperparameters, Used for bilinear sampling of spatiotemporal neighborhoods.

4. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to implement the depth image video generation method as described in any one of claims 1-2.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the depth image video generation method as described in any one of claims 1-2.

Images