Large visual model-based sparse point cloud guided video depth prediction method

Abstract

The invention discloses a sparse point cloud guide video depth prediction method based on a visual large model. The method aims to guide a visible light video to carry out depth prediction by means of a sparse point cloud sequence collected by high-precision radar equipment. A pair of visible light videos and a sparse point cloud sequence collected by radar equipment are given, and the method comprises the following steps: firstly, generating an accurate low-resolution video depth through a depth completion module guided by a visual large model; and then generating a high-resolution video depth with stable time domain and rich details through a high-resolution depth reasoning module based on a visual large model. Visible light video features of scene robustness are extracted through generalization ability of a visual large model, and scale information is injected into the visible light video features in a targeted manner from a sparse point cloud sequence collected by radar equipment through designing a local least square method based on a space-time neighborhood and a feature fusion module of time domain alignment. And a high-precision video depth prediction result with a stable time domain is realized.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and in particular relates to a method for predicting video depth based on sparse point clouds using a large visual model. Background Technology

[0002] Depth estimation is a fundamental task in computer vision, and accurate depth information is crucial for downstream tasks such as autonomous driving, 3D reconstruction, and robot manipulation. The applications of depth estimation are diverse, encompassing a wide range of indoor and outdoor scenes, which places high demands on the generalization ability of depth estimation methods. Large-scale visual models, trained on large-scale simulation and real-world datasets, possess strong generalization capabilities and can meet the scene robustness requirements of depth estimation. Currently, commonly used depth estimation methods based on large-scale visual models can only handle visible light image input. Due to the lack of scale information, these methods can only estimate relative depth, i.e., the relative positional relationships of objects in the scene, and cannot obtain accurate positional coordinates, which limits their application in downstream fields. Furthermore, many models cannot directly process visible light video input. The processing requires splitting the image into consecutive image frames, performing inference on each frame, and then stitching them together to reconstruct the video depth. Video depths generated using this method often lack temporal stability, exhibiting significant jitter in the temporal domain, affecting the quality of the generated depth.

[0003] Currently, mainstream depth estimation methods based on large visual models include Depth Anything V2, VideoDepth Anything, and Prompt Depth Anything.

[0004] Depth Anything V2 uses a visual Transformer and a depth prediction network as the encoder and decoder, respectively, to predict the relative depth of a scene. The method first trains a teacher model with a large number of parameters on a high-quality synthetic dataset. Then, this model is used to generate accurate pseudo-depth labels on a large scale of unlabeled real images, which are then used to train a student model with fewer parameters, thus ensuring the accuracy and generalization ability of the trained model. Although the network can fine-tune the output absolute depth on small datasets, this method is still limited to a single visible light image input and cannot acquire and predict accurate scale information. Furthermore, this method cannot directly process visible light video input, resulting in some temporal jitter in the predicted depth.

[0005] Video Depth Anything incorporates a temporal module into the depth prediction network of Depth Anything V2 to extract temporally consistent features, thereby ensuring temporal consistency in the generated depth. This method, trained on large-scale video depth datasets, achieves high accuracy and good temporal consistency in prediction results. However, similar to Depth Anything V2, this method is limited by a single visible light video input and cannot acquire and predict accurate scale information.

[0006] Prompt Depth Anything extracts depth features at different resolutions from low-resolution depth data and superimposes them onto the features extracted step-by-step by the Depth Anything V2 depth prediction network to obtain accurate scale information. Guided by low-resolution depth, this method achieves high accuracy. However, in real-world applications, some sensors can only acquire sparse depth data, and this method does not provide a robust low-resolution depth generation approach, limiting its application scenarios. Furthermore, similar to Depth Anything V2, this method cannot directly process visible light video input; temporal jitter affects the quality of the output depth. Summary of the Invention

[0007] To address the shortcomings of the aforementioned background technology, this invention provides a sparse point cloud-guided video depth prediction method based on a large visual model. This method processes sparse point cloud sequences acquired by visible light video and radar equipment, which can significantly improve the accuracy, temporal stability, and detail richness of the predicted depth.

[0008] This invention employs the following technical solution: a sparse point cloud-guided video depth prediction method based on a large visual model, comprising: Visual large model-guided depth completion: Given a pair of visible light video and a sparse point cloud sequence acquired by a radar device, the visible light video is first downsampled to match the effective height and effective width of the radar device's field of view. Then, a visual large model is used to predict a low-resolution relative depth sequence based on the downsampled visible light video. Finally, a local least squares method based on spatiotemporal neighborhood is used to generate a low-resolution video depth based on the input sparse point cloud sequence and the predicted relative depth sequence. High-resolution depth inference based on a large visual model: First, the input visible light video and low-resolution video are divided into several temporally overlapping video segments; then, each visible light video segment is subjected to two-dimensional block embedding encoding, and a visual Transformer encoder loaded with pre-trained weights of a large visual model is used to generate... Layer-encoded features are transformed in dimension and then fed into convolutional layers to construct... The system first extracts visible light video features at different scales; then, it scales the depth of the predicted low-resolution video segments and inputs them into a convolutional layer to construct a structure with a resolution consistent with the visible light video features. Video depth features at different scales; finally, the... Visible light video features and video depth features at different scales are fused and deduplicated layer by layer to output a temporally stable high-resolution video depth.

[0009] Predicting low-resolution relative depth sequences, including: First, the input visible light video... Downsampling is performed to obtain low-resolution visible light video. ,in and These represent the effective height and effective width of the radar equipment's field of view, respectively. The video frame number is represented; then a pre-trained large visual model is used to predict the low-resolution relative depth sequence in the disparity domain. This is then converted to the depth domain to obtain a low-resolution relative depth sequence in the depth domain.

[0010] Furthermore, low-resolution relative depth sequences in the disparity domain Converted to depth domain representation: in, Represents a low-resolution relative depth sequence in the depth domain. , and These represent the x-axis, y-axis, and time axis, respectively. and These represent the minimum and maximum values ​​of the predicted scene depth, respectively. This represents the truncation function.

[0011] Furthermore, using a local least squares method based on spatiotemporal neighborhood, low-resolution video depth is generated based on the input sparse point cloud sequence and the predicted relative depth sequence, including: Model the conversion relationship between relative depth and absolute depth, using... Represents pixels The coordinates, where , and Representing pixels The time coordinates, x-axis, and y-axis, and their specific modeling relationships are represented as follows: in, and Representing pixels The relative depth and absolute depth, The affine parameters are specifically represented as follows: in, and These are the scaling and translation parameters, respectively. This represents the transpose operation; then, for each pixel in all frames... Find the nearest sparse depth points in their spatiotemporal neighborhoods, calculate the four-dimensional distance, and represent it as: in, and These represent the coordinates of the center pixel and the neighboring pixels, respectively. , , and These represent the differences in the x-coordinate, y-coordinate, time coordinate, and estimated relative depth between the center pixel and its neighboring pixels, respectively. and These are used to adjust the scale of the temporal coordinate difference and the relative depth difference, respectively. Then, the nearest spatiotemporal sparse depth point in the four-dimensional distance is selected, and the absolute depth vector is constructed using the sparse depth values ​​of the neighboring points and the relative depth values ​​predicted by the large visual model. and homogeneous relative depth matrix ,in This represents the number of neighboring points after filtering; then, the optimization objective is constructed, expressed as: in, The weight matrix is ​​represented as follows: in, This indicates the construction of a diagonal matrix. This represents the four-dimensional distance in formula (4). and Representing the center pixel and the first pixel respectively Each neighboring pixel Let the standard deviation of the Gaussian kernel function be represented; then, the least squares method is used to solve the optimization objective in formula (5) to obtain the affine parameters. The least squares solution is specifically expressed as: Finally, formula (2) is used to estimate the absolute depth of each pixel to generate the completed low-resolution video depth. .

[0012] Furthermore, construct Visible light video features at different scales, including: First, each frame of the visible light video segment is divided into several non-overlapping blocks. After expanding the pixel values ​​within each block, block embedding encoding is performed using a fully connected layer to obtain the encoded features. ,in, Indicates the number of frames in a video segment. This indicates the number of image blocks in each frame. This represents the feature dimension of the embedding layer; then, a cascaded Transformer encoder is used to further extract features from the embedding layer features, specifically as follows: in, Indicates the first Output features of the layer Transformer encoder Indicates the first Features aggregated by self-attention in a layer encoder , and These represent the multi-head self-attention module, the fully connected layer, and the layer normalization structure, respectively; finally, select... The output features of the layer Transformer encoder are used as encoded features. , Indicates the generation of the first Transformer encoder number for layer-encoded features; For each selected layer of encoded features Perform dimensional transformation to obtain features ,in, and These represent the number of image blocks divided vertically and horizontally, respectively. Indicate the feature layer number; then use convolutional layers to generate Visible light video features at different resolutions .

[0013] Furthermore, construct a system with a resolution consistent with visible light video features. Video depth features at different scales, including: By scaling the depth of the low-resolution video at different scales, we obtain... Depth of layers at different resolutions; Using convolutional networks, from Extracting from depths at different resolutions of layers Video depth features at different scales .

[0014] Furthermore, finally... Visible light video features and video depth features at different scales are fused and dedopted layer by layer to output temporally stable high-resolution video depth, including: Starting from the top layer with the lowest resolution, the extracted video depth features and visible light video features are fused layer by layer, as follows: in, Indicates the use of convolutional layers from the 1st... Video depth features extracted from layered video depth. Indicates the use of convolutional layers from the 1st... Visible light video features extracted from layer coding features. Indicates the upsampling operator, Represents a residual convolutional network. Indicates the first Layer fusion features Indicates the first The temporally stable fusion features of the layers are obtained through inference using the temporal alignment module. For the time-domain unstable first Layer fusion features First, perform dimensional transformation to obtain the features. Then, self-attention is calculated in the time domain to obtain the attention matrix. Specifically, it is expressed as: in, and These represent the weight matrices used to generate the query vector and key vector, respectively. This indicates the transpose operation; then in the feature An attention matrix is ​​applied and a dimensionality transformation is performed to obtain temporally stable output features. , is represented as: in, The weight matrix represents the generated value vector. This represents a dimensionality transformation; then, the temporally stable lowest-level fusion features are used. Input convolutional layers and upsampling layers to obtain the depth of high-resolution video segments. , is represented as: Finally, the predicted high-resolution video segment depths are stitched together to generate a temporally stable high-resolution video depth. The depth estimates of the overlapping portions of adjacent video segments are linearly weighted in the temporal domain to ensure the temporal stability of the video segment switching process, specifically expressed as follows: in, Indicates the first Frame depth estimation, and They represent the first Depth prediction of frames in the preceding and following video segments. This indicates the number of overlapping frames between two consecutive video segments. Indicates the first The number of frames from the starting frame of the overlapping frame.

[0015] Furthermore, .

[0016] Furthermore, the training process for a high-resolution deep inference model includes the following steps: (1) Select a video depth dataset that includes both high-resolution visible light video and high-resolution video depth ground truth, denoted as follows: and ,in, and These represent the height and width of the high-resolution image, respectively. Indicates the number of video frames; (2) After downsampling the depth of the high-resolution video, randomly select several scattered points in each frame to generate a sparse scattered point sequence. , used to simulate sparse point cloud sequences acquired by radar equipment, wherein, and These represent the height and width of the low-resolution depth image, respectively. (3) Use a depth completion method guided by a large visual model to generate low-resolution video depth. ; (4) Visible light video Low-resolution video depth and high-resolution video depth truth Divide the video into several overlapping video segments in the temporal domain, each pair of video segments , and This constitutes a set of training samples; (5) Initialize the high-resolution deep inference model, in which the visual Transformer module loads the pre-trained visual large model weights; (6) Input a set of visible light video segments and low-resolution video segment depth Using a high-resolution deep inference model, high-resolution video segment depth is generated. ; (7) Calculate and The L1 loss between the two is used to update the learnable parameters of the neural network using the Adam optimizer, and the learning rate is dynamically adjusted during the optimization process using the cosine annealing algorithm. (8) Repeat steps (6)-(7) until the number of repetitions reaches the preset number of training iterations, and the training process is completed to obtain the final model.

[0017] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method for sparse point cloud-guided video depth prediction based on a large visual model.

[0018] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention improves the accuracy of depth prediction by utilizing scale information extracted from sparse point clouds, obtains robust depth estimation of the scene by leveraging the generalization ability of a large visual model, and enhances the temporal stability of the predicted depth by employing a local least squares method based on spatiotemporal neighborhood and a temporal alignment module. For input visible light video and sparse point cloud sequences acquired by radar equipment, a robust low-resolution relative depth of the scene is first estimated using a large visual model. A linear model for converting relative depth to absolute depth is solved using a least squares method based on spatiotemporal neighborhood. The linear model can preserve the generalization of relative depth, and the constraints of spatiotemporal neighborhood can improve the temporal consistency of the predicted depth. A high-resolution depth prediction module based on a large visual model is used to generate temporally stable high-resolution video depth. Within the module, pre-trained weights of the large visual model are used to obtain generalization ability, and a temporal alignment module is used to improve the temporal consistency of the predicted depth. The parameters in the high-resolution depth prediction module are tuned using the Adam optimizer, thereby obtaining superior depth prediction performance. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 The flowchart illustrates a sparse point cloud-guided video depth prediction method based on a large visual model, as provided in this embodiment of the invention.

[0021] Figure 2 This is a schematic diagram of a low-resolution depth completion module guided by a large visual model, provided in an embodiment of the present invention.

[0022] Figure 3 This is a schematic diagram of a high-resolution depth prediction module based on a large visual model, provided in an embodiment of the present invention.

[0023] Figure 4 This is a schematic diagram of the temporal-domain aligned feature fusion module in the high-resolution depth prediction module provided in an embodiment of the present invention.

[0024] Figure 5 This is a flowchart of parameter tuning using the Adam optimizer provided in an embodiment of the present invention.

[0025] Figure 6 The following is an example diagram of the depth prediction of the present invention, wherein (a) is the input high-resolution visible light video, (b) is the input low-resolution sparse point cloud sequence, (c) is the generated low-resolution video depth, and (d) is the output high-resolution video depth. Detailed Implementation

[0026] The present invention will now be described in detail with reference to the accompanying drawings. Unless otherwise specified, the features of the following embodiments and implementations can be combined with each other.

[0027] like Figure 1 As shown, this invention provides a sparse point cloud-guided video depth prediction method based on a large visual model. The specific steps of the method are as follows: 1) Visual large model-guided depth completion: Given a pair of visible light video and a sparse point cloud sequence acquired by a radar device, the visible light video is first downsampled to match the effective height and effective width of the radar device's field of view. Then, a visual large model is used to predict a low-resolution relative depth sequence based on the downsampled visible light video. Finally, a local least squares method based on spatiotemporal neighborhood is used to generate a low-resolution video depth based on the input sparse point cloud sequence and the predicted relative depth sequence. 2) High-resolution depth inference based on a large visual model: First, the input visible light video and low-resolution video depth are divided into several temporally overlapping video segments. Then, each visible light video segment is encoded using two-dimensional block embedding. A visual Transformer encoder with pre-trained weights from a large visual model is used to generate four layers of encoded features. The encoded features of each layer are dimensionally transformed and input into convolutional layers to construct four layers of visible light video features at different scales. Next, the predicted low-resolution video segment depth is scaled and input into convolutional layers to construct four layers of video depth features at different scales with the same resolution as the visible light video features. Finally, the four layers of visible light video features and video depth features at different scales are fused layer by layer and de-jittered to output a temporally stable high-resolution video depth.

[0028] Visual large-scale model-guided depth completion, such as Figure 2 As shown, the specific reasoning process includes the following steps: (1) Relative depth sequence prediction: First, the input visible light video is... Downsampling is performed to obtain low-resolution visible light video. ,in and These represent the effective height and effective width of the radar equipment's field of view, respectively. The video frame number is represented; then a pre-trained large visual model is used to predict the low-resolution relative depth sequence in the disparity domain. This is then converted to the depth domain and represented as: in, Represents a low-resolution relative depth sequence in the depth domain. , and These represent the x-axis, y-axis, and time axis, respectively. and These represent the minimum and maximum values ​​of the predicted scene depth, respectively. This represents the truncation function; (2) Low-resolution video depth completion: Low-resolution video depth completion is performed using a local least squares method based on spatiotemporal neighborhood. First, the conversion relationship between relative depth and absolute depth is modeled, and then... Represents pixels The coordinates, where , and Representing pixels The time coordinates, x-axis, and y-axis, and the specific modeling relationship are represented as follows: in, and Representing pixels The relative depth and absolute depth, The affine parameters are specifically represented as follows: in, and These are the scaling and translation parameters, respectively. This represents the transpose operation; then, for each pixel in all frames... Find the nearest sparse depth points in their spatiotemporal neighborhoods, calculate the four-dimensional distance, and represent it as: in, and These represent the coordinates of the center pixel and the neighboring pixels, respectively. , , and These represent the differences in the x-coordinate, y-coordinate, time coordinate, and estimated relative depth between the center pixel and its neighboring pixels, respectively. and These are used to adjust the scale of the temporal coordinate difference and the relative depth difference, respectively. Then, the nearest spatiotemporal sparse depth point in the four-dimensional distance is selected, and the absolute depth vector is constructed using the sparse depth values ​​of the neighboring points and the relative depth values ​​predicted by the large visual model. and homogeneous relative depth matrix ,in This represents the number of neighboring points after filtering; then, the optimization objective is constructed, expressed as: in, The weight matrix is ​​represented as follows: in, This indicates the construction of a diagonal matrix. This represents the four-dimensional distance in formula (4). and Representing the center pixel and the first pixel respectively Each neighboring pixel Let the standard deviation of the Gaussian kernel function be represented; then, the least squares method is used to solve the optimization objective in formula (5) to obtain the affine parameters. The least squares solution is specifically expressed as: Finally, formula (2) is used to estimate the absolute depth of each pixel to generate the completed low-resolution video depth. .

[0029] High-resolution deep reasoning based on large visual models, such as Figure 3 As shown, the specific reasoning process includes the following steps: (1) Video segment division: The depth of the input visible light video and low-resolution video is divided into several video segments in the time domain, and the depth of the high-resolution video is estimated respectively. Different video segments have a certain overlap in the time domain. (2) Visible light video feature extraction: A visual Transformer encoder with pre-trained weights of a large visual model is used to extract features from the input visible light video segment. First, each frame of the visible light video segment is divided into several non-overlapping blocks. After expanding the pixel values ​​in each block, fully connected layers are used to perform block embedding encoding to obtain the encoded features. ,in, Indicates the number of frames in a video segment. This indicates the number of image blocks in each frame. This represents the feature dimension of the embedding layer; then, a cascaded Transformer encoder is used to further extract features from the embedding layer features, specifically as follows: in, Indicates the first Output features of the layer Transformer encoder Indicates the first Features aggregated by self-attention in a layer encoder , and These represent the multi-head self-attention module, the fully connected layer, and the layer normalization structure, respectively. Finally, the output features of the four Transformer encoder layers are selected as the encoded features to generate an encoded feature set. Specifically, it is expressed as: in, Indicates the generation of the first Transformer encoder number for layer-encoded features; (3) Low-resolution video depth feature extraction: The depth of the low-resolution video generated by the depth completion module is scaled at different scales to obtain 4 layers of depth at different resolutions, and a depth set is constructed. Specifically, it is expressed as: in, Indicates the first Layer scaling depth, and They represent the first The layer scales the height and width of the depth; using convolutional networks, features at different scales are extracted from depths of different resolutions to generate feature sets. Specifically, it is expressed as: in, Indicates the dimension of the encoded features. Indicates the use of convolutional layers from the 1st... Features extracted from the depth of layered video; (4) Temporally stable high-resolution video depth prediction: First, the selected layer of encoded features are... Perform dimensional transformation to obtain features ,in, and These represent the number of image blocks divided vertically and horizontally, respectively. Indicates the feature layer number. Indicates the first The Transformer encoder numbers corresponding to the layer features are then used; convolutional layers are then used to generate four layers of visible light video features at different resolutions. Specifically, it is expressed as: in, and They represent the first The height and width of the layer features are consistent with formula (12). This represents a convolutional layer; then, starting from the top layer with the lowest resolution, the extracted video depth features and visible light video features are fused layer by layer, as shown below: in, Indicates the upsampling operator, This represents a residual convolutional network, consisting of two convolutional layers and one activation function. Indicates the first Layer fusion features Indicates the first The temporally stable fusion features of the first layer are obtained using the temporal alignment module; for the temporally unstable first layer... Layer fusion features This module first performs dimensional transformation to obtain features. Then, self-attention is calculated in the time domain to obtain the attention matrix. Specifically, it is expressed as: in, and These represent the weight matrices used to generate the query vector and key vector, respectively. This indicates the transpose operation; then in the feature An attention matrix is ​​applied and a dimensionality transformation is performed to obtain temporally stable output features. , is represented as: in, The weight matrix represents the generated value vector. The flowchart for multi-scale feature fusion with dimensional transformation and temporal alignment is shown below. Figure 4 As shown; then, the time-domain stable lowest-level fusion features are... Input convolutional layers and upsampling layers to obtain the depth of high-resolution video segments. , is represented as: Finally, the predicted high-resolution video segment depths are stitched together to generate a temporally stable high-resolution video depth. The depth estimates of the overlapping portions of adjacent video segments are linearly weighted in the temporal domain to ensure the temporal stability of the video segment switching process, specifically expressed as follows: in, Indicates the first Frame depth estimation, and They represent the first Depth prediction of frames in the preceding and following video segments. This indicates the number of overlapping frames between two consecutive video segments. Indicates the first The number of frames from the starting frame of the overlapping frame.

[0030] The Adam optimizer is used to update all parameters to be optimized in the high-resolution deep inference model, thereby obtaining better parameters. The parameter tuning algorithm used in this invention is as follows: Figure 5 As shown, the specific steps include: (1) Select a video depth dataset that includes both high-resolution visible light video and high-resolution video depth ground truth, denoted as follows: and ,in, and These represent the height and width of the high-resolution image, respectively. Indicates the number of video frames; (2) After downsampling the depth of the high-resolution video, randomly select several scattered points in each frame to generate a sparse scattered point sequence. , used to simulate sparse point cloud sequences acquired by radar equipment, wherein, and These represent the height and width of the low-resolution depth image, respectively. (3) Use a depth completion method guided by a large visual model to generate low-resolution video depth. ; (4) Visible light video Low-resolution video depth and high-resolution video depth truth Divide the video into several overlapping video segments in the temporal domain, each pair of video segments , and This constitutes a set of training samples; (5) Initialize the high-resolution deep inference model, in which the visual Transformer module loads the pre-trained visual large model weights; (6) Input a set of visible light video segments and low-resolution video segment depth Using a high-resolution deep inference model, high-resolution video segment depth is generated. ; (7) Calculate and The L1 loss between the two is used to update the learnable parameters of the neural network using the Adam optimizer, and the learning rate is dynamically adjusted during the optimization process using the cosine annealing algorithm. (8) Repeat steps (6)-(7) until the number of repetitions reaches the preset number of training iterations, and the training process is completed to obtain the final model.

[0031] like Figure 6 The figure shows a visualization of the predicted video depth of the present invention, where (a) is the input high-resolution visible light video, (b) is the input low-resolution sparse point cloud sequence, (c) is the generated low-resolution video depth, and (d) is the output high-resolution video depth. As can be seen from the figure, the method of the present invention can simultaneously extract reliable scale information and robust scene features from the sparse point cloud sequence and visible light video collected by radar equipment, respectively, and generate a realistic, reliable and detailed video depth.

[0032] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method for sparse point cloud-guided video depth prediction based on a large visual model.

[0033] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method for sparse point cloud-guided video depth prediction based on a large visual model.

[0034] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0035] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0036] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0037] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0038] The above embodiments are only used to illustrate the design concept and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, all equivalent changes or modifications made based on the principles and design ideas disclosed in the present invention are within the protection scope of the present invention.

Claims

1. A sparse point cloud-guided video depth prediction method based on a large visual model, characterized in that, include: Visual large model-guided depth completion: Given a pair of visible light video and a sparse point cloud sequence acquired by a radar device, the visible light video is first downsampled to match the effective height and effective width of the radar device's field of view. Then, a visual large model is used to predict a low-resolution relative depth sequence based on the downsampled visible light video. Finally, a local least squares method based on spatiotemporal neighborhood is used to generate a low-resolution video depth based on the input sparse point cloud sequence and the predicted relative depth sequence. High-resolution depth inference based on a large visual model: First, the input visible light video and low-resolution video are divided into several temporally overlapping video segments; then, each visible light video segment is subjected to two-dimensional block embedding encoding, and a visual Transformer encoder loaded with pre-trained weights of a large visual model is used to generate... Layer-encoded features are transformed in dimension and then fed into convolutional layers to construct... The system first extracts visible light video features at different scales; then, it scales the depth of the predicted low-resolution video segments and inputs them into a convolutional layer to construct a structure with a resolution consistent with the visible light video features. Video depth features at different scales; finally, the... Visible light video features and video depth features at different scales are fused and deduplicated layer by layer to output a temporally stable high-resolution video depth.

2. The method according to claim 1, characterized in that, Predicting low-resolution relative depth sequences, including: First, the input visible light video... Downsampling is performed to obtain low-resolution visible light video. ,in and These represent the effective height and effective width of the radar equipment's field of view, respectively. The video frame number is represented; then a pre-trained large visual model is used to predict the low-resolution relative depth sequence in the disparity domain. This is then converted to the depth domain to obtain a low-resolution relative depth sequence in the depth domain.

3. The method according to claim 2, characterized in that, Low-resolution relative depth sequences in the parallax domain Converted to depth domain representation: in, Represents a low-resolution relative depth sequence in the depth domain. , and These represent the x-axis, y-axis, and time axis, respectively. and These represent the minimum and maximum values ​​of the predicted scene depth, respectively. This represents the truncation function.

4. The method according to claim 1, characterized in that, Using a local least squares method based on spatiotemporal neighborhood, low-resolution video depth is generated from the input sparse point cloud sequence and the predicted relative depth sequence, including: Model the conversion relationship between relative depth and absolute depth, using... Represents pixels The coordinates, where , and Representing pixels The time coordinates, x-axis, and y-axis, and their specific modeling relationships are represented as follows: in, and Representing pixels The relative depth and absolute depth, The affine parameters are specifically represented as follows: in, and These are the scaling and translation parameters, respectively. This represents the transpose operation; then, for each pixel in all frames... Find the nearest sparse depth points in their spatiotemporal neighborhoods, calculate the four-dimensional distance, and represent it as: in, and These represent the coordinates of the center pixel and the neighboring pixels, respectively. , , and These represent the differences in the x-coordinate, y-coordinate, time coordinate, and estimated relative depth between the center pixel and its neighboring pixels, respectively. and These are used to adjust the scale of the temporal coordinate difference and the relative depth difference, respectively. Then, the nearest spatiotemporal sparse depth point in the four-dimensional distance is selected, and the absolute depth vector is constructed using the sparse depth values ​​of the neighboring points and the relative depth values ​​predicted by the large visual model. and homogeneous relative depth matrix ,in This represents the number of neighboring points after filtering; then, the optimization objective is constructed, expressed as: in, The weight matrix is ​​represented as follows: in, This indicates the construction of a diagonal matrix. This represents the four-dimensional distance in formula (4). and Representing the center pixel and the first pixel respectively Each neighboring pixel Let the standard deviation of the Gaussian kernel function be represented; then, the least squares method is used to solve the optimization objective in formula (5) to obtain the affine parameters. The least squares solution is specifically expressed as: Finally, formula (2) is used to estimate the absolute depth of each pixel to generate the completed low-resolution video depth. .

5. The method according to claim 1, characterized in that, Build Visible light video features at different scales, including: First, each frame of the visible light video segment is divided into several non-overlapping blocks. After expanding the pixel values ​​within each block, block embedding encoding is performed using a fully connected layer to obtain the encoded features. ,in, Indicates the number of frames in a video segment. This indicates the number of image blocks in each frame. This represents the feature dimension of the embedding layer; then, a cascaded Transformer encoder is used to further extract features from the embedding layer features, specifically as follows: in, Indicates the first Output features of the layer Transformer encoder Indicates the first Features aggregated by self-attention in a layer encoder , and These represent the multi-head self-attention module, the fully connected layer, and the layer normalization structure, respectively; finally, select... The output features of the layer Transformer encoder are used as encoded features. , Indicates the generation of the first Transformer encoder number for layer-encoded features; For each selected layer of encoded features Perform dimensional transformation to obtain features ,in, and These represent the number of image blocks divided vertically and horizontally, respectively. Indicate the feature layer number; then use convolutional layers to generate Visible light video features at different resolutions .

6. The method according to claim 1, characterized in that, Constructing a resolution consistent with visible light video features Video depth features at different scales, including: By scaling the depth of the low-resolution video at different scales, we obtain... Depth of layers at different resolutions; Using convolutional networks, from Extracting from depths at different resolutions of layers Video depth features at different scales .

7. The method according to claim 1, characterized in that, Finally, Visible light video features and video depth features at different scales are fused and dedopted layer by layer to output temporally stable high-resolution video depth, including: Starting from the top layer with the lowest resolution, the extracted video depth features and visible light video features are fused layer by layer, as follows: in, Indicates the use of convolutional layers from the 1st... Video depth features extracted from layered video depth. Indicates the use of convolutional layers from the 1st... Visible light video features extracted from layer coding features. Indicates the upsampling operator, Represents a residual convolutional network. Indicates the first Layer fusion features Indicates the first The temporally stable fusion features of the layers are obtained through inference using the temporal alignment module. For the time-domain unstable first Layer fusion features First, perform dimensional transformation to obtain the features. Then, self-attention is calculated in the time domain to obtain the attention matrix. Specifically, it is expressed as: in, and These represent the weight matrices used to generate the query vector and key vector, respectively. This indicates the transpose operation; then in the feature An attention matrix is ​​applied and a dimensionality transformation is performed to obtain temporally stable output features. , is represented as: in, The weight matrix represents the generated value vector. This represents a dimensionality transformation; then, the temporally stable lowest-level fusion features are used. Input convolutional layers and upsampling layers to obtain the depth of high-resolution video segments. , is represented as: Finally, the predicted high-resolution video segment depths are stitched together to generate a temporally stable high-resolution video depth. The depth estimates of the overlapping portions of adjacent video segments are linearly weighted in the temporal domain to ensure the temporal stability of the video segment switching process, specifically expressed as follows: in, Indicates the first Frame depth estimation, and They represent the first Depth prediction of frames in the preceding and following video segments. This indicates the number of overlapping frames between two consecutive video segments. Indicates the first The number of frames from the starting frame of the overlapping frame.

8. The method according to claim 1, 5, 6 or 7, characterized in that, 。 9. The method according to claim 1, characterized in that, The training process for a high-resolution deep inference model includes the following steps: (1) Select a video depth dataset that includes both high-resolution visible light video and high-resolution video depth ground truth, denoted as follows: and ,in, and These represent the height and width of the high-resolution image, respectively. Indicates the number of video frames; (2) After downsampling the depth of the high-resolution video, randomly select several scattered points in each frame to generate a sparse scattered point sequence. , used to simulate sparse point cloud sequences acquired by radar equipment, wherein, and These represent the height and width of the low-resolution depth image, respectively. (3) Use a depth completion method guided by a large visual model to generate low-resolution video depth. ; (4) Visible light video Low-resolution video depth and high-resolution video depth truth Divide the video into several overlapping video segments in the temporal domain, each pair of video segments , and This constitutes a set of training samples; (5) Initialize the high-resolution deep inference model, in which the visual Transformer module loads the pre-trained visual large model weights; (6) Input a set of visible light video segments and low-resolution video segment depth Using a high-resolution deep inference model, high-resolution video segment depth is generated. ; (7) Calculate and The L1 loss between the two is used to update the learnable parameters of the neural network using the Adam optimizer, and the learning rate is dynamically adjusted during the optimization process using the cosine annealing algorithm. (8) Repeat steps (6)-(7) until the number of repetitions reaches the preset number of training iterations, and the training process is completed to obtain the final model.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements a sparse point cloud-guided video depth prediction method based on a large visual model, as described in any one of claims 1-9.

Images