Method, device and vehicle for external parameter calibration

Abstract

An embodiment of the present application provides a method for extrinsic parameter calibration, including: obtaining a visual image provided by a vehicle camera; obtaining a depth image generated by three-dimensional point cloud data provided by a vehicle laser radar; obtaining uncertainty Q and optical flow F based on the visual image, depth image, and a preset optical flow network, wherein the optical flow F represents the coordinate difference matrix between the associated pixel pairs of the visual image and the depth image estimated by the optical flow network, and the uncertainty Q is used to indicate the uncertainty of the optical flow F; obtaining the extrinsic parameter value between the laser radar and the camera based on the uncertainty Q and the optical flow F. This solution introduces uncertainty Q during the training and use of the extrinsic parameter calibration model, which is used to make uncertainty predictions on the optical flow output by the optical flow network to evaluate the uncertainty of the pixel correlation between the depth image generated by the laser radar point cloud and the visual image of the camera, thereby improving the accuracy of the extrinsic parameter calibration.

Description

Technical Field

[0001] The present application relates to the field of information technology, and in particular to a method, device, and vehicle for extrinsic parameter calibration. Background Art

[0002] With the development and advancement of autonomous driving technology, the number of sensors installed in vehicles is increasing. The realization of multi-sensor fusion is also a hot topic in autonomous driving research. For example, the fusion of LiDAR and camera data is a key issue. Since LiDAR data is a 3D point cloud image, while camera data is a 2D image, it is necessary to project the 3D points onto the image plane to achieve LiDAR and camera data fusion.

[0003] A prerequisite for achieving multi-sensor fusion is accurate extrinsic calibration between sensors. Extrinsic parameters are used to determine the transformation relationship between a sensor and an external coordinate system. These typically include pose parameters, namely rotation and translation parameters. For multi-sensor fusion, extrinsic calibration refers to the process of calibrating the coordinate system transformation between two sensors. As a vehicle moves, sensors also move. Due to collisions and vibrations, extrinsic parameters may drift, affecting the accuracy of subsequent fusion components and, in turn, the safety of the vehicle.

[0004] Therefore, how to improve the accuracy of external parameter calibration methods for lidar and cameras in the field of autonomous driving is one of the research directions that the industry is focusing on. Summary of the Invention

[0005] The present application provides a method for extrinsic parameter calibration to improve the accuracy of extrinsic parameter calibration of lidar and cameras in the field of autonomous driving.

[0006] In a first aspect, a method for extrinsic parameter calibration is provided, comprising: acquiring a visual image provided by a vehicle camera; acquiring a depth image generated by three-dimensional point cloud data provided by a vehicle lidar; acquiring uncertainty Q and optical flow F based on the visual image, the depth image, and a preset optical flow network, wherein the optical flow F represents a coordinate difference matrix between associated pixel pairs of the visual image and the depth image estimated by the optical flow network, and the uncertainty Q is used to indicate the uncertainty of the optical flow F; acquiring extrinsic parameter values ​​between the lidar and the camera based on the uncertainty Q and the optical flow F.

[0007] In combination with the first aspect, in a possible implementation method, obtaining the uncertainty Q and the optical flow F based on the visual image, the depth image and the preset optical flow network includes: performing feature extraction on the visual image and the depth image to obtain a visual image feature map and a depth image feature map; obtaining a correlation matrix based on the dot product between each pixel point of the visual image feature map and each pixel point of the depth image feature map; performing multiple rounds of iterations based on the correlation matrix and the optical flow network to obtain the uncertainty Q and the optical flow F.

[0008] In combination with the first aspect, in a possible implementation, the multiple iterations include N iterations, N is an integer greater than or equal to 1, and the i-th iteration in the N iterations includes: obtaining the i-1th optical flow F output by the optical flow network at the i-1th iteration i-1 , 0<i≤N; according to the i-1th optical flow F i-1 , sample the corresponding correlation region in the correlation matrix; input the correlation region into the optical flow network to output the i-th optical flow F i and the i-th uncertainty Q i .

[0009] In combination with the first aspect, in a possible implementation, obtaining the extrinsic parameter value between the lidar and the camera based on the uncertainty Q and the optical flow F includes: screening the pixel pairs associated between the visual image and the depth image based on the uncertainty Q to determine the target pixel pairs whose uncertainty Q is greater than the uncertainty threshold; and calculating the extrinsic parameter value between the lidar and the camera based on the target pixel pairs.

[0010] In combination with the first aspect, in a possible implementation method, obtaining the extrinsic parameter value between the laser radar and the camera based on the uncertainty Q and the optical flow F includes: constructing a differentiable objective function for solving the extrinsic parameter value based on the uncertainty Q and the optical flow F; solving the objective function using the least squares method to obtain the extrinsic parameter value between the laser radar and the camera.

[0011] In combination with the first aspect, in a possible implementation, a probability model is used in the optical flow network to model the uncertainty Q and the optical flow F, and the probability model is a Gaussian distribution model or a Laplace distribution model.

[0012] In a second aspect, a training method for extrinsic parameter calibration is provided, comprising: obtaining training data, wherein the training data comprises a visual image provided by a vehicle camera and a depth image generated by three-dimensional point cloud data provided by a vehicle lidar; inputting the visual image and the depth image into an extrinsic parameter estimation model to train the parameters of the extrinsic parameter estimation model, wherein the extrinsic parameter estimation model is used to output extrinsic parameter values ​​between the lidar and the camera, and inputting the visual image and the depth image into the extrinsic parameter estimation model to train the parameters of the extrinsic parameter estimation model comprises: obtaining uncertainty Q and optical flow F according to the visual image, the depth image and a preset optical flow network, wherein the optical flow F represents a coordinate difference matrix between associated pixel pairs of the visual image and the depth image estimated by the optical flow network, and the uncertainty Q is used to indicate the uncertainty of the optical flow F; and obtaining extrinsic parameter values ​​between the lidar and the camera according to the uncertainty Q and the optical flow F.

[0013] In combination with the second aspect, in a possible implementation method, obtaining the uncertainty Q and the optical flow F based on the visual image, the depth image and the preset optical flow network includes: performing feature extraction on the visual image and the depth image to obtain a visual image feature map and a depth image feature map; obtaining a correlation matrix based on the dot product between each pixel point of the visual image feature map and each pixel point of the depth image feature map; performing multiple rounds of iterations based on the correlation matrix and the optical flow network to obtain the uncertainty Q and the optical flow F.

[0014] In conjunction with the second aspect, in a possible implementation, the multiple iterations include N iterations, where N is an integer greater than or equal to 1, and the i-th iteration of the N iterations includes: obtaining the i-1th optical flow F output by the optical flow network at the i-1th iteration i-1 , 0<i≤N; according to the i-1th optical flow F i-1 , sample the corresponding correlation region in the correlation matrix; input the correlation region into the optical flow network to output the i-th optical flow F i and the i-th uncertainty Q i .

[0015] In combination with the second aspect, in a possible implementation method, obtaining the extrinsic parameter value between the lidar and the camera based on the uncertainty Q and the optical flow F includes: screening the pixel pairs associated between the visual image and the depth image based on the uncertainty Q to determine the target pixel pairs whose uncertainty Q is greater than the uncertainty threshold; and calculating the extrinsic parameter value between the lidar and the camera based on the target pixel pairs.

[0016] In combination with the second aspect, in a possible implementation method, obtaining the extrinsic parameter value between the laser radar and the camera based on the uncertainty Q and the optical flow F includes: constructing a differentiable objective function for solving the extrinsic parameter value based on the uncertainty Q and the optical flow F; solving the objective function using the least squares method to obtain the extrinsic parameter value between the laser radar and the camera.

[0017] In combination with the second aspect, in a possible implementation, a probability model is used in the optical flow network to model the uncertainty Q and the optical flow F, and the probability model is a Gaussian distribution model or a Laplace distribution model.

[0018] In a third aspect, a device for extrinsic parameter calibration is provided, including: an acquisition module for acquiring a visual image provided by a vehicle camera; the acquisition module is also used to acquire a depth image generated by three-dimensional point cloud data provided by a vehicle lidar; a determination module is used to acquire uncertainty Q and optical flow F based on the visual image, the depth image and a preset optical flow network, wherein the optical flow F represents a coordinate difference matrix between associated pixel pairs of the visual image and the depth image estimated by the optical flow network, and the uncertainty Q is used to indicate the uncertainty of the optical flow F; the determination module is also used to acquire extrinsic parameter values ​​between the lidar and the camera based on the uncertainty Q and the optical flow F.

[0019] In combination with the third aspect, in a possible implementation method, the determination module is specifically used to: perform feature extraction on the visual image and the depth image to obtain a visual image feature map and a depth image feature map; obtain a correlation matrix based on the dot product between each pixel point of the visual image feature map and each pixel point of the depth image feature map; perform multiple rounds of iterations based on the correlation matrix and the optical flow network to obtain the uncertainty Q and optical flow F.

[0020] In conjunction with the third aspect, in a possible implementation, the determining module is specifically configured to: obtain the i-1th optical flow F output by the optical flow network at the i-1th iteration i-1 , 0<i≤N; according to the i-1th optical flow F i-1 , sample the corresponding correlation region in the correlation matrix; input the correlation region into the optical flow network to output the i-th optical flow F i and the i-th uncertainty Q i .

[0021] In combination with the third aspect, in one possible implementation, the determination module is specifically used to: construct a differentiable objective function for solving the extrinsic parameter value based on the uncertainty Q and the optical flow F; and solve the objective function using the least squares method to obtain the extrinsic parameter value between the lidar and the camera.

[0022] In combination with the third aspect, in a possible implementation, a probability model is used in the optical flow network to model the uncertainty Q and the optical flow F, and the probability model is a Gaussian distribution model or a Laplace distribution model.

[0023] In a fourth aspect, a device for extrinsic parameter calibration is provided, including: an acquisition module for acquiring training data, wherein the training data includes a visual image provided by a vehicle camera and a depth image generated by three-dimensional point cloud data provided by a vehicle lidar; a processing module for inputting the visual image and the depth image into an extrinsic parameter estimation model to train the parameters of the extrinsic parameter estimation model, wherein the extrinsic parameter estimation model is used to output the extrinsic parameter value between the lidar and the camera, and the processing module is specifically used to: acquire uncertainty Q and optical flow F according to the visual image, the depth image and a preset optical flow network, wherein the optical flow F represents a coordinate difference matrix between associated pixel pairs of the visual image and the depth image estimated by the optical flow network, and the uncertainty Q is used to indicate the uncertainty of the optical flow F; and acquire the extrinsic parameter value between the lidar and the camera according to the uncertainty Q and the optical flow F.

[0024] In combination with the fourth aspect, in a possible implementation method, the determination module is specifically used to: perform feature extraction on the visual image and the depth image to obtain a visual image feature map and a depth image feature map; obtain a correlation matrix based on the dot product between each pixel point of the visual image feature map and each pixel point of the depth image feature map; and perform multiple rounds of iterations based on the correlation matrix and the optical flow network to obtain the uncertainty Q and optical flow F.

[0025] In conjunction with the fourth aspect, in a possible implementation, the multiple iterations include N iterations, where N is an integer greater than or equal to 1, and the i-th iteration of the N iterations includes: obtaining the i-1th optical flow F output by the optical flow network at the i-1th iteration i-1 , 0<i≤N; according to the i-1th optical flow F i-1 , sampling the corresponding correlation region in the correlation matrix; and inputting the correlation region into the optical flow network to output the i-th optical flow F i and the i-th uncertainty Q i .

[0026] In combination with the fourth aspect, in one possible implementation, the processing module is specifically used to: construct a differentiable objective function for solving the extrinsic parameter value based on the uncertainty Q and the optical flow F; and solve the objective function using the least squares method to obtain the extrinsic parameter value between the lidar and the camera.

[0027] In combination with the fourth aspect, in a possible implementation, a probability model is used in the optical flow network to model the uncertainty Q and the optical flow F, and the probability model is a Gaussian distribution model or a Laplace distribution model.

[0028] In a fifth aspect, a computer device is provided, comprising a processor, which is used to call a computer program from a memory. When the computer program is executed, the processor is used to execute the method in the above-mentioned first aspect or any possible implementation of the first aspect, or to execute the method in the above-mentioned second aspect or any possible implementation of the second aspect.

[0029] In the sixth aspect, a computer-readable storage medium is provided for storing a computer program, which includes a code for executing the method in the above-mentioned first aspect or any possible implementation of the first aspect, or a code for executing the method in the above-mentioned second aspect or any possible implementation of the second aspect.

[0030] In the seventh aspect, a computer program product is provided, comprising a computer program, which includes code for executing the method in the above-mentioned first aspect or any possible implementation of the first aspect, or code for executing the method in the above-mentioned second aspect or any possible implementation of the second aspect.

[0031] In an eighth aspect, a vehicle is provided, comprising a computing device, wherein the computing device is used to execute the method in the above-mentioned first aspect or any possible implementation of the first aspect, or to execute the method in the above-mentioned second aspect or any possible implementation of the second aspect.

[0032] In an embodiment of the present application, a scheme for extrinsic parameter calibration is provided, which introduces probabilistic modeling in the training and use of the extrinsic parameter calibration model. The probabilistic modeling introduces uncertainty Q, which is used to predict the uncertainty of the optical flow output by the optical flow network, so as to evaluate the uncertainty of the correlation between the pixel points of the depth image generated by the lidar point cloud and the visual image of the camera, thereby improving the accuracy of the extrinsic parameter calibration. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0034] Figure 1 is a schematic diagram of an application scenario of an embodiment of the present application;

[0035] Figure 2 2 is a schematic diagram of a framework of an extrinsic parameter estimation model 200 according to an embodiment of the present application;

[0036] Figure 3 is with Figure 2 Schematic diagram of the working principle of the corresponding training method for extrinsic parameter calibration;

[0037] Figure 4 2 is a schematic diagram of a framework of an extrinsic parameter estimation model 200 according to another embodiment of the present application;

[0038] Figure 5 is with Figure 4 Schematic diagram of the working principle of the corresponding method for performing external parameter calibration;

[0039] Figure 6 1 is a flow chart of a method for extrinsic parameter calibration according to an embodiment of the present application;

[0040] Figure 7 1 is a flow chart of a training method for extrinsic parameter calibration according to an embodiment of the present application;

[0041] Figure 8 is a schematic structural diagram of an apparatus 800 according to an embodiment of the present application;

[0042] Figure 9 is a schematic structural diagram of an apparatus 900 according to an embodiment of the present application;

[0043] Figure 10 1 is a schematic structural diagram of an apparatus 1000 according to an embodiment of the present application. DETAILED DESCRIPTION

[0044] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0045] First, let’s explain the terms involved in this application:

[0046] Extrinsic parameters, also known as external parameters or pose parameters, are the parameters required to transform a coordinate point in one coordinate system to another through rotation and translation. Extrinsic parameters include six degrees of freedom (DOF), which refers to the six degrees of freedom an object has in a three-dimensional coordinate system: translation (the x, y, and z axes) and rotation (the rotation) about three axes.

[0047] External parameter calibration: refers to the calibration process of coordinate system conversion between two sensors.

[0048] Depth image: refers to an image whose pixel values ​​are the distance from the image sensor to each point in the scene. The grayscale value of each pixel in the depth image can represent the depth of a certain point in the scene from the image sensor.

[0049] Visual image: refers to the image acquired by the camera, that is, RGB image.

[0050] 3D point cloud: This refers to a collection of point data obtained by LiDAR. Point cloud information is represented by 3D coordinates. Depth images can be converted into point cloud data through coordinate conversion, and point cloud data can also be converted into depth images.

[0051] The embodiments of the present application provide a method, device, and vehicle for extrinsic parameter calibration, aiming to solve the above technical problems in the prior art.

[0052] The following specific embodiments describe in detail the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0053] In order to solve the problems described above, an embodiment of the present application provides a solution for extrinsic parameter calibration, which introduces probabilistic modeling in the training and use of the extrinsic parameter calibration model. The probabilistic modeling introduces uncertainty Q, which is used to predict the uncertainty of the optical flow output by the optical flow network, so as to evaluate the uncertainty of the correlation between the pixel points of the depth image generated by the lidar point cloud and the visual image of the camera, thereby improving the accuracy of the extrinsic parameter calibration.

[0054] Figure 1 This is a schematic diagram of an application scenario of an embodiment of the present application. Figure 1As shown, vehicle 100 is equipped with a lidar 110, a camera 120, and a computing device 150. The data from the lidar 110 and camera 120 can be combined to provide the vehicle with reliable identification data of static and dynamic environments and objects in the driving environment, which can be used for obstacle detection, mapping and positioning, and navigation. Specifically, the lidar 110 can be used to obtain three-dimensional point clouds, and the camera 120 can be used to obtain image or video data.

[0055] Optionally, the computing device 150 may include a vehicle computer, or may be another processor, processing chip, or device installed inside the vehicle. Alternatively, the functions implemented by the computing device 150 may be performed by multiple distributed processors in the vehicle. Alternatively, the computing device 150 does not have to be placed inside the vehicle, but may be other devices outside the vehicle. For example, the computing device 150 may also be replaced by a network device. The network devices include but are not limited to cloud servers. For example, the vehicle 100 may upload images and three-dimensional point clouds to a cloud server, which analyzes and processes the data and returns the processing results to the vehicle 100.

[0056] For the sake of brevity, Figure 1 Only modules related to the embodiment of the present application are shown. Optionally, the vehicle 100 may further include more functional modules or units. For example, the vehicle 100 may further be equipped with a millimeter wave radar or other devices.

[0057] It should be understood that Figure 1 The description of the application scenario is only for example and not for limitation. In practice, appropriate deformation and addition and subtraction can be made on the basis of the above scenario, and it is still applicable to the solution of the embodiment of the present application.

[0058] Next reference Figure 2 and Figure 3 , Figure 2 Schematic diagram of the framework of the external parameter estimation model 200 according to an embodiment of the present application. Figure 2 The extrinsic parameter estimation model 200 in is used to perform a training method for extrinsic parameter calibration. Figure 2 The external parameter estimation model in can be set in Figure 1 The computing device 150 may be configured in the computing device 150, or may be configured in any processing device having neural network processing capabilities. Figure 3 is with Figure 2 Schematic diagram of the working principle of the corresponding training method for extrinsic parameter calibration.

[0059] In the training method, the embodiment of the present application constructs a correlation matrix by calculating the similarity of the features of all pixels between the depth image generated by the LiDAR 3D point cloud and the visual image. An optical flow network is then used to iteratively estimate the optical flow F between the depth image and the visual image, as well as the uncertainty Q of the optical flow F. Finally, the optical flow F and the corresponding uncertainty Q output by the optical flow network are used to optimize the extrinsic parameters between the LiDAR and camera coordinate systems.

[0060] like Figure 2 and Figure 3 As shown, the extrinsic parameter estimation model 200 includes: a feature extraction module 210, a correlation calculation module 220, a search module 230, an optical flow network 240, and a differentiable extrinsic parameter calculation module 250. If the extrinsic parameter estimation model 200 is used for training data, the extrinsic parameter estimation model 200 also includes a flow residual module 260 and an extrinsic parameter residual module 270.

[0061] During the training process, the training data received by the extrinsic parameter estimation model 200 include visual images and depth images. The visual images can be obtained by a camera, and the depth images can be converted from three-dimensional point cloud data obtained by a laser radar.

[0062] The feature extraction module 210 is a feature encoder that can be used to extract features from an input image and output a feature map. For example, the feature extraction module 210 can be used to extract features from a visual image and a depth image, respectively, and output a visual image feature map and a depth image feature map.

[0063] In some examples, two independent feature extraction modules 210 can be used to extract feature maps from the depth image and the visual image. These two feature extraction modules 210 can share the same architecture. However, due to the different input image types, the two feature extraction modules 210 can use different weights. For example, the feature extraction module 210 can adopt the architecture of a residual convolutional network and extract feature maps with a resolution of 1 / 8 its resolution.

[0064] The correlation calculation module 220 is used to receive the visual image feature map and the depth image feature map, and calculate the correlation between each pixel in the visual image feature map and each pixel in the depth image feature map to output a correlation matrix.

[0065] In some examples, the correlation calculation module 220 can calculate the dot product between all pixel features between the depth image and the visual image to establish a first 4-dimensional correlation matrix. The first correlation matrix is ​​then average-pooled three times along the last two dimensions, and the three pooling results and the first correlation matrix are combined to form a second correlation matrix group, which can also be called a correlation pyramid.

[0066] The search module 230 is used to sample a corresponding correlation region in the correlation matrix according to the optical flow F or the initial optical flow value F0 output by the optical flow network.

[0067] Continue to see Figure 3 The optical flow network 240 is a deep neural network model that receives the correlation region output by the search module 230 and estimates the coordinate difference matrix and corresponding uncertainty between the associated pixel pairs in the visual image and the depth image. The coordinate difference matrix is ​​the optical flow, also known as the calibration flow, which refers to the matrix of coordinate differences between the associated pixel pairs in the visual image and the depth image. The matrix includes three dimensions: the horizontal coordinate, the vertical coordinate, and the coordinate difference of the pixel.

[0068] The optical flow network 240 can calculate the optical flow F and uncertainty Q through multiple iterations. Figure 3 As shown, the optical flow F can be obtained through N iterations, where N is an integer greater than or equal to 1. The optical flow network 240 can set an initial optical flow value F0. During each iteration, the optical flow network outputs the optical flow increment ΔF output by this iteration. i , increase the optical flow of this iteration by ΔF i Compared with the optical flow F obtained in the last iteration i-1 Adding them together, we get the optical flow F output by this iteration i The optical flow increment ΔF1 obtained in the first iteration can be added to the initial optical flow value F0 to obtain the optical flow F1 output in the first iteration. The optical flow F output in the last iteration N That is the final optical flow F.

[0069] The optical flow F output at each iteration i The calculation method of can be shown in the following formula (1).

[0070] F i =F i-1 +ΔF i (1)

[0071] Or, in the case of F0=0, the optical flow F output at each iteration is i The calculation method can also be shown as the following formula.

[0072]

[0073] The present embodiment performs probabilistic modeling on the output of the optical flow network 240 to simultaneously estimate the optical flow between images and the corresponding uncertainty Q. The uncertainty Q can be used to indicate the uncertainty of the optical flow F calculated at each iteration. For example, a higher uncertainty in the optical flow indicates a lower accuracy of the optical flow calculated by the network.

[0074] The embodiments of the present application do not limit the type of the above-mentioned probability modeling. For example, the above-mentioned probability modeling may adopt a Gaussian distribution model, a Laplace distribution model, or other types of probability modeling models.

[0075] Taking the probability modeling as a two-dimensional Gaussian distribution as an example, the probability density function can be expressed as the following formula (3).

[0076]

[0077] Where μ=[μ u , μ v ] represents the mean of the two-dimensional Gaussian distribution, δ=[δ u , δ v ] represents the standard deviation of the two-dimensional Gaussian distribution, F=[F u , F v ] represents optical flow.

[0078] The output results predicted by the optical flow network 240 may correspond to the mean and confidence of a two-dimensional Gaussian distribution, respectively. The optical flow F may be the mean of the two-dimensional Gaussian distribution, and the uncertainty Q may be the confidence of the two-dimensional Gaussian distribution. The confidence is proportional to the inverse of the variance δ of the Gaussian distribution.

[0079] Optionally, in practice, in addition to Gaussian distribution, probability modeling can be adjusted to other probability modeling types according to camera characteristics, for example, Laplace distribution can be used for probability modeling.

[0080] The differentiable extrinsic parameter calculation module 250 is used to calculate the extrinsic parameter values ​​between the lidar and the camera based on the optical flow F and uncertainty Q output by the optical flow network, that is, to calculate the six-degree-of-freedom posture parameters. The differentiable extrinsic parameter calculation module 250 can construct a differentiable objective function for solving the extrinsic parameters, or a nonlinear objective function, and perform least squares optimization based on the output results of the optical flow network to obtain the extrinsic parameter values. Optionally, in the process of optimizing the objective function, the uncertainties Q and F output by the optical flow network can be used for optimization to solve the extrinsic parameter values. As an example, the objective function can be expressed as formula (4):

[0081]

[0082] Among them, T represents the external parameter value, T init represents the initial value of the external parameter, ΔT represents the difference between the current estimated value and the initial value of the external parameter, ΔT * Indicates the optimal solution for the difference between the current estimate and the initial value of the external parameter. Represents point P in the 3D point cloud acquired by the lidar i Based on the initial value of external parameter T initThe coordinates projected onto the visual image are the pixel coordinates after the optical flow network output F updates them. π(P, T) represents the projection function, which calculates the pixel coordinates of the point P projected onto the visual image after being transformed by the external parameter value T. The pixel coordinates are The corresponding uncertainty.

[0083] In an embodiment of the present application, the differentiable extrinsic parameter calculation module 250 may adopt a differentiable posture estimation algorithm, so that the external error can be reversely supervised in the optical flow network.

[0084] The flow residual module 260 is used to substitute the optical flow F and uncertainty Q output by the optical flow network into the optical flow loss function to calculate the optical flow loss value, which is used to supervise the optical flow network. For example, it can be used to supervise the network to maximize the likelihood probability of the true value of the optical flow in the two-dimensional Gaussian distribution predicted by the network. The true value of the optical flow refers to the true value of the optical flow, which can be calculated based on the true value of the external parameter and the depth image.

[0085] The embodiment of the present application does not limit the specific implementation of the optical flow loss function. As an example, the optical flow loss function can be expressed as the following formula (5):

[0086]

[0087] Among them, L f represents the optical flow loss value, N represents the number of iterations, Q i Indicates the uncertainty calculated at each iteration, F i represents the optical flow updated according to the ΔFi output of each iteration, γ represents the weight of the output results of different iterations, F gt Represents the true value of optical flow.

[0088] The extrinsic parameter residual module 270 is used to substitute the calculated extrinsic parameter values ​​into the extrinsic parameter loss function to calculate the extrinsic parameter loss value, which is used to supervise the optical flow network. The extrinsic parameter loss function is used to calculate the error between the true value of the extrinsic parameter and the extrinsic parameter value estimated by the network.

[0089] The embodiment of the present application does not limit the specific implementation of the external parameter loss function. As an example, the external parameter loss function can be expressed as the following formula (6):

[0090] L g =||Log(T intt ΔT * ) -1 *T gt || (6)

[0091] Among them, L g Represents the external parameter loss value, T gtRepresents the true value of the external parameter, Log() represents the logarithmic mapping, which is used to map the Lie group to the corresponding Lie algebra.

[0092] Next, continue to refer to Figure 4 and Figure 5 , Figure 4 2 is a schematic diagram of the framework of an external parameter estimation model 200 according to another embodiment of the present application. Figure 4 2 is a schematic diagram of the structure of the extrinsic parameter estimation model 200 for executing the extrinsic parameter calibration method. Figure 4 The external parameter estimation model in can be set in Figure 1 The computing device 150 may be configured in the computing device 150, or may be configured in any processing device having neural network processing capabilities. Figure 5 is with Figure 4 The corresponding working principle diagram of the method for performing external parameter calibration. Figure 4 and Figure 5 The same or similar content as the previous article will not be repeated here.

[0093] like Figure 4 and Figure 5 As shown, during the extrinsic parameter calibration process, a gating module 280 and an extrinsic parameter calculation module 290 may be added to the extrinsic parameter estimation model 200 .

[0094] Gating module 280 is used to receive uncertainty Q and optical flow F, and filter pixel pairs in the visual image and depth image based on uncertainty Q, remove pixel pairs with lower uncertainty, and use the remaining pixel pairs to calculate extrinsic parameter values. Since the points with lower uncertainty are removed, the accuracy of the calculated extrinsic parameter values ​​can be improved.

[0095] In some examples, the gating module 280 may set an uncertainty threshold and, based on the uncertainty threshold, retain pixel pairs in the visual image and depth image whose uncertainty is higher than the uncertainty region. The optical flow of the pixel pairs that meet the requirements is sent to the extrinsic parameter calculation module 250.

[0096] Optionally, the extrinsic parameter calculation module 290 may adopt a differentiable attitude estimation algorithm or another type of attitude estimation algorithm, which is not limited in the embodiment of the present application.

[0097] Optionally, Figure 4 and Figure 5 The gating module 280 may not be included in the extrinsic parameter estimation model 200. The extrinsic parameter calculation module 290 receives the unfiltered uncertainty Q and optical flow F and calculates the extrinsic parameter value.

[0098] Figure 6 This is a flow chart of a method for extrinsic parameter calibration according to an embodiment of the present application. Figure 1The computing device 150 in the embodiment of the present invention may also be executed by other processing devices with neural network computing capabilities. The computing device 150 or the processing device may include the external parameter estimation model mentioned above. Figure 6 The same or similar parts as above will not be described here. Figure 6 As shown, the method includes the following contents.

[0099] S601: Acquire a visual image provided by a vehicle camera.

[0100] S602: Acquire a depth image generated by three-dimensional point cloud data provided by the vehicle laser radar.

[0101] S603. Obtain uncertainty Q and optical flow F based on the visual image, the depth image, and the preset optical flow network, where the optical flow F represents the coordinate difference matrix between the associated pixel pairs of the visual image and the depth image estimated by the optical flow network, and the uncertainty Q is used to indicate the uncertainty of the optical flow F.

[0102] Optionally, uncertainty Q and optical flow F are obtained based on the visual image, depth image and a preset optical flow network, including: performing feature extraction on the visual image and depth image to obtain a visual image feature map and a depth image feature map; obtaining a correlation matrix based on the dot product between each pixel point of the visual image feature map and each pixel point of the depth image feature map; performing multiple rounds of iterations based on the correlation matrix and the optical flow network to obtain uncertainty Q and optical flow F.

[0103] Among them, the specific principles of calculating uncertainty Q and optical flow F can be referred to Figures 1 to 5 For the sake of brevity, the relevant description in is not repeated here. For example, Figure 4 or Figure 5 The feature extraction module 210 performs feature extraction and outputs a feature map; the correlation calculation module 220 calculates the correlation matrix, and multiple rounds of iterations are performed between the search module 230 and the optical flow network 240 to obtain the uncertainty Q and the optical flow F.

[0104] In some examples, the multiple iterations include N iterations, where N is an integer greater than or equal to 1, and the i-th iteration in the N iterations includes: obtaining the i-1th optical flow F output by the optical flow network at the i-1th iteration i-1 , 0<i≤N; according to the i-1th optical flow F i-1 , sample the corresponding correlation area in the correlation matrix; input the correlation area into the optical flow network to output the i-th optical flow F i and the i-th uncertainty Q i .

[0105] S604: Obtain extrinsic parameter values ​​between the laser radar and the camera based on the uncertainty Q and the optical flow F.

[0106] In some examples, based on the uncertainty Q and the optical flow F, the extrinsic parameter values ​​between the lidar and the camera are obtained, including: based on the uncertainty Q, filtering the pixel pairs associated between the visual image and the depth image to determine the target pixel pairs whose uncertainty Q is greater than the uncertainty threshold; based on the target pixel pairs, calculating the extrinsic parameter values ​​between the lidar and the camera.

[0107] Among them, for the specific principles of external parameter values, please refer to Figures 1 to 5 For the sake of brevity, the relevant description in is not repeated here. For example, Figure 4 or Figure 5 The gate control module 280 performs the pixel screening process, and the extrinsic parameter calculation module 290 performs the extrinsic parameter value calculation process.

[0108] In some examples, the extrinsic parameter values ​​between the lidar and the camera are obtained based on the uncertainty Q and the optical flow F, including: constructing a differentiable objective function for solving the extrinsic parameter values ​​based on the uncertainty Q and the optical flow F; solving the objective function using the least squares method to obtain the extrinsic parameter values ​​between the lidar and the camera.

[0109] In some examples, a probability model is used in the optical flow network to model uncertainty Q and optical flow F, and the probability model is a Gaussian distribution model or a Laplace distribution model.

[0110] In an embodiment of the present application, a scheme for extrinsic parameter calibration is provided, which introduces probabilistic modeling in the process of using the extrinsic parameter calibration model. The probabilistic modeling introduces uncertainty Q, which is used to predict the uncertainty of the optical flow output by the optical flow network to evaluate the uncertainty of the correlation between the pixel points of the depth image generated by the lidar point cloud and the visual image of the camera, which can improve the accuracy of the extrinsic parameter calibration.

[0111] Figure 7 This is a flow chart of a training method for external parameter calibration according to an embodiment of the present application. Figure 1 The computing device 150 in the embodiment of the present invention may also be executed by other processing devices with neural network computing capabilities. The computing device 150 or the processing device may include the external parameter estimation model mentioned above. Figure 7 The same or similar parts as above will not be described here. Figure 7 As shown, the method includes the following contents.

[0112] S701. Acquire training data, where the training data includes visual images provided by a vehicle camera and depth images generated by three-dimensional point cloud data provided by a vehicle lidar.

[0113] S702: Input the visual image and the depth image into the extrinsic parameter estimation model to train the parameters of the extrinsic parameter estimation model.

[0114] Among them, the extrinsic parameter estimation model is used to output the extrinsic parameter values ​​between the lidar and the camera, and the visual image and the depth image are input into the extrinsic parameter estimation model to train the parameters of the extrinsic parameter estimation model, including: obtaining uncertainty Q and optical flow F based on the visual image, the depth image and the preset optical flow network, wherein the optical flow F represents the coordinate difference matrix between the associated pixel pairs of the visual image and the depth image estimated by the optical flow network, and the uncertainty Q is used to indicate the uncertainty of the optical flow F; based on the uncertainty Q and the optical flow F, the extrinsic parameter values ​​between the lidar and the camera are obtained.

[0115] Optionally, uncertainty Q and optical flow F are obtained based on the visual image, depth image and a preset optical flow network, including: performing feature extraction on the visual image and depth image to obtain a visual image feature map and a depth image feature map; obtaining a correlation matrix based on the dot product between each pixel point of the visual image feature map and each pixel point of the depth image feature map; performing multiple rounds of iterations based on the correlation matrix and the optical flow network to obtain uncertainty Q and optical flow F.

[0116] Among them, the specific principles of calculating uncertainty Q and optical flow F can be referred to Figures 1 to 5 For the sake of brevity, the relevant description in is not repeated here. For example, Figure 2 or Figure 3 The feature extraction module 210 performs feature extraction and outputs a feature map; the correlation calculation module 220 calculates the correlation matrix, and multiple rounds of iterations are performed between the search module 230 and the optical flow network 240 to obtain the uncertainty Q and the optical flow F.

[0117] In some examples, the multiple iterations include N iterations, where N is an integer greater than or equal to 1, and the i-th iteration in the N iterations includes: obtaining the i-1th optical flow F output by the optical flow network at the i-1th iteration i-1 , 0<i≤N; according to the i-1th optical flow F i-1 , sample the corresponding correlation area in the correlation matrix; input the correlation area into the optical flow network to output the i-th optical flow F i and the i-th uncertainty Q i .

[0118] In some examples, the extrinsic parameter values ​​between the lidar and the camera are obtained based on the uncertainty Q and the optical flow F, including: constructing a differentiable objective function for solving the extrinsic parameter values ​​based on the uncertainty Q and the optical flow F; solving the objective function using the least squares method to obtain the extrinsic parameter values ​​between the lidar and the camera.

[0119] In some examples, a probability model is used in the optical flow network to model uncertainty Q and optical flow F, and the probability model is a Gaussian distribution model or a Laplace distribution model.

[0120] In an embodiment of the present application, a scheme for extrinsic parameter calibration is provided, which introduces probabilistic modeling in the training process of the extrinsic parameter calibration model. The probabilistic modeling introduces uncertainty Q, which is used to predict the uncertainty of the optical flow output by the optical flow network, so as to evaluate the uncertainty of the correlation between the pixel points of the depth image generated by the lidar point cloud and the visual image of the camera, thereby improving the accuracy of the extrinsic parameter calibration.

[0121] Figure 8 800 is a schematic diagram of the structure of an embodiment of the present application. The device 800 can be used to perform Figure 2 、 Figure 3 、 Figure 6 , or execute the method performed by the computing device 150 as described above.

[0122] The device 800 includes: an acquisition module 810, used to acquire a visual image provided by a vehicle camera; the acquisition module is also used to acquire a depth image generated by three-dimensional point cloud data provided by a vehicle laser radar; a determination module 820, used to acquire uncertainty Q and optical flow F based on the visual image, the depth image and a preset optical flow network, wherein the optical flow F represents a coordinate difference matrix between associated pixel pairs of the visual image and the depth image estimated by the optical flow network, and the uncertainty Q is used to indicate the uncertainty of the optical flow F; the determination module is also used to acquire external parameter values ​​between the laser radar and the camera based on the uncertainty Q and the optical flow F.

[0123] Figure 9 FIG. 9 is a schematic diagram of the structure of an apparatus 900 according to an embodiment of the present application. The apparatus 900 may be used to perform Figure 4 、 Figure 5 、 Figure 7 , or execute the method performed by the computing device 150 as described above.

[0124] The device 900 includes: an acquisition module 910 for acquiring training data, wherein the training data includes a visual image provided by a vehicle camera and a depth image generated by three-dimensional point cloud data provided by a vehicle lidar; a processing module 920 for inputting the visual image and the depth image into an extrinsic parameter estimation model to train the parameters of the extrinsic parameter estimation model, wherein the extrinsic parameter estimation model is used to output extrinsic parameter values ​​between the lidar and the camera, and the processing module 920 is specifically used to: acquire uncertainty Q and optical flow F based on the visual image, the depth image and a preset optical flow network, wherein the optical flow F represents a coordinate difference matrix between associated pixel pairs of the visual image and the depth image estimated by the optical flow network, and the uncertainty Q is used to indicate the uncertainty of the optical flow F; and acquire extrinsic parameter values ​​between the lidar and the camera based on the uncertainty Q and the optical flow F.

[0125] Figure 10 1 is a schematic diagram of the structure of the device 1000 according to an embodiment of the present application. The device 1000 is used to perform the above Figures 2 to 7 or for executing a method performed by computing device 150.

[0126] The apparatus 1000 includes a processor 1010, which is configured to execute computer programs or instructions stored in a memory 1020, or read data stored in the memory 1020, to perform the methods in the above method embodiments. Optionally, there are one or more processors 1010.

[0127] Alternatively, as Figure 10 As shown, the device 1000 further includes a memory 1020, which is used to store computer programs or instructions and / or data. The memory 1020 can be integrated with the processor 1010, or can be separately provided. Optionally, there are one or more memories 1020.

[0128] Alternatively, as Figure 10 As shown, the apparatus 1000 further includes a communication interface 1030, which is used to receive and / or send signals. For example, the processor 1010 is used to control the communication interface 1030 to receive and / or send signals.

[0129] For example, the processor 1010 is used to execute the computer program or instructions stored in the memory 1020 to implement the above Figures 2 to 7 or related operations of the computing device 150 for implementing various method embodiments.

[0130] It should be pointed out that Figure 10The device 1000 may be the computing device in the aforementioned embodiment, or a component of the computing device (such as a chip), which is not limited here.

[0131] In an embodiment of the present application, a processor is a circuit with a signal processing capability. In one implementation, the processor may be a circuit with instruction reading and execution capability, such as a CPU, a microprocessor, a GPU (which may be understood as a microprocessor), or a DSP, etc.; in another implementation, the processor may implement certain functions through the logical relationship of a hardware circuit, and the logical relationship of the hardware circuit may be fixed or reconfigurable, such as a hardware circuit implemented by an ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document to implement the hardware circuit configuration may be understood as the process of the processor loading instructions to implement the functions of some or all of the above units. In addition, it may also be a hardware circuit designed for artificial intelligence, which may be understood as an ASIC, such as an NPU, TPU, DPU, etc.

[0132] It can be seen that each unit in the above device can be one or more processors (or processing circuits) configured to implement the above method, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0133] In addition, the various units in the above devices can be fully or partially integrated together, or can be implemented independently. In one implementation, these units are integrated together and implemented in the form of a system-on-a-chip (SOC). The SOC may include at least one processor for implementing any of the above methods or implementing the functions of the various units of the device. The type of the at least one processor can be different, for example, including a CPU and FPGA, a CPU and an artificial intelligence processor, a CPU and a GPU, etc.

[0134] Accordingly, the embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when the computer program / instruction is executed by a processor, causes the processor to execute Figure 2 、 Figure 3 、 Figure 6 , or execute the steps in the method performed by the computing device 150 above.

[0135] Accordingly, the embodiment of the present application further provides a computer program product, including a computer program / instruction, which, when executed by a processor, causes the processor to execute Figure 4 、 Figure 5 、 Figure 7, or execute the steps in the method performed by the computing device 150 above.

[0136] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0137] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0138] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0139] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0140] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0141] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0142] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0143] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a..." does not preclude the presence of additional identical elements in the process, method, commodity, or apparatus comprising the aforementioned elements.

[0144] The above are merely embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A method for external parameter calibration, characterized in that: include: Acquire visual images provided by vehicle cameras; Obtain a depth image generated by the three-dimensional point cloud data provided by the vehicle's lidar; Obtaining an uncertainty Q and an optical flow F according to the visual image, the depth image, and a preset optical flow network, wherein the optical flow F represents a coordinate difference matrix between associated pixel pairs of the visual image and the depth image estimated by the optical flow network, and the uncertainty Q is used to indicate the uncertainty of the optical flow F; Obtaining an extrinsic parameter value between the laser radar and the camera according to the uncertainty Q and the optical flow F; The obtaining of extrinsic parameter values ​​between the laser radar and the camera according to the uncertainty Q and the optical flow F includes: According to the uncertainty Q, pixel pairs associated between the visual image and the depth image are screened to determine target pixel pairs whose uncertainty Q is greater than an uncertainty threshold; Calculating extrinsic parameter values ​​between the laser radar and the camera based on the target pixel pair; or, The obtaining of extrinsic parameter values ​​between the laser radar and the camera according to the uncertainty Q and the optical flow F includes: Constructing a differentiable objective function for solving an extrinsic parameter value according to the uncertainty Q and the optical flow F; Solving the objective function using a least squares method to obtain extrinsic parameter values ​​between the laser radar and the camera; The obtaining of uncertainty Q and optical flow F according to the visual image, the depth image, and a preset optical flow network includes: Performing feature extraction on the visual image and the depth image to obtain a visual image feature map and a depth image feature map; Obtaining a correlation matrix based on a dot product between each pixel of the visual image feature map and each pixel of the depth image feature map; Multiple rounds of iterations are performed based on the correlation matrix and the optical flow network, and each iteration outputs the optical flow F and its corresponding uncertainty Q; each uncertainty Q is used to indicate the uncertainty of the optical flow F calculated in each iteration.

2. The method according to claim 1, wherein The multiple rounds of iterations include N iterations, where N is an integer greater than or equal to 1, and the i-th iteration of the N iterations includes: Get the i-1th optical flow F output by the optical flow network at the i-1th iteration i-1 , 0<i≤N; According to the i-1th optical flow F i-1 , sampling the corresponding correlation region in the correlation matrix; Input the correlation region into the optical flow network to output the i-th optical flow F i and the i-th uncertainty Q i .

3. The method according to claim 1, wherein A probability model is used in the optical flow network to model the uncertainty Q and the optical flow F, and the probability model is a Gaussian distribution model or a Laplace distribution model.

4. A device for external parameter calibration, characterized in that: include: An acquisition module, used for acquiring visual images provided by a vehicle camera; The acquisition module is also used to acquire a depth image generated by three-dimensional point cloud data provided by the vehicle laser radar; a determination module, configured to obtain an uncertainty Q and an optical flow F based on the visual image, the depth image, and a preset optical flow network, wherein the optical flow F represents a coordinate difference matrix between associated pixel pairs of the visual image and the depth image estimated by the optical flow network, and the uncertainty Q is used to indicate the uncertainty of the optical flow F; The determination module is further configured to obtain an extrinsic parameter value between the laser radar and the camera based on the uncertainty Q and the optical flow F; The determination module is specifically configured to: screen pixel pairs associated between the visual image and the depth image according to the uncertainty Q to determine target pixel pairs whose uncertainty Q is greater than an uncertainty threshold; and calculate extrinsic parameter values ​​between the laser radar and the camera based on the target pixel pairs; or, The determination module is specifically configured to: construct a differentiable objective function for solving an extrinsic parameter value based on the uncertainty Q and the optical flow F; solve the objective function using a least squares method to obtain an extrinsic parameter value between the laser radar and the camera; The determination module is specifically used to: perform feature extraction on the visual image and the depth image to obtain a visual image feature map and a depth image feature map; obtain a correlation matrix based on the dot product between each pixel point of the visual image feature map and each pixel point of the depth image feature map; perform multiple rounds of iterations based on the correlation matrix and the optical flow network, and output the optical flow F and its corresponding uncertainty Q in each iteration; each uncertainty Q is used to indicate the uncertainty of the optical flow F calculated in each iteration.

5. The device according to claim 4, characterized in that The determination module is specifically used to obtain the i-1th optical flow F output by the optical flow network at the i-1th iteration i-1 , 0<i≤N; according to the i-1th optical flow F i-1 , sample the corresponding correlation region in the correlation matrix; input the correlation region into the optical flow network to output the i-th optical flow F i and the i-th uncertainty Q i .

6. An electronic device comprising: a processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 3.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 3 when executed by a processor.

8. A vehicle, characterized in that: A computing device is provided in the vehicle, and the computing device is used to implement the method according to any one of claims 1 to 3.

Images