Dense point cloud estimation method, device, computer equipment and storage medium
Through the camera device, the environmental image frame sequence is collected and the objective function is constructed, the problem of high configuration of dense point cloud acquisition hardware equipment in the field of unmanned driving is solved, and the reconstruction and recognition accuracy of dense point clouds is improved, and the cost of 4D labeling is reduced.
Patent Information
- Application Number
- CN202410766752.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-14
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-06-14
AI Technical Summary
In the field of unmanned driving, the hardware equipment required to obtain dense point clouds is high, resulting in an increase in 4D labeling costs.
The camera device collects the environmental image frame sequence, recognizes the position information of the environmental vehicle, and builds the objective function based on this to output a dense point cloud that meets the constraints, reducing the dependence on the hardware device.
The reconstruction of dense point clouds is achieved, reducing the cost of 4D annotation, and improving point cloud density and recognition accuracy.
Smart Images

Figure CN118334125B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a dense point cloud estimation method, device, computer equipment and storage medium. Background Art
[0002] In the field of unmanned driving, real-time perception of the surrounding environment of the vehicle is the basis for the decision-making and control of the unmanned driving system. The relevant environmental perception technology has also received much attention from academia and industry. In order to reduce the annotation cost of deep learning in the field of unmanned driving and improve the annotation efficiency, a dense map corresponding to the driving environment can be constructed. The annotators perform 4D annotation on the dense map and then back-project the annotation results back into the image, which consistently completes the task of annotating the image. However, in the relevant 4D annotation schemes, it is often necessary to collect the dense point cloud corresponding to the environment through sensors that perceive external information such as lidar and millimeter-wave radar to obtain a dense map. Therefore, there is a high demand for hardware equipment configuration, which increases the cost of 4D annotation. Summary of the invention
[0003] In view of this, the present invention provides a dense point cloud estimation method, apparatus, computer equipment and storage medium to solve the problem that obtaining dense point clouds has high requirements for hardware equipment configuration, thereby increasing the cost of 4D annotation.
[0004] In a first aspect, the present invention provides a dense point cloud estimation method, the method comprising:
[0005] Acquire an environmental image frame sequence captured by a camera device of the target vehicle, and identify the environmental vehicle in the environmental image frame sequence to obtain an identification result;
[0006] Based on the recognition result, determining the first position information of the environment vehicle, wherein the first position information is used to indicate the position information corresponding to the environment vehicle in the adjacent image frames of the environment image frame sequence;
[0007] Identify the second pose information corresponding to the target vehicle in the adjacent image frames, and determine the point cloud data of the environment vehicle according to the first pose information and the second pose information;
[0008] An objective function is constructed based on the first pose information, the second pose information and the point cloud data to output a dense point cloud of the environment vehicle that meets the constraints.
[0009] In an optional implementation, determining point cloud data of the environment vehicle according to the first pose information and the second pose information includes:
[0010] Based on the extrinsic parameters corresponding to the camera device, the three-dimensional coordinates of the two-dimensional point cloud of the environment vehicle at the time corresponding to the adjacent image frames are estimated, wherein the extrinsic parameters are used to indicate the parameters of the rotation and translation relationship between the coordinate system of the camera device and the coordinate system of the target vehicle;
[0011] Based on the first pose information, the second pose information and the three-dimensional coordinates, the two-dimensional point cloud is back-projected to obtain three-dimensional point cloud data.
[0012] In the disclosed embodiment, the point cloud data of the above-mentioned environment vehicle can be solved by two adjacent image frames. It should be understood that each two-dimensional point cloud in the environment vehicle can correspond to the solution results of multiple three-dimensional point clouds, thereby improving the point cloud density and recognition accuracy in the final output three-dimensional dense point cloud, and providing a technical basis for the following filtering of the point cloud data.
[0013] In an optional implementation, the adjacent image frames include an image frame at a first moment and an image frame at a second moment;
[0014] Based on the first pose information, the second pose information and the corresponding pixel coordinates, the two-dimensional point cloud is back-projected to obtain three-dimensional point cloud data, including:
[0015] Based on the first sub-pose information corresponding to the first pose information in the image frame at the first moment, the second sub-pose information corresponding to the second pose information in the image frame at the first moment, the three-dimensional first sub-coordinates corresponding to the two-dimensional point cloud at the first moment, and the preset scale coefficient, a first projection function is constructed, wherein the first projection function is used to back-project the two-dimensional point cloud based on the image frame at the first moment;
[0016] Based on the third sub-pose information corresponding to the first pose information in the image frame at the second moment, the fourth sub-pose information corresponding to the second pose information in the image frame at the second moment, the three-dimensional second sub-coordinates corresponding to the two-dimensional point cloud at the second moment, and the preset scale coefficient, a second projection function is constructed, wherein the second projection function is used to back-project the two-dimensional point cloud based on the image frame at the second moment;
[0017] A set of simultaneous equations is constructed according to the first projection function and the second projection function, and three-dimensional point cloud data corresponding to the two-dimensional point cloud is calculated based on the set of simultaneous equations.
[0018] In the disclosed embodiment, the point cloud data of the above-mentioned environment vehicle can be solved by two adjacent image frames. It should be understood that each two-dimensional point cloud in the environment vehicle can correspond to the solution results of multiple three-dimensional point clouds, thereby improving the point cloud density and recognition accuracy in the final output three-dimensional dense point cloud, and providing a technical basis for the following filtering of the point cloud data.
[0019] In an optional embodiment, the method further includes:
[0020] After determining the point cloud data of the environment vehicle according to the first pose information and the second pose information, identifying the target points in the point cloud data that do not meet the observation conditions;
[0021] Filter the point cloud data based on the target point to obtain the filtering result;
[0022] When the noise value of the filtering result does not meet the preset threshold, the filtering result is filtered until the noise value meets the preset threshold, and the filtered point cloud data is output.
[0023] In the disclosed embodiment, the above-mentioned point cloud data can be filtered to make the three-dimensional point cloud in the point cloud data converge, so that the point cloud data converges as a whole, providing a technical basis for further global optimization of the point cloud data through the objective function described below.
[0024] In an optional implementation, constructing an objective function according to the first pose information, the second pose information and the point cloud data includes:
[0025] The acquired environment point clouds in adjacent image frames;
[0026] The environment point cloud, the first pose information, the second pose information and the point cloud data are used as decision variables, and the constraints corresponding to the decision variables are determined;
[0027] Construct the objective function based on decision variables and constraints.
[0028] In the disclosed embodiments, an objective function can be constructed based on multiple decision variables, so as to find the optimal solution for each decision variable in a larger search space, which provides more possibilities and flexibility for optimization. At the same time, multiple decision variables enable the model corresponding to the objective function to consider multiple factors or objectives at the same time, so as to better balance the conflicts and trade-offs between different decision variables, so as to further optimize the solution results of each decision variable.
[0029] In an optional implementation, the objective function includes: a first constraint function, the first constraint function being used to constrain a solution result of the second posture information based on a constraint condition;
[0030] Based on the decision variables and constraints, construct the objective function, including:
[0031] Based on the corresponding turning angle data and wheel speed pulse signals of the target vehicle between adjacent image frames, the position and posture of the target vehicle in the adjacent image frames are calculated to obtain a calculation result;
[0032] Based on the second pose information and the environment point cloud, a first reprojection constraint is constructed, wherein the first reprojection constraint is used to constrain the position difference between the projected point cloud and the environment point cloud obtained by projecting the second pose information onto adjacent image frames to satisfy a constraint condition;
[0033] Based on the second pose information, the calculated result and the preset unit matrix, a first pose constraint is constructed, wherein the first pose constraint is used to indicate that the difference between the second pose information, the calculated result and the preset unit matrix satisfies the constraint condition;
[0034] A first constraint function is constructed based on the weights corresponding to the first reprojection constraint and the first pose constraint respectively.
[0035] In the embodiment of the present disclosure, the first objective sub-function can be used to solve the current moment The optimal decision variable for the corresponding image frame , thus Pre-optimization is performed to avoid solving the objective function through global optimization , the solution result does not converge, and at the same time, the optimization of the target vehicle's running trajectory is achieved.
[0036] In an optional implementation, the objective function includes: a second constraint function, the second constraint function is used to constrain the solution result of the first pose information based on the constraint condition;
[0037] Based on the decision variables and constraints, construct the objective function, which also includes:
[0038] Based on the first pose information, a second pose constraint is constructed, wherein the second pose constraint is used to constrain the turning angle corresponding to the second estimated result between adjacent image frames to meet the constraint condition;
[0039] Determine the posture observation information of the environment vehicle according to the recognition result, and construct an observation constraint based on the second inference result, the posture observation information and the preset unit matrix, wherein the observation constraint is used to indicate that the difference between the inference result and the observed posture and the preset unit matrix satisfies the constraint condition;
[0040] A second constraint function is constructed based on the weights corresponding to the second posture constraint and the observation constraint respectively.
[0041] In the embodiment of the present disclosure, the first objective sub-function can be used to solve the current moment The optimal decision variable for the corresponding image frame , thus Pre-optimization is performed to avoid solving the objective function through global optimization When the solution is not converged, the smooth optimization of the vehicle trajectory is achieved.
[0042] In an optional implementation, constructing a second pose constraint based on the first pose information includes:
[0043] Based on the first pose information, a first rotation angle constraint is constructed, wherein the first rotation angle constraint is used to constrain the difference of the rotation angles corresponding to the first pose information in adjacent image frames to meet the constraint condition;
[0044] Based on the first pose information, calculate the angular velocity of the environment vehicle in adjacent image frames;
[0045] Obtaining a turning angular velocity threshold of the environment vehicle, and constructing a second turning angle constraint based on the turning angular velocity threshold and the angular velocity, wherein the second turning angle constraint is used to constrain the angular velocity to satisfy a constraint condition;
[0046] A second posture constraint is constructed based on the weights corresponding to the first rotation angle constraint and the second rotation angle constraint respectively.
[0047] In the disclosed embodiment, the turning angle of the environment vehicle between adjacent image frames can be optimized by the second posture constraint, thereby achieving smooth optimization of the running trajectory of the environment vehicle.
[0048] In an optional embodiment, the objective function includes: a second reprojection constraint, the second reprojection constraint being used to constrain a solution result for the point cloud data based on the constraint condition;
[0049] Based on the decision variables and constraints, construct the objective function, which also includes:
[0050] Acquire extrinsic parameters corresponding to the camera device, wherein the extrinsic parameters are used to indicate parameters of a rotation and translation relationship between a coordinate system of the camera device and a coordinate system of the target vehicle;
[0051] Based on the external parameters, the point cloud data, the first pose information and the second pose information, a second reprojection constraint is constructed, wherein the second reprojection constraint is used to constrain that the difference between the projection coordinates obtained by projecting the point cloud data into adjacent image frames and the original coordinates corresponding to the point cloud data in the adjacent image frames satisfies the constraint condition.
[0052] In the embodiment of the present disclosure, the second reprojection constraint can be used to solve the current moment The optimal decision variable for the corresponding image frame , thus Pre-optimization is performed to avoid solving the objective function through global optimization When , the solution does not converge.
[0053] In a second aspect, the present invention provides a dense point cloud estimation device, the device comprising:
[0054] An acquisition module is used to acquire an environmental image frame sequence collected by a camera device of a target vehicle, and identify the environmental vehicle in the environmental image frame sequence to obtain an identification result;
[0055] A first determination module is used to determine the first position information of the environment vehicle based on the recognition result, wherein the first position information is used to indicate the position information corresponding to the environment vehicle in the adjacent image frames of the environment image frame sequence;
[0056] A second determination module is used to identify the second posture information corresponding to the target vehicle in the adjacent image frames, and determine the point cloud data of the environment vehicle according to the first posture information and the second posture information;
[0057] The construction module is used to construct an objective function according to the first pose information, the second pose information and the point cloud data to output a dense point cloud of the environment vehicle that meets the constraint conditions.
[0058] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor are communicatively connected to each other, computer instructions are stored in the memory, and the processor executes the dense point cloud estimation method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.
[0059] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the dense point cloud estimation method of the first aspect or any corresponding embodiment thereof.
[0060] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions, wherein the computer instructions are used to enable a computer to execute the dense point cloud estimation method of the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS
[0061] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0062] Figure 1 is a flow chart of a dense point cloud estimation method according to an embodiment of the present invention;
[0063] Figure 2 is a flow chart of another dense point cloud estimation method according to an embodiment of the present invention;
[0064] Figure 3 It is a schematic diagram for determining three-dimensional point cloud data;
[0065] Figure 4 is a flowchart of another dense point cloud estimation method according to an embodiment of the present invention;
[0066] Figure 5 It is the corresponding position change of the environment vehicle in adjacent image frames when turning;
[0067] Figure 6 is a structural block diagram of a dense point cloud estimation device according to an embodiment of the present invention;
[0068] Figure 7 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION
[0069] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.
[0070] In conjunction with the application scenarios on which the execution of the dense point cloud estimation method depends, the application scenarios are described here.
[0071] In the field of unmanned driving, real-time perception of the surrounding environment of the vehicle is the basis for the decision-making and control of the unmanned driving system. The relevant environmental perception technology has also received much attention from academia and industry. In order to reduce the annotation cost of deep learning in the field of unmanned driving and improve the annotation efficiency, a dense map corresponding to the driving environment can be constructed. The annotators perform 4D annotation on the dense map and then back-project the annotation results back into the image, which consistently completes the task of annotating the image. However, in the relevant 4D annotation schemes, it is often necessary to collect the dense point cloud corresponding to the environment through sensors that perceive external information such as lidar and millimeter-wave radar to obtain a dense map. Therefore, there is a high demand for hardware equipment configuration, which increases the cost of 4D annotation.
[0072] However, compared with sensors that perceive external information such as lidar and millimeter-wave radar, cameras can perceive the richest external information, such as the color, structure, texture and some semantic information of the scenery around the vehicle (such as roads, pedestrians, traffic signs, etc.). Thanks to the breakthrough of deep learning in the field of computer vision, this information can be identified, tracked and even restored to its three-dimensional structure.
[0073] On this basis, the present disclosure can first obtain the environmental image frame sequence collected by the camera device of the target vehicle, and identify the environmental vehicle in the environmental image frame sequence to obtain the recognition result, so as to determine the first pose information of the environmental vehicle based on the recognition result, wherein the first pose information is used to indicate the pose information corresponding to the environmental vehicle in the adjacent image frames of the environmental image frame sequence. Then, the second pose information corresponding to the target vehicle in the adjacent image frame can be identified, and the point cloud data of the environmental vehicle can be determined based on the first pose information and the second pose information. Next, a constraint function can be constructed based on the first pose information, the second pose information and the point cloud data to output a dense point cloud of the environmental vehicle that satisfies the constraint function, thereby realizing the dense point cloud reconstruction of the environmental vehicle based on the environmental image frame sequence collected by the camera device, and reducing the cost of 4D annotation.
[0074] According to an embodiment of the present invention, an embodiment of a dense point cloud estimation method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order different from that shown here.
[0075] In this embodiment, a dense point cloud estimation method is provided, which can be used in vehicle terminal equipment. Figure 1 is a flow chart of a dense point cloud estimation method according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:
[0076] Step S101 , obtaining an environment image frame sequence captured by a camera device of a target vehicle, and identifying an environment vehicle in the environment image frame sequence to obtain a recognition result.
[0077] In the disclosed embodiment, the above-mentioned camera device can be a terminal device such as a camera installed on the target vehicle, and the camera device is used to collect environmental images during the driving process of the target vehicle to obtain an environmental image frame sequence.
[0078] Here, the above-mentioned environmental image frame sequence monocular image sequence, that is, a series of image frames of the driving environment captured by a single camera, is used to construct the following dense point cloud based on the monocular image sequence, without the need to use multiple camera devices to capture environmental images at different angles to construct the dense point cloud, which further reduces the implementation cost of the present solution.
[0079] After acquiring the above-mentioned environmental image frame sequence, the environmental image frame sequence can be input into the trained neural network to output the recognition results such as the 3D frame coordinates of the environmental vehicle in the environmental image frame sequence, the environmental vehicle instance segmentation, and the optical flow sequence of the environmental vehicle. Among them, the 3D frame coordinates are used to indicate the coordinates of the 3D recognition frame of the environmental vehicle in the corresponding image frame, the instance segmentation is used to indicate the result of segmenting the environmental vehicle in the environmental image frame, and the optical flow sequence refers to the two-dimensional velocity field extracted from the video or continuous image frame by the image processing method, which represents the moving speed and direction of each pixel point in the image over time.
[0080] Step S102, based on the recognition result, determine the first position information of the environment vehicle, wherein the first position information is used to indicate the corresponding position information of the environment vehicle in the adjacent image frames of the environment image frame sequence.
[0081] In the embodiment of the present disclosure, the adjacent image frames may include the current moment The corresponding image frame and the next moment Therefore, here, if the above-mentioned environment vehicle is the first one in the environment image frame sequence If there is an environmental vehicle, then the first position information may include the environmental vehicle exist The posture corresponding to the moment ,in, Used to indicate the surrounding vehicle exist The direction of the moment, Used to indicate the surrounding vehicle exist In addition, the first position information can also include the environment vehicle exist The posture corresponding to the moment ,in, Used to indicate the surrounding vehicle exist The direction of the moment, Used to indicate the surrounding vehicle exist The position at the moment.
[0082] Step S103, identifying the second posture information corresponding to the target vehicle in the adjacent image frames, and determining the point cloud data of the environment vehicle according to the first posture information and the second posture information.
[0083] In the embodiment of the present disclosure, the above environment image frame sequence can be expressed as [ ], based on this, the position of the target vehicle in frame 0 can be calculated by dead reckoning. The posture corresponding to the moment , and the target vehicle in The posture corresponding to the moment .
[0084] When calculating the second position information of the target vehicle by dead reckoning, the position of the first frame can be calculated based on the position of the target vehicle in the 0th frame. Specifically, the steering wheel angle and wheel speed pulse data can be obtained based on the sensor system in the target vehicle. Here, the direction of the target vehicle in the first frame can be determined by the steering wheel angle, and the distance moved by the vehicle from the 0th frame to the 1st frame can be calculated by the wheel speed pulse data, wherein the length of the vehicle movement can be determined by the wheel speed. Similarly, forward propagation can be performed based on the dead reckoning method to calculate the position of the target vehicle in the 1st frame. The posture corresponding to the moment and The posture corresponding to the moment .
[0085] After determining the above-mentioned first pose information and second pose information, the point cloud in the environment vehicle can be triangulated based on the first pose information and the second pose information to obtain point cloud data of the environment vehicle, thereby improving the point cloud density and recognition accuracy, which is convenient for the following filtering of the point cloud data.
[0086] Step S104, constructing an objective function according to the first pose information, the second pose information and the point cloud data to output a dense point cloud of the environment vehicle that meets the constraint conditions.
[0087] In the disclosed embodiment, the above-mentioned first pose information, second pose information and point cloud data can be used as decision variables, and the decision variables can be constrained by constraints to obtain an objective function, wherein the objective function is a minimization function, and the constraints limit the value range of the decision variables to satisfy the minimum value of the objective function.
[0088] Specifically, the objective function can be optimized by the constraint function in the objective function to solve the optimal first pose information, second pose information and point cloud data based on the objective function, wherein the point cloud data is a dense point cloud of the environment vehicle.
[0089] It can be known from the above description that in the embodiment of the present disclosure, first, the environment image frame sequence collected by the camera device of the target vehicle can be obtained, and the environment vehicle in the environment image frame sequence can be identified to obtain the identification result, so as to determine the first pose information of the environment vehicle based on the identification result, wherein the first pose information is used to indicate the pose information corresponding to the environment vehicle in the adjacent image frames of the environment image frame sequence. Then, the second pose information corresponding to the target vehicle in the adjacent image frame can be identified, and the point cloud data of the environment vehicle can be determined based on the first pose information and the second pose information. Next, a constraint function can be constructed based on the first pose information, the second pose information and the point cloud data to output a dense point cloud of the environment vehicle that satisfies the constraint function, thereby realizing the dense point cloud reconstruction of the environment vehicle based on the environment image frame sequence collected by the camera device, and reducing the cost of 4D annotation.
[0090] In this embodiment, another dense point cloud estimation method is provided, which can be used for the above-mentioned vehicle terminal device. Figure 2 is a flowchart of another dense point cloud estimation method according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:
[0091] Step S201, obtain the environment image frame sequence collected by the camera device of the target vehicle, and identify the environment vehicle in the environment image frame sequence to obtain the recognition result. Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.
[0092] Step S202: Based on the recognition result, determine the first pose information of the environment vehicle, wherein the first pose information is used to indicate the pose information corresponding to the environment vehicle in the adjacent image frames of the environment image frame sequence. Figure 1 Step S102 of the illustrated embodiment will not be described in detail here.
[0093] Step S203, identifying the second posture information corresponding to the target vehicle in the adjacent image frame, and determining the point cloud data of the environment vehicle according to the first posture information and the second posture information.
[0094] Specifically, the above step S203 includes:
[0095] Step S2031, based on the external parameters corresponding to the camera device, estimate the three-dimensional coordinates of the two-dimensional point cloud of the environment vehicle at the time corresponding to the adjacent image frames, wherein the external parameters are used to indicate the parameters of the rotation and translation relationship between the coordinate system of the camera device and the coordinate system of the target vehicle.
[0096] Step S2032, based on the first pose information, the second pose information and the corresponding pixel coordinates, back-project the two-dimensional point cloud to obtain three-dimensional point cloud data.
[0097] In the embodiment of the present disclosure, the above external parameters can be expressed as ,in, is a parameter indicating the rotation relationship between the coordinate system of the camera device and the vehicle coordinate system, is a parameter indicating the translation relationship between the coordinate system of the camera device and the coordinate system of the target vehicle.
[0098] It should be understood that if the adjacent image frames include the current moment The corresponding image frame and the next moment The corresponding image frame can calculate the coordinates of each three-dimensional point cloud in the environment vehicle based on the adjacent image frame to obtain the above-mentioned three-dimensional coordinates, and determine the three-dimensional coordinates as the above-mentioned three-dimensional point cloud data.
[0099] like Figure 3 The figure shows a schematic diagram of determining the above three-dimensional point cloud data, wherein the environment vehicle is calculated 3D point cloud in For example, For environmental vehicles At the present moment The corresponding local coordinate system is For environmental vehicles At the next moment The corresponding local coordinate system is A camera device for representing time k, for Specifically, the camera device is The pose at the moment can be expressed as , the camera device is The pose at the moment can be expressed as , specifically solve the three-dimensional point cloud The coordinate method is as follows and will not be repeated here.
[0100] Step S204: construct an objective function based on the first pose information, the second pose information and the point cloud data to output a dense point cloud of the environment vehicle that meets the constraint conditions. Figure 1 Step S104 of the illustrated embodiment will not be described in detail here.
[0101] In the disclosed embodiment, the point cloud data of the above-mentioned environment vehicle can be solved by two adjacent image frames. It should be understood that each two-dimensional point cloud in the environment vehicle can correspond to the solution results of multiple three-dimensional point clouds, thereby improving the point cloud density and recognition accuracy in the final output three-dimensional dense point cloud, and providing a technical basis for the following filtering of the point cloud data.
[0102] In some optional implementations, the adjacent image frames include an image frame at a first moment and an image frame at a second moment, and the step S2032 includes:
[0103] Step a1, constructing a first projection function based on the first sub-pose information corresponding to the first pose information in the image frame at the first moment, the second sub-pose information corresponding to the second pose information in the image frame at the first moment, the three-dimensional first sub-coordinate corresponding to the two-dimensional point cloud at the first moment, and a preset scale coefficient, wherein the first projection function is used to back-project the two-dimensional point cloud based on the image frame at the first moment.
[0104] In the embodiment of the present disclosure, the intrinsic parameters of the above-mentioned camera device can be obtained, wherein the intrinsic parameters include the projection function With the back projection function Specifically, the projection function Used to map 3D point cloud to 2D, back projection function Used to map 2D point clouds to 3D.
[0105] Based on this, we can use the above back-projection function Construct the first projection function, which is as follows:
[0106]
[0107] in, For environmental vehicles The 3D point cloud on is the first sub-pose information mentioned above, is the second sub-pose information mentioned above, For environmental vehicles The three-dimensional point cloud on The two-dimensional coordinates of the time, is the first sub-coordinate of the above three dimensions, Preset scale factor.
[0108] Step a2, constructing a second projection function based on the third sub-pose information corresponding to the first pose information in the image frame at the second moment, the fourth sub-pose information corresponding to the second pose information in the image frame at the second moment, the three-dimensional second sub-coordinate corresponding to the two-dimensional point cloud at the second moment, and a preset scale coefficient, wherein the second projection function is used to back-project the two-dimensional point cloud based on the image frame at the second moment.
[0109] In the embodiment of the present disclosure, based on the above back-projection function Constructing the first projection function to solve the above second projection function is as follows:
[0110]
[0111] in, For environmental vehicles The 3D point cloud on is the third sub-pose information mentioned above, is the fourth sub-pose information mentioned above, For environmental vehicles The three-dimensional point on The two-dimensional coordinates of the time, is the second sub-coordinate of the above three dimensions, Preset scale factor.
[0112] Step a3: construct a set of simultaneous equations according to the first projection function and the second projection function, and calculate three-dimensional point cloud data corresponding to the two-dimensional point cloud based on the set of simultaneous equations.
[0113] In the embodiment of the present disclosure, after constructing a system of simultaneous equations based on the first projection function and the second projection function, the unknowns in the system of simultaneous equations can be solved based on the least squares method, and the solution result is: Figure 3 middle .
[0114] In the disclosed embodiment, the point cloud data of the above-mentioned environment vehicle can be solved by two adjacent image frames. It should be understood that each two-dimensional point cloud in the environment vehicle can correspond to the solution results of multiple three-dimensional point clouds, thereby improving the point cloud density and recognition accuracy in the final output three-dimensional dense point cloud, and providing a technical basis for the following filtering of the point cloud data.
[0115] In some optional implementations, after determining the point cloud data of the environment vehicle according to the first pose information and the second pose information, the above Figure 1 The corresponding embodiment also includes the following process:
[0116] Step b1, identifying target points in the point cloud data that do not meet the observation conditions.
[0117] Step b2: filtering the point cloud data based on the target point to obtain a filtering result.
[0118] Step b3: When the noise value of the filtering result does not meet the preset threshold, the filtering result is filtered until the noise value meets the preset threshold, and the filtered point cloud data is output.
[0119] In the disclosed embodiment, the optical flow sequence in the above recognition result also has certain errors, which may lead to errors in the recognition of the two-dimensional point cloud. At the same time, it can be seen from the above that when solving the point cloud data of the above environment vehicle through two adjacent image frames, each two-dimensional point cloud can correspond to the solution results of multiple three-dimensional point clouds. Therefore, based on this, the solved point cloud data can be filtered to improve the confidence of the point cloud data. Specifically, the above point cloud data can be filtered first, that is, the target points that do not meet the observation conditions, such as outliers, can be filtered out to obtain the filtering result. Here, the filtered point cloud is recorded as .
[0120] Next, two frames can be randomly selected from the above environment image frame sequence, and the corresponding filtering results can be obtained. For example, Frame and k The filtering result of the frame is recorded as , and based on Construct an observation model as shown below:
[0121]
[0122] in, represents the probability of correct observation, Indicates correct observation of obedience The mean value is is a Gaussian distribution with covariance, Indicates that the error observation obeys the uniform distribution. Based on this, the approximately optimal posterior distribution can be inferred by variational method as:
[0123]
[0124] in, express distributed, To a certain extent, it indicates the number of correct observations and the number of incorrect observations. Then, according to the Bayesian filter update strategy, we have: = .
[0125] in, is the normalization constant. With an approximately optimal posterior distribution form, The first-order moment and the second-order moment are equal to approximate update .here, is the mean of the 3D point cloud in the filter, is the covariance matrix of the three-dimensional point cloud, Represents the number of correct observations and the number of incorrect observations.
[0126] When determining whether the noise value of the above filtering result meets the preset threshold, the covariance matrix of the three-dimensional point cloud in the above filtering result can be sequentially Compared with the preset threshold th, if Less than th, it means The corresponding three-dimensional point cloud has converged. If Greater than th, it means The corresponding three-dimensional point cloud is a non-convergent noise point. Next, the noise value of the filtering result can be calculated based on the proportion of noise points in the filtering result. If the noise value is greater than the preset threshold, the filtering result is filtered through the above filtering process until the noise value meets the preset threshold to output the filtered point cloud data.
[0127] In the disclosed embodiment, the above-mentioned point cloud data can be filtered to make the three-dimensional point cloud in the point cloud data converge, so that the point cloud data converges as a whole, providing a technical basis for further global optimization of the point cloud data through the objective function described below.
[0128] In this embodiment, another dense point cloud estimation method is provided, which can be used for the above-mentioned vehicle terminal device. Figure 4 is a flowchart of another dense point cloud estimation method according to an embodiment of the present invention. Figure 4 As shown, the process includes the following steps:
[0129] Step S401, obtain the environment image frame sequence collected by the camera device of the target vehicle, and identify the environment vehicle in the environment image frame sequence to obtain the recognition result. Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.
[0130] Step S402: Based on the recognition result, determine the first pose information of the environment vehicle, wherein the first pose information is used to indicate the pose information corresponding to the environment vehicle in the adjacent image frames of the environment image frame sequence. Figure 1 Step S102 of the illustrated embodiment will not be described in detail here.
[0131] Step S403, identifying the second pose information corresponding to the target vehicle in the adjacent image frame, and determining the point cloud data of the surrounding vehicle according to the first pose information and the second pose information. Figure 1 Step S103 of the illustrated embodiment will not be described in detail here.
[0132] Step S404, constructing an objective function according to the first pose information, the second pose information and the point cloud data to output a dense point cloud of the environment vehicle that meets the constraint conditions.
[0133] Specifically, the above step S204 includes:
[0134] Step S4041, obtaining the environment point cloud in adjacent image frames.
[0135] Step S4042, taking the environment point cloud, the first pose information, the second pose information and the point cloud data as decision variables, and determining the constraint conditions corresponding to the decision variables.
[0136] Step S4043, constructing an objective function based on decision variables and constraints.
[0137] In the embodiment of the present disclosure, the environment point cloud in the adjacent image frames may be first obtained, and the initial position value of the environment point cloud in the environment image frame sequence may be determined. , so as to pass the initial position value in the following objective function To represent the environment point cloud, wherein the initial position value is used to indicate the position value of the environment point cloud when it is first monitored in the above-mentioned environment image frame sequence.
[0138] In the above environment point cloud , first position information , Second pose information And point cloud data As decision variables, and determine the constraints corresponding to each constraint variable respectively, to construct the objective function as shown below:
[0139]
[0140] Among them, the objective function is used to solve the value of the decision variable that satisfies the minimization value, so as to solve the current moment The optimal first pose information, second pose information and point cloud data of the corresponding image frame.
[0141] In the disclosed embodiments, an objective function can be constructed based on multiple decision variables, so as to find the optimal solution for each decision variable in a larger search space, which provides more possibilities and flexibility for optimization. At the same time, multiple decision variables enable the model corresponding to the objective function to consider multiple factors or objectives at the same time, so as to better balance the conflicts and trade-offs between different decision variables, so as to further optimize the solution results of each decision variable.
[0142] In some optional implementations, the objective function includes: a first constraint function, the first constraint function is used to constrain the solution result of the second posture information based on the constraint condition, and the above step S4043 includes the following process:
[0143] Step c1, based on the corresponding turning angle data and wheel speed pulse signals of the target vehicle between adjacent image frames, the position and posture of the target vehicle in adjacent image frames are calculated to obtain a calculation result.
[0144] In the embodiment of the present disclosure, the above-mentioned inference results may include the relative position and posture of the target vehicle between adjacent image frames. Here, the relative position can be calculated based on the corresponding turning angle data and wheel speed pulse signals of the target vehicle between adjacent image frames by the above-mentioned dead reckoning method. , wherein the angle data is the steering wheel angle, the wheel speed pulse signal is the wheel speed pulse data, and the specific track calculation method is as described in the above step S103, which will not be repeated here.
[0145] Step c2: construct a first reprojection constraint based on the second pose information and the environment point cloud, wherein the first reprojection constraint is used to constrain the position difference between the projection point cloud obtained by projecting the second pose information into adjacent image frames and the environment point cloud to meet the constraint condition.
[0146] In the embodiment of the present disclosure, it can be seen from the above that the second posture information includes: as well as Therefore, when constructing the first reprojection constraint, we can first obtain the projection function in the above intrinsic parameters: , based on this Construct the first reprojection constraint, where the Can be used to constrain the two-pose information And the above environment point cloud Specifically, the first projection constraint is as follows:
[0147]
[0148] in, Point cloud for the environment At the current moment in the above adjacent image frames The pixel coordinates in the corresponding image frame, For the general The projection coordinates of the projected point cloud obtained by projecting into the adjacent image frame. Therefore, the first projection constraint is used to constrain the difference between the pixel coordinates and the projection coordinates to satisfy the constraint condition. When the constraint condition is 0, it means that the projection coordinates coincide with the pixel coordinates, that is, the error between the predicted second pose information and the true value is small.
[0149] Step c3, constructing a first pose constraint based on the second pose information, the calculated result and the preset unit matrix, wherein the first pose constraint is used to indicate that the difference between the second pose information, the calculated result and the preset unit matrix satisfies the constraint condition.
[0150] In the embodiment of the present disclosure, the first pose constraint is used to constrain the second pose information based on the calculation result. , so that As close to the true value as possible, the first pose constraint is as follows:
[0151]
[0152] in, is the preset unit matrix mentioned above.
[0153] Step c3: construct a first constraint function based on the weights corresponding to the first reprojection constraint and the first pose constraint.
[0154] In the embodiment of the present disclosure, the weight corresponding to the first reprojection constraint can be , the weight corresponding to the first pose constraint can be , the weight can be obtained by assuming that the observation follows a Gaussian distribution, and the specific method of determining the weight is not repeated in this disclosure. Based on this, the above first constraint function can be constructed In addition, the first objective sub-function shown below can be constructed based on the first constraint function:
[0155]
[0156] Among them, the objective function is used to solve the value of the decision variable that satisfies the minimization value, so as to solve the current moment The optimal decision variable for the corresponding image frame .
[0157] In the embodiment of the present disclosure, the first objective sub-function can be used to solve the current moment The optimal decision variable for the corresponding image frame , thus Pre-optimization is performed to avoid solving the objective function through global optimization , the solution result does not converge, and at the same time, the optimization of the target vehicle's running trajectory is achieved.
[0158] In some optional implementations, the objective function includes: a second constraint function, the second constraint function is used to constrain the solution result of the first pose information based on the constraint condition, and the above step S4043 also includes the following process:
[0159] Step d1, constructing a second pose constraint based on the first pose information, wherein the second pose constraint is used to constrain the turning angle corresponding to the second inference result between adjacent image frames to meet the constraint condition.
[0160] Specifically, the above step d1 includes the following process:
[0161] (1) Based on the first pose information, a first rotation angle constraint is constructed, wherein the first rotation angle constraint is used to constrain the difference of the rotation angles corresponding to the second pose constraint in adjacent image frames to meet the constraint condition;
[0162] (2) Based on the first pose information, calculate the angular velocity of the environment vehicle in adjacent image frames;
[0163] (3) obtaining a turning angular velocity threshold of the environment vehicle, and constructing a second turning angle constraint based on the turning angular velocity threshold and the angular velocity, wherein the second turning angle constraint is used to constrain the angular velocity to satisfy a constraint condition;
[0164] (4) Construct a second posture constraint based on the weights corresponding to the first rotation angle constraint and the second rotation angle constraint respectively.
[0165] In the embodiment of the present disclosure, it can be seen from the above that the first position information includes as well as Based on this, the first rotation angle constraint constructed based on the first pose information is as follows:
[0166]
[0167] in, represents the cross product, express and Therefore, the first rotation angle constraint is used to constrain the difference between the rotation angles of adjacent image frames when the vehicle turns in the environment. and The difference in rotation angle between them is 0.
[0168] Specifically, Figure 5 The figure shows the corresponding position change of the environment vehicle in adjacent image frames when turning. for The corresponding rotation angle, for The corresponding rotation angle is, For environmental vehicles The position at the time The difference in position at each moment can be known from the kinematic formula: and The first turning angle constraint is used to constrain the adjacent image frames when the environment vehicle turns. and The difference in rotation angle between them is 0.
[0169] Considering that in the first rotation angle constraint above, When is 0, that is, when the environment vehicle is stationary but oriented in any direction, the first rotation angle constraint also holds. Therefore, the first pose information solved when the environment vehicle is stationary can be constrained by constructing the above-mentioned second rotation angle constraint.
[0170] Specifically, the angular velocity of the environment vehicle in the adjacent image frames can be calculated based on the first pose information. ,Should ,in, is a function that maps a rotation matrix to the corresponding angular axis angle, is the time interval between adjacent image frames.
[0171] It should be understood that the above angular velocity Should meet ,in, is the maximum turning angular velocity of the vehicle, that is, the turning angular velocity threshold. ,in, is the minimum turning radius of the vehicle.
[0172] Based on this, we can The following is constructed to constrain the angular velocity The second corner constraint:
[0173]
[0174] in, is a pre-set positive parameter. The constraint condition corresponding to this constraint function is to satisfy This constraint function is used to constrain the first pose information so that , as well as In this case, the angular velocity determined based on the first pose information is Always satisfied .
[0175] In the disclosed embodiment, the turning angle of the environment vehicle between adjacent image frames can be optimized by the second posture constraint, thereby achieving smooth optimization of the running trajectory of the environment vehicle.
[0176] Step d2, determines the posture observation information of the environmental vehicle according to the recognition result, and constructs the observation constraint based on the first posture information, the posture observation information and the preset unit matrix, wherein the observation constraint is used to indicate that the calculated result and the difference between the observed posture and the preset unit matrix satisfy the constraint condition.
[0177] Step d3, constructing a second constraint function based on the weights corresponding to the second posture constraint and the observation constraint respectively.
[0178] In the embodiment of the present disclosure, it can be seen from the above that the recognition results include: the 3D frame coordinates of the environment vehicle in the environment image frame sequence, the environment vehicle instance segmentation and the optical flow sequence of the environment vehicle, etc. Based on this, the 3D frame coordinates can be observed through a neural network to obtain the 3D frame pose observation information of the environment vehicle. ,Should .
[0179] Therefore, the observation constraints constructed based on the above first pose information, pose observation information and preset unit matrix are as follows:
[0180]
[0181] Next, a second constraint function can be constructed based on the second posture constraint and the observation constraint. The second constraint function can be expressed as: ,in, is the weight corresponding to the first corner constraint, is the weight corresponding to the second corner constraint, is the weight corresponding to the observation constraint.
[0182] In addition, the second objective sub-function shown below can be constructed based on the above second constraint function:
[0183]
[0184] Among them, the objective function is used to solve the value of the decision variable that satisfies the minimization value, so as to solve the current moment The optimal decision variable for the corresponding image frame .
[0185] In the embodiment of the present disclosure, the first objective sub-function can be used to solve the current moment The optimal decision variable for the corresponding image frame , thus Pre-optimization is performed to avoid solving the objective function through global optimization When the solution is not converged, the smooth optimization of the vehicle trajectory is achieved.
[0186] In some optional implementations, the objective function includes: a second reprojection constraint, the second reprojection constraint is used to constrain the solution result of the point cloud data based on the constraint condition, and the above step S4043 also includes the following process:
[0187] Step d1, obtaining extrinsic parameters corresponding to the camera device, wherein the extrinsic parameters are used to indicate parameters of the rotation and translation relationship between the coordinate system of the camera device and the coordinate system of the target vehicle.
[0188] Step d2, constructing a second reprojection constraint based on external parameters, point cloud data, first pose information and second pose information, wherein the second reprojection constraint is used to constrain the difference between the projection coordinates obtained by projecting the point cloud data into adjacent image frames and the original coordinates corresponding to the point cloud data in the adjacent image frames to satisfy the constraint condition.
[0189] In the embodiment of the present disclosure, the projection function in the internal parameters of the above-mentioned camera device can be obtained first. , based on this Construct the second reprojection constraint as shown below:
[0190]
[0191] in, are the external parameters mentioned above, For the above point cloud data, is the first pose information mentioned above, is the second pose information mentioned above, Used to indicate The current moment projected into the adjacent image frame The corresponding image frame, the projection coordinates are obtained, for At the present moment The corresponding original coordinates in the corresponding image frame. In the above second reprojection constraint, when the difference between the projection coordinates and the original coordinates satisfies the constraint condition 0, it means that the projection coordinates coincide with the original coordinates, that is, the error between the point cloud data and the corresponding true value is small.
[0192] In the embodiment of the present disclosure, the second reprojection constraint can be used to solve the current moment The optimal decision variable for the corresponding image frame , thus Pre-optimization is performed to avoid solving the objective function through global optimization When , the solution does not converge.
[0193] It should be understood that after respectively determining the first constraint function, the second constraint function and the second reprojection function to construct the global optimization objective function ,in:
[0194]
[0195] in, The weight corresponding to the second reprojection constraint.
[0196] In summary, in the disclosed embodiment, first, the environment image frame sequence captured by the camera device of the target vehicle can be obtained, and the environment vehicle in the environment image frame sequence can be identified to obtain an identification result, so as to determine the first pose information of the environment vehicle based on the identification result, wherein the first pose information is used to indicate the pose information corresponding to the environment vehicle in the adjacent image frames of the environment image frame sequence. Then, the second pose information corresponding to the target vehicle in the adjacent image frames can be identified, and the point cloud data of the environment vehicle can be determined based on the first pose information and the second pose information. Next, a constraint function can be constructed based on the first pose information, the second pose information and the point cloud data to output a dense point cloud of the environment vehicle that satisfies the constraint function, thereby realizing the dense point cloud reconstruction of the environment vehicle based on the environment image frame sequence captured by the camera device, and reducing the cost of 4D annotation.
[0197] In this embodiment, a dense point cloud estimation device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and will not be repeated here. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.
[0198] This embodiment provides a dense point cloud estimation device, such as Figure 6 As shown, including:
[0199] The acquisition module 601 is used to acquire the environment image frame sequence collected by the camera device of the target vehicle, and identify the environment vehicle in the environment image frame sequence to obtain the recognition result;
[0200] A first determination module 602 is used to determine the first position information of the environment vehicle based on the recognition result, wherein the first position information is used to indicate the position information corresponding to the environment vehicle in the adjacent image frames of the environment image frame sequence;
[0201] The second determination module 603 is used to identify the second posture information corresponding to the target vehicle in the adjacent image frame, and determine the point cloud data of the environment vehicle according to the first posture information and the second posture information;
[0202] The construction module 604 is used to construct an objective function according to the first pose information, the second pose information and the point cloud data to output a dense point cloud of the environment vehicle that meets the constraint conditions.
[0203] In some optional implementations, the second determining module 603 includes:
[0204] An estimation unit, used to estimate the extrinsic parameters corresponding to the camera device and the three-dimensional coordinates of the two-dimensional point cloud of the environment vehicle at the time corresponding to the adjacent image frames, wherein the extrinsic parameters are used to indicate the parameters of the rotation and translation relationship between the coordinate system of the camera device and the coordinate system of the target vehicle;
[0205] The back-projection unit is used to back-project the two-dimensional point cloud based on the first pose information, the second pose information and the three-dimensional coordinates to obtain three-dimensional point cloud data.
[0206] In some optional implementations, the adjacent image frames include an image frame at a first moment and an image frame at a second moment; and the back projection unit includes:
[0207] A first construction subunit is used to construct a first projection function based on first sub-pose information corresponding to the first pose information in the image frame at the first moment, second sub-pose information corresponding to the second pose information in the image frame at the first moment, a three-dimensional first sub-coordinate corresponding to the two-dimensional point cloud at the first moment, and a preset scale coefficient, wherein the first projection function is used to back-project the two-dimensional point cloud based on the image frame at the first moment;
[0208] A second construction subunit is used to construct a second projection function based on the third sub-pose information corresponding to the first pose information in the image frame at the second moment, the fourth sub-pose information corresponding to the second pose information in the image frame at the second moment, the three-dimensional second sub-coordinates corresponding to the two-dimensional point cloud at the second moment, and the preset scale coefficient, wherein the second projection function is used to back-project the two-dimensional point cloud based on the image frame at the second moment;
[0209] The calculation subunit is used to construct a set of simultaneous equations according to the first projection function and the second projection function, and calculate three-dimensional point cloud data corresponding to the two-dimensional point cloud based on the set of simultaneous equations.
[0210] In some optional embodiments, the above device further includes:
[0211] An identification module, for identifying target points in the point cloud data that do not meet the observation conditions after determining the point cloud data of the environment vehicle according to the first pose information and the second pose information;
[0212] A filtering module is used to filter the point cloud data based on the target point to obtain a filtering result;
[0213] The output module is used to filter the filtering result when the noise value of the filtering result does not meet the preset threshold value, until the noise value meets the preset threshold value, and output the filtered point cloud data.
[0214] In some optional implementations, the building module 604 includes:
[0215] The acquired environment point clouds in adjacent image frames;
[0216] A determination unit, used to use the environment point cloud, the first pose information, the second pose information and the point cloud data as decision variables, and determine the constraint conditions corresponding to the decision variables;
[0217] The construction unit is used to construct the objective function based on the decision variables and constraints.
[0218] In some optional implementations, the objective function includes: a first constraint function, the first constraint function is used to constrain the solution result of the second posture information based on the constraint condition; the construction unit includes:
[0219] The inference subunit is used to infer the position and posture of the target vehicle in adjacent image frames based on the corresponding turning angle data and wheel speed pulse signals of the target vehicle between adjacent image frames to obtain an inference result;
[0220] A third construction subunit is used to construct a first reprojection constraint based on the second pose information and the environment point cloud, wherein the first reprojection constraint is used to constrain the position difference between the projection point cloud and the environment point cloud obtained by projecting the second pose information onto adjacent image frames to satisfy a constraint condition;
[0221] A fourth construction subunit is used to construct a first pose constraint based on the second pose information, the calculation result and the preset unit matrix, wherein the first pose constraint is used to indicate that the difference between the second pose information, the calculation result and the preset unit matrix satisfies the constraint condition;
[0222] The fifth construction subunit is used to construct a first constraint function based on the weights corresponding to the first reprojection constraint and the first pose constraint respectively.
[0223] In some optional implementations, the objective function includes: a second constraint function, the second constraint function is used to constrain the solution result of the first pose information based on the constraint condition; the construction unit includes:
[0224] a sixth construction subunit, configured to construct a second posture constraint based on the first posture information, wherein the second posture constraint is configured to constrain a turning angle corresponding to the second estimated result between adjacent image frames to satisfy a constraint condition;
[0225] A seventh construction subunit is used to determine the posture observation information of the environment vehicle according to the recognition result, and to construct an observation constraint based on the second inference result, the posture observation information and the preset unit matrix, wherein the observation constraint is used to indicate that the difference between the inference result and the observed posture and the preset unit matrix satisfies the constraint condition;
[0226] The eighth construction subunit is used to construct a second constraint function based on the weights corresponding to the second posture constraint and the observation constraint respectively.
[0227] In some optional embodiments, the sixth building block is used for:
[0228] Based on the first pose information, a first rotation angle constraint is constructed, wherein the first rotation angle constraint is used to constrain the difference of the rotation angles corresponding to the first pose information in adjacent image frames to meet the constraint condition;
[0229] Based on the first pose information, calculate the angular velocity of the environment vehicle in adjacent image frames;
[0230] Obtaining a turning angular velocity threshold of the environment vehicle, and constructing a second turning angle constraint based on the turning angular velocity threshold and the angular velocity, wherein the second turning angle constraint is used to constrain the angular velocity to satisfy a constraint condition;
[0231] A second posture constraint is constructed based on the weights corresponding to the first rotation angle constraint and the second rotation angle constraint respectively.
[0232] In some optional embodiments, the objective function includes: a second reprojection constraint, the second reprojection constraint is used to constrain the solution result of the point cloud data based on the constraint condition; the construction unit includes:
[0233] An acquisition subunit, used to acquire extrinsic parameters corresponding to the camera device, wherein the extrinsic parameters are used to indicate parameters of a rotation and translation relationship between a coordinate system of the camera device and a coordinate system of the target vehicle;
[0234] The ninth construction subunit is used to construct a second reprojection constraint based on external parameters, point cloud data, first pose information and second pose information, wherein the second reprojection constraint is used to constrain the difference between the projection coordinates obtained by projecting the point cloud data into adjacent image frames and the original coordinates corresponding to the point cloud data in the adjacent image frames to meet the constraint condition.
[0235] The dense point cloud estimation device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.
[0236] The embodiment of the present invention also provides a computer device having the above Figure 6 The dense point cloud estimation device shown.
[0237] See also Figure 7 , Figure 7 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 7As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 7 A processor 10 is taken as an example.
[0238] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.
[0239] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.
[0240] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0241] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.
[0242] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 7 The example of connecting through bus is taken in the following.
[0243] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device may be a touch screen.
[0244] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.
[0245] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.
[0246] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.
Claims
1. A dense point cloud estimation method, characterized in that: The method comprises: Acquire an environmental image frame sequence captured by a camera device of the target vehicle, and identify the environmental vehicle in the environmental image frame sequence to obtain an identification result; Based on the recognition result, determining first pose information of the environment vehicle, wherein the first pose information is used to indicate the pose information corresponding to the environment vehicle in adjacent image frames of the environment image frame sequence; Identify the second posture information corresponding to the target vehicle in the adjacent image frame, and determine the point cloud data of the environment vehicle according to the first posture information and the second posture information, wherein the determining the point cloud data of the environment vehicle according to the first posture information and the second posture information includes: Based on the extrinsic parameters corresponding to the camera device, the three-dimensional coordinates of the two-dimensional point cloud of the environment vehicle at the time corresponding to the adjacent image frames are estimated, wherein the extrinsic parameters are used to indicate the parameters of the rotation and translation relationship between the coordinate system of the camera device and the coordinate system of the target vehicle; Based on the first posture information, the second posture information and the three-dimensional coordinates, back-project the two-dimensional point cloud to obtain three-dimensional point cloud data; Constructing an objective function according to the first pose information, the second pose information, and the point cloud data to output a dense point cloud of the environment vehicle that satisfies the constraint conditions, wherein constructing an objective function according to the first pose information, the second pose information, and the point cloud data includes: Acquire the environment point cloud in the adjacent image frames; Taking the environment point cloud, the first pose information, the second pose information and the point cloud data as decision variables, and determining the constraint conditions corresponding to the decision variables; Based on the decision variables and constraints, an objective function is constructed.
2. The method according to claim 1, characterized in that The adjacent image frames include an image frame at a first moment and an image frame at a second moment; The back-projecting of the two-dimensional point cloud based on the first posture information, the second posture information and the corresponding pixel coordinates to obtain three-dimensional point cloud data includes: Constructing a first projection function based on first sub-pose information corresponding to the first pose information in the image frame at the first moment, second sub-pose information corresponding to the second pose information in the image frame at the first moment, three-dimensional first sub-coordinates corresponding to the two-dimensional point cloud at the first moment, and a preset scale coefficient, wherein the first projection function is used to back-project the two-dimensional point cloud based on the image frame at the first moment; Constructing a second projection function based on third sub-pose information corresponding to the first pose information in the image frame at the second moment, fourth sub-pose information corresponding to the second pose information in the image frame at the second moment, three-dimensional second sub-coordinates corresponding to the two-dimensional point cloud at the second moment, and a preset scale coefficient, wherein the second projection function is used to back-project the two-dimensional point cloud based on the image frame at the second moment; A set of simultaneous equations is constructed according to the first projection function and the second projection function, and three-dimensional point cloud data corresponding to the two-dimensional point cloud is calculated based on the set of simultaneous equations.
3. The method according to claim 1, characterized in that The method further comprises: After determining the point cloud data of the environment vehicle according to the first posture information and the second posture information, identifying target points in the point cloud data that do not meet the observation condition; Filtering the point cloud data based on the target point to obtain a filtering result; When the noise value of the filtering result does not meet the preset threshold, the filtering result is filtered until the noise value meets the preset threshold, and the filtered point cloud data is output.
4. The method according to claim 1, characterized in that: The objective function includes: a first constraint function, the first constraint function is used to constrain the solution result of the second posture information based on the constraint condition; The constructing of an objective function based on the decision variables and the constraints includes: Based on the corresponding turning angle data and wheel speed pulse signals of the target vehicle between the adjacent image frames, the position and posture of the target vehicle in the adjacent image frames are estimated to obtain an estimation result; Based on the second pose information and the environment point cloud, construct a first reprojection constraint, wherein the first reprojection constraint is used to constrain the position difference between the projected point cloud obtained by projecting the second pose information onto the adjacent image frame and the environment point cloud to satisfy a constraint condition; Based on the second pose information, the calculated result and the preset unit matrix, constructing a first pose constraint, wherein the first pose constraint is used to indicate that the difference between the second pose information, the calculated result and the preset unit matrix satisfies the constraint condition; A first constraint function is constructed based on the weights corresponding to the first reprojection constraint and the first pose constraint, respectively.
5. The method according to claim 4, characterized in that The objective function includes: a second constraint function, the second constraint function is used to constrain the solution result of the first posture information based on the constraint condition; The constructing of the objective function based on the decision variables and the constraints further includes: Based on the first posture information, construct a second posture constraint, wherein the second posture constraint is used to constrain the turning angle corresponding to the first posture information between the adjacent image frames to meet a constraint condition; Determine the posture observation information of the environment vehicle according to the recognition result, and construct an observation constraint based on the inferred result, the posture observation information and a preset unit matrix, wherein the observation constraint is used to indicate that the difference between the inferred result and the observed posture and the preset unit matrix satisfies the constraint condition; A second constraint function is constructed based on the weights corresponding to the second posture constraint and the observation constraint respectively.
6. The method according to claim 5, characterized in that The constructing a second posture constraint based on the first posture information includes: Based on the first posture information, construct a first rotation angle constraint, wherein the first rotation angle constraint is used to constrain the difference of the rotation angles corresponding to the first posture information in the adjacent image frames to meet a constraint condition; Based on the first pose information, calculating the angular velocity corresponding to the environment vehicle in the adjacent image frames; Acquire a turning angular velocity threshold of the environment vehicle, and construct a second turning angle constraint based on the turning angular velocity threshold and the angular velocity, wherein the second turning angle constraint is used to constrain the angular velocity to satisfy a constraint condition; A second posture constraint is constructed based on the weights corresponding to the first rotation angle constraint and the second rotation angle constraint respectively.
7. The method according to claim 1, characterized in that The objective function includes: a second reprojection constraint, the second reprojection constraint being used to constrain a solution result of the point cloud data based on the constraint condition; The constructing of the objective function based on the decision variables and the constraints further includes: Acquire extrinsic parameters corresponding to the camera device, wherein the extrinsic parameters are used to indicate parameters of a rotation and translation relationship between a coordinate system of the camera device and a coordinate system of the target vehicle; Based on the external parameters, the point cloud data, the first pose information and the second pose information, a second reprojection constraint is constructed, wherein the second reprojection constraint is used to constrain that the difference between the projection coordinates obtained by projecting the point cloud data into an adjacent image frame and the original coordinates corresponding to the point cloud data in the adjacent image frame satisfies the constraint condition.
8. A dense point cloud estimation device, characterized in that: The device comprises: An acquisition module is used to acquire an environmental image frame sequence collected by a camera device of a target vehicle, and identify the environmental vehicle in the environmental image frame sequence to obtain an identification result; A first determination module is used to determine the first pose information of the environment vehicle based on the recognition result, wherein the first pose information is used to indicate the pose information corresponding to the environment vehicle in adjacent image frames of the environment image frame sequence; A second determination module is used to identify the second posture information corresponding to the target vehicle in the adjacent image frames, and determine the point cloud data of the environment vehicle according to the first posture information and the second posture information, wherein the determination of the point cloud data of the environment vehicle according to the first posture information and the second posture information includes: based on the external parameters corresponding to the camera device, estimating the three-dimensional coordinates of the two-dimensional point cloud of the environment vehicle at the time corresponding to the adjacent image frames, wherein the external parameters are used to indicate the parameters of the rotation and translation relationship between the coordinate system of the camera device and the coordinate system of the target vehicle; based on the first posture information, the second posture information and the three-dimensional coordinates, back-projecting the two-dimensional point cloud to obtain three-dimensional point cloud data; A construction module is used to construct an objective function based on the first pose information, the second pose information and the point cloud data to output a dense point cloud of the environment vehicle that satisfies the constraint conditions, wherein the construction of the objective function based on the first pose information, the second pose information and the point cloud data includes: obtaining the environment point cloud in the adjacent image frames; using the environment point cloud, the first pose information, the second pose information and the point cloud data as decision variables, and determining the constraint conditions corresponding to the decision variables; and constructing the objective function based on the decision variables and the constraint conditions.
9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the dense point cloud estimation method according to any one of claims 1 to 7 by executing the computer instructions.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the dense point cloud estimation method according to any one of claims 1 to 7.
11. A computer program product, characterized in that The method comprises computer instructions for causing a computer to execute the dense point cloud estimation method according to any one of claims 1 to 7.
