A method for optimizing dynamic target pose information

Abstract

The present invention provides a method for optimizing dynamic target pose information, including: generating a corresponding dynamic target 3D-box based on the position information of the dynamic target 2D-box, the mask image of a single frame of image, and the first detection line segment set; optimizing the position information of the dynamic target 2D-box in each frame of image by using the reprojection error according to the position information of the corresponding dynamic target 3D-box; and optimizing the pose of the dynamic target 3D-box by combining the pose of the dynamic target 3D-box generated from multiple frames of images and the vehicle body pose information to generate the final 3D-box pose information of the dynamic target. The present invention optimizes the reprojection error of the dynamic target 2D-box in each frame of image, improves the position accuracy of the 2D-box in the image coordinate system; and optimizes the pose information of the 3D-box based on continuous multiple frames of images and combines the pose information of the vehicle body at this time to improve the pose estimation accuracy of the dynamic target.

Description

Technical Field

[0001] The present invention relates to the field of high-precision crowdsourcing maps, and more specifically, to a method for optimizing dynamic target pose information. Background Art

[0002] Currently, in the field of high-precision map crowdsourcing, most of the focus is on the extraction of static traffic signs and road sign features to obtain traffic sign elements, with the emphasis on static targets. Currently, compared with traditional vision systems for obtaining target depth information such as binocular vision and RGB-D cameras, such devices may generate errors during installation, debugging, and operation. In contrast to monocular vision, it has a low cost, and its accuracy and stability can be stabilized through algorithm optimization. Summary of the Invention

[0003] In view of the technical problems existing in the prior art, the present invention provides a method for optimizing dynamic target pose information, which restores the depth and pose information of dynamic targets based on a monocular camera, and jointly optimizes them in combination with the vehicle pose to improve the pose accuracy of the vehicle body and dynamic targets.

[0004] According to a first aspect of the present invention, there is provided a method for optimizing dynamic target pose information, including:

[0005] For a single-frame image captured by a monocular camera, based on a deep learning model, output the position information of the dynamic target 2D-box in the single-frame image, and generate a mask image of the single-frame image;

[0006] For a single-frame image, based on a line detection algorithm, generate all detected line segments of the single-frame image to form a first detected line segment set;

[0007] Based on the position information of the dynamic target 2D-box, the mask image of the single-frame image, and the first detected line segment set, generate the corresponding dynamic target 3D-box;

[0008] According to the position information of the corresponding dynamic target 3D-box, use the reprojection error to optimize the position information of the dynamic target 2D-box in each frame of image;

[0009] Based on the optimized position information of the dynamic target 2D-box, generate the position information of the corresponding dynamic target 3D-box;

[0010] Combine the dynamic target 3D-box pose and vehicle pose information generated from multiple frames of images to optimize the dynamic target 3D-box pose and generate the final 3D-box pose information of the dynamic target.

[0011] Based on the above technical solutions, the present invention can also be improved as follows.

[0012] Optionally, for a single-frame image captured by a monocular camera, based on a deep learning model, the position information of the dynamic target 2D-box in the single-frame image is output, including:

[0013] Input the single-frame image into the deep learning model to obtain the position information of the dynamic target 2D-box recognized by the deep learning model in the image coordinate system. The position information is represented by a position vector [x, y, width, height], where [x, y] represents the upper-left corner coordinates of the dynamic target 2D-box, and [width, height] represents the width and height of the dynamic target 2D-box.

[0014] Optionally, for the single-frame image, based on a line detection algorithm, all the detected line segments of the single-frame image are generated to form a first detected line segment set, including:

[0015] For the single-frame image, based on the canny line detection algorithm, all the detected line segments of the single-frame image are generated. All the detected line segments in the single-frame image form a first detected line segment set. The vector representation of each detected line segment is: [x1, y1, x2, y2], where (x1, y1) are the starting coordinates of the detected line segment, and (x2, y2) are the ending coordinates of the detected line segment, and x1 < x2.

[0016] Optionally, based on the position information of the dynamic target 2D-box, the mask image of the single-frame image, and the first detected line segment set, the corresponding dynamic target 3D-box is generated, including:

[0017] Based on the mask image, filter the detected line segments with a gray value of 255 in the first detected line segment set L, and perform merging and deletion operations on the detected line segments with a gray value of 0 to obtain a second detected line segment set Lv;

[0018] Based on the pose angle information of the monocular camera, filter the detected line segments in the second detected line segment set Lv to obtain a third detected line segment set Lvp;

[0019] Based on the position information of the dynamic target 2D-box and the third detected line segment set Lvp, generate the position information of the dynamic target 3D-box corresponding to the dynamic target 2D-box.

[0020] Optionally, the operation of merging and deleting the detected line segments with a gray value of 0 to obtain the second detected line segment set Lv includes:

[0021] Calculate the angles of all the detected line segments with a gray value of 0 in the first detected line segment set. The angles are characterized by their slopes;

[0022] Calculate the angular difference and distance between every two adjacent detection line segments. The distance between every two adjacent detection line segments refers to the distance between the end point of one detection line segment and the start point of the other detection line segment.

[0023] When the angular difference is less than a preset angular threshold and the distance is less than a preset distance threshold, merge the two adjacent detection line segments.

[0024] Delete the detection line segments with a line segment length greater than a preset length threshold to obtain the second detection line segment set Lv.

[0025] Optionally, screening the detection line segments in the second detection line segment set Lv based on the pose angle information of the monocular camera to obtain the third detection line segment set Lvp, including:

[0026] Sample the pose angle information of the monocular camera.

[0027] Calculate the vanishing point based on the sampled pose angle information of the camera.

[0028] Search for the detection line segments forming the vanishing point from the second detection line segment set Lv based on the vanishing point.

[0029] Calculate the angle between the vanishing point and the midpoint of each detection line segment.

[0030] Screen the detection line segments based on the angle threshold to obtain the screened third detection line segment set Lvp.

[0031] Optionally, generating the position information of the dynamic target 3D-box corresponding to the dynamic target 2D-box based on the position information of the dynamic target 2D-box and the third detection line segment set Lvp, including:

[0032] Sample the upper edge of the target box corresponding to the position information of the dynamic target 2D-box to obtain the coordinate set {top_pos} of the top sampling points of the dynamic target 3D-box.

[0033] For each sampling point, calculate the 8 edge corner coordinates of the corresponding 3D-box to form the position information of the dynamic target 3D-box.

[0034] Optionally, optimizing the position information of the dynamic target 2D-box in each frame of image according to the position information of the corresponding dynamic target 3D-box using the reprojection error, including:

[0035] According to the pose transformation matrix Rt of the current monocular camera, reproject the position information of the dynamic target 3D-box and convert it to the 2D-box position information in the image coordinate system.

[0036] Optimize the original 2D-box position information of the dynamic target based on the 2D-box position information after reprojection and the original 2D-box position information extracted by the deep learning model.

[0037] Optionally, the optimizing the original 2D-box position information of the dynamic target based on the 2D-box position information after reprojection and the original 2D-box position information extracted by the deep learning model includes:

[0038] Establish an optimization residual model using the least squares method:

[0039] error = ||pos_cal - pos_g||2;

[0040] where pos_cal is the 2D-box position information of the dynamic target obtained after reprojection, and pos_g is the original 2D-box position information of the dynamic target;

[0041] Combine the association of the same dynamic target in multiple frames of images, and use a non-linear optimization algorithm to deduce the optimal 2D-box position information of the dynamic target in each frame.

[0042] Optionally, the optimizing the 3D-box pose of the dynamic target by combining the 3D-box pose and vehicle pose information of the dynamic target generated from multiple frames of images to generate the final 3D-box pose information of the dynamic target includes:

[0043] Based on the obtained 3D-box pose of the dynamic target, establish a motion model of the dynamic target to optimize the pose information of the 3D-box. Among them, the following model is established based on the differential speed model of the same dynamic target between consecutive frames:

[0044]

[0045] where, (t x , t y , θ) represents the displacement in the x direction, the displacement in the y direction, and the yaw angle of the vehicle head of the dynamic target in the relative coordinate system at the previous moment, (t x ’, t y ’) represents the displacement in the x direction, the displacement in the y direction, and the yaw angle of the vehicle head of the dynamic target in the relative coordinate system at the next moment, v represents the motion speed of the dynamic target, Δt represents the time period, Φ represents the steering angle of the vehicle, and L represents the distance between the centers of the front and rear wheels of the vehicle;

[0046] The residual equation between the motion models is:

[0047] e mo = [t′ x , t′y , θ'] - [t x , t y , θ];

[0048] Optimize the pose of the dynamic target 3D-box based on the established motion model and residual equation to generate the tracking trajectory of the dynamic target.

[0049] According to the second aspect of the present invention, there is provided a dynamic target pose information optimization system, including:

[0050] A first generation module for, for a single-frame image captured by a monocular camera, outputting the position information of the dynamic target 2D-box in the single-frame image based on a deep learning model and generating a mask image of the single-frame image;

[0051] A second generation module for, for a single-frame image, generating all detected line segments of the single-frame image based on a line detection algorithm to form a first detected line segment set;

[0052] A third generation module for generating a corresponding dynamic target 3D-box based on the position information of the dynamic target 2D-box, the mask image of the single-frame image, and the first detected line segment set;

[0053] A first optimization module for optimizing the position information of the dynamic target 2D-box in each frame of image by using the reprojection error according to the position information of the corresponding dynamic target 3D-box;

[0054] A fourth generation module for generating the position information of the corresponding dynamic target 3D-box based on the optimized position information of the dynamic target 2D-box;

[0055] A second optimization module for jointly optimizing the pose of the dynamic target 3D-box by combining the pose of the dynamic target 3D-box generated from multiple frames of images and the vehicle pose information to generate the final 3D-box pose information of the dynamic target.

[0056] According to the third aspect of the present invention, there is provided an electronic device including a memory and a processor, and the processor is used to implement the steps of the dynamic target pose information optimization method when executing a computer management program stored in the memory.

[0057] According to the fourth aspect of the present invention, there is provided a computer-readable storage medium on which a computer management program is stored, and the computer management program is used to implement the steps of the dynamic target pose information optimization method when executed by a processor.

[0058] A method for optimizing dynamic target pose information provided by the present invention obtains continuous multiple frames of images through a monocular camera. By analyzing the edge line detection of each frame of image, that is, based on the detection of the dynamic target box, multiple frames of images are combined to jointly estimate the 3D-box of the dynamic target. The 2D-box of the dynamic target is optimized, and the reprojection error in each frame of image is improved to increase the position accuracy of the 2D-box in the image coordinate system. Based on continuous multiple frames of images, the pose information of the vehicle body at this time is combined to perform local optimization of the dynamic target pose information, thereby improving the accuracy of dynamic target pose estimation. Description of the Drawings

[0059] Figure 1 It is a flowchart of a method for optimizing dynamic target pose information provided by the present invention;

[0060] Figure 2 It is a schematic diagram for generating a dynamic target 3D-box;

[0061] Figure 3 It is a schematic diagram of the optimization process of dynamic target pose information;

[0062] Figure 4 It is a schematic diagram of the structure of a dynamic target pose information optimization system provided by the present invention;

[0063] Figure 5 It is a schematic diagram of the hardware structure of a possible electronic device provided by the present invention;

[0064] Figure 6 It is a schematic diagram of the hardware structure of a possible computer-readable storage medium provided by the present invention. Detailed Embodiments

[0065] The following combines the drawings and embodiments to further describe in detail the specific embodiments of the present invention. The following embodiments are used to illustrate the present invention, but are not used to limit the scope of the present invention.

[0066] Embodiment 1

[0067] A method for optimizing dynamic target pose information, see Figure 1 , and this method mainly includes the following steps:

[0068] S1. For a single frame of image captured by a monocular camera, based on the deep learning model, output the position information of the dynamic target 2D-box in the single frame of image, and generate a mask image of the single frame of image.

[0069] As an example, for a single-frame image captured by a monocular camera, the position information of the dynamic target 2D-box in the single-frame image is output based on a deep learning model, including: inputting the single-frame image into the deep learning model to obtain the position information of the dynamic target 2D-box recognized by the deep learning model in the image coordinate system. The position information is represented by a position vector [x, y, width, height], where [x, y] represents the upper-left corner coordinates of the dynamic target 2D-box, and [width, height] represents the width and height of the dynamic target 2D-box.

[0070] Specifically, for a single-frame image captured by a monocular camera, a deep learning model is used to identify the dynamic target in the single-frame image, and the position and size information of the dynamic target 2D-box in the single-frame image (frame_i) is generated. Among them, the position and size of the 2D-box are represented by a vector as [x, y, width, height], where (x, y) represents the pixel coordinates in the image coordinate system (the origin is the upper-left corner of the image), width: the width of the image, and height: the height of the image.

[0071] For a single-frame image, a mask image of the dynamic target therein is generated.

[0072] S2. For a single-frame image, based on a line detection algorithm, all the detected line segments of the single-frame image are generated to form a first detected line segment set.

[0073] Among them, for a single-frame image, based on the canny line detection algorithm, all the detected line segments of the single-frame image are generated. All the detected line segments in the single-frame image form a first detected line segment set. The vector representation of each detected line segment is: [x1, y1, x2, y2], where (x1, y1) is the starting coordinate of the detected line segment, and (x2, y2) is the ending coordinate of the detected line segment, and x1 < x2.

[0074] S3. Based on the position information of the dynamic target 2D-box, the mask image of the single-frame image, and the first detected line segment set, the corresponding dynamic target 3D-box is generated.

[0075] As an embodiment, generating a corresponding dynamic target 3D-box based on the position information of the dynamic target 2D-box, the mask image of the single-frame image, and the first set of detected line segments includes: filtering the detected line segments with a gray value of 255 in the first set of detected line segments L based on the mask image, and performing merging and deletion operations on the detected line segments with a gray value of 0 to obtain a second set of detected line segments Lv; screening the detected line segments in the second set of detected line segments Lv based on the attitude angle information of the monocular camera to obtain a third set of detected line segments Lvp; and generating the position information of the dynamic target 3D-box corresponding to the dynamic target 2D-box based on the position information of the dynamic target 2D-box and the third set of detected line segments Lvp.

[0076] As an embodiment, performing the merging and deletion operations on the detected line segments with a gray value of 0 to obtain the second set of detected line segments Lv includes: calculating the angles of all the detected line segments with a gray value of 0 in the first set of detected line segments, where the angles are characterized by their slopes; calculating the angle difference and distance between each adjacent pair of detected line segments, where the distance between each adjacent pair of detected line segments refers to the distance between the end point of one detected line segment and the start point of the other detected line segment; merging the two adjacent detected line segments when the angle difference is less than a preset angle threshold and the distance is less than a preset distance threshold; and deleting the detected line segments with a length greater than a preset length threshold to obtain the second set of detected line segments Lv.

[0077] As an embodiment, screening the detected line segments in the second set of detected line segments Lv based on the attitude angle information of the monocular camera to obtain the third set of detected line segments Lvp includes: sampling the attitude angle information of the monocular camera; calculating the vanishing point based on the sampled attitude angle information of the camera; finding the detected line segments that form the vanishing point from the second set of detected line segments Lv based on the vanishing point; calculating the angle between the vanishing point and the midpoint of each detected line segment; and screening the detected line segments based on an angle threshold to obtain the screened third set of detected line segments Lvp.

[0078] As an embodiment, generating the position information of the dynamic target 3D-box corresponding to the dynamic target 2D-box based on the position information of the dynamic target 2D-box and the third set of detected line segments Lvp includes: sampling the upper edge of the target box corresponding to the position information of the dynamic target 2D-box to obtain a coordinate set {top_pos} of the top sampling points of the dynamic target 3D-box; and for each sampling point, calculating the 8 edge corner coordinates of the corresponding 3D-box to constitute the position information of the dynamic target 3D-box.

[0079] Specifically, by combining the position information of the dynamic target 2D-box, the mask image, and the set of detected line segments, the position information of the dynamic target 3D-box is generated. Among them, the input data is: 2D-box: [x, y, width, height], the mask image, and the first set of detected line segments {L}. The detailed process algorithm for generating the 3D-box from the above data is as follows:

[0080] Step-1: Using the image mask image, filter all the line segments in the first set of detected line segments {L} that are in the region with gray value value = 255, and at the same time perform the following operations on the line segments with gray value value = 0:

[0081] (1) Calculate the angles (slopes) of all the detected line segments with gray value value = 0;

[0082] (2) Calculate the angle differences between every two adjacent detected line segments;

[0083] (3) Calculate the distances between every two adjacent detected line segments (the distance between the end point and the start point of the line segment);

[0084] (4) When the angle difference between two adjacent detected line segments is less than the preset angle threshold (such as 5°) and the distance is less than the preset distance threshold, such as 20 pix (pixels), then merge the two adjacent detected line segments.

[0085] (5) Delete the detected line segments with a length greater than the threshold of 30 pix (pixels).

[0086] After the above steps, the final second set of detected line segments {Lv} is obtained.

[0087] Step-2: Sample the roll (roll angle) and pitch (pitch angle) of the monocular camera's pose;

[0088] (1) Sample at intervals of 3° with the camera's yaw angle of ±6° to obtain a sampling set.

[0089] (2) Calculate the vanishing point based on the current camera's yaw angle yaw, pitch angle pitch, and roll angle roll:

[0090] υp x =K·R -1 .(cos(yaw), sin(yaw), 0) T

[0091] υp y =K·R -1 ·(-sin(yaw), cos(yaw), 0) T

[0092] υp z= K·R -1 ·(0, 0, 1) T

[0093] where (vp x , vp y , vp z ) are the coordinates of the calculated vanishing point, K is the internal parameter of the monocular camera, R is the rotation matrix from the monocular camera to the world coordinate system, and yaw is the camera yaw angle.

[0094] (3) According to the vanishing point coordinates obtained above, find the detection line segments that form the vanishing point in {Lv}. For each vanishing point, there are multiple corresponding detection line segments, and calculate the angle between the vanishing point and the midpoint of each corresponding detection line segment. It is stipulated that the angle threshold between vanishing points 1 and 2 is 15°, and the angle threshold with vanishing point 3 is 10°. According to the above thresholds, retain the detection line segments that meet the conditions, and the third detection line segment set {Lvp} that forms the vanishing point can be obtained;

[0095] (4) Filter the third detection line segment set {Lvp} to detect multiple line segments that meet the conditions. Principle: For the multiple detection line segments corresponding to each vanishing point, calculate the angle between the vanishing point and the midpoint of each corresponding detection line segment, and select the detection line segments with the largest and smallest angles as the sides (support lines) that form the vanishing point.

[0096] Step-3: According to the upper edge of the 2D-box frame, with a sampling interval of 3pix, obtain the coordinate set {top_pos} of the top points of the 3D-box, and calculate the eight edge corner coordinates of all 3D-boxes based on each sampling coordinate point. See Figure 2 , the calculation process is as follows:

[0097] According to the above process, the following conditions are obtained:

[0098] The top corner1 is abbreviated as cor_1(x, y), and the coordinates of the three vanishing points are vp1, vp2, and vp3. Based on the coordinates of these 4 points and the above figure, the coordinates of the remaining 7 corners are deduced in two cases. The intersection lines of the rays from the vanishing points and the upper edge sampling points (i.e., cor_1) with the left and right boundaries of the 2D-box are divided into two cases: having an intersection with the left boundary and having an intersection with the right boundary.

[0099] (1) When vp1 intersects with the right boundary, the calculation of cor_2:

[0100] The intersection point of the ray Lvp1_cor_1 of vp1 and cor_1 ∩ BC is the coordinate of cor_2. The calculation of cor_3 and cor_4:

[0101] There are two scenarios: observing three faces (the left and middle in the above figure) and two faces (the right in the above figure). When observing three faces, the calculation of cor_4: The intersection point of the ray Lvp2_cor_1 between vp2 and cor_1 and AD is the coordinate of cor_4; the calculation of cor_3: The intersection point of the ray Lvp1_cor_4 between vp1 and cor_4, the vanishing point vp2 and the ray Lvp2_cor_2 between vp2 and cor_2, and the intersection point of the two rays Lvp1_cor_4 ∩ Lvp2_cor_2 is cor_3.

[0102] When observing two faces, the calculation of cor_3: The intersection point of the ray Lvp2_cor_2 between vp2 and cor_2 and AD = cor_3 is the coordinate of cor_3; the calculation of cor_4: The intersection point of the ray Lvp1_cor_3 between vp1 and cor_3 and the ray Lvp2_cor_1 between vp2 and cor_1, and the intersection point of Lvp1_cor_3 ∩ Lvp2_cor_1 = cor_4 is the coordinate of cor_4. The calculation of cor_5: (At this time, the calculation methods of the above two scenarios are the same) The intersection point of the ray Lvp3_cor_3 between vp3 and cor_3 and DC = cor_5 is the coordinate of cor_5; the calculation of cor_6: (At this time, the calculation methods of the above two scenarios are the same) The intersection point of the ray Lvp2_cor_5 between vp2 and cor_5 and the ray Lvp2_cor_2 between vp2 and cor_2, and Lvp2_cor_5 ∩ Lvp2_cor_2 = cor_6 is cor_6; the calculation of cor_7: (At this time, the calculation methods of the above two scenarios are the same) The intersection point of the ray Lvp1_cor_6 between vp1 and cor_6 and the ray Lvp3_cor_1 between vp3 and cor_1, and Lvp1_cor_6 ∩ Lvp3_cor_1 = cor_7 is cor_7; the calculation of cor_8: (At this time, the calculation methods of the above two scenarios are the same) The intersection point of the ray Lvp1_cor_5 between vp1 and cor_5 and the ray Lvp2_cor_7 between vp2 and cor_7, and Lvp1_cor_5 ∩ Lvp2_cor_7 = cor_8 is cor_8.

[0103] (2) When vp1 intersects with the right boundary: At this time, the coordinate calculation of the 7 cor points is the same as the above entire process, and will not be elaborated in detail. According to the above process, the coordinate information of the 8 cor points of the 3d-box is deduced, and then the position information of the dynamic target 3D-box in each frame of the image is obtained.

[0104] S4. According to the position information of the corresponding dynamic target 3D-box, use the reprojection error to optimize the position information of the dynamic target 2D-box in each frame of the image.

[0105] It is understandable that based on the 3D-box of the dynamic target obtained in each frame above, combined with the vehicle pose information of each frame, the reprojection error is calculated to optimize and improve the pose accuracy of the 2D-box, including the following steps:

[0106] (1) According to the pose transformation matrix Rt of the current monocular camera (the rotation and translation matrix from the camera image coordinate system to the world coordinate system), the coordinates of the current 3D-box are reprojected and transformed into the coordinates in the image coordinate system:

[0107] [u, v]min = min{π(R[±dx, ±dy, ±dz] / 2 + t))};

[0108] [u, v]max = max{π(R[±dx, ±dy, ±dz] / 2 + t))};

[0109] c = ([u, v]min + [u, v]max) / 2;

[0110] s = [u, v]max - [u, v]min;

[0111] Among them, [u, v]min is the upper left corner coordinate, [u, v]max is the lower right corner coordinate, c is the center point coordinate of the 2D-box position frame, and s is the length and width of the 2D-box position frame. The above information is represented as a vector as pos_cal.

[0112] (2) The observed value is the dynamic target 2D-box position information pos_g[x, y, width, height] obtained through the deep learning model.

[0113] According to the above information, an optimization residual model is established using the least squares method:

[0114] error = ||pos_cal - pos_g||2;

[0115] Combined with the associated observations of the same target in multiple frames, using the nonlinear optimization algorithm, the optimal pos[x, y, width, height] of the dynamic target in each frame is derived for subsequent pose optimization of the 3D-box.

[0116] S5. Based on the optimized position information of the dynamic target 2D-box, generate the corresponding position information of the dynamic target 3D-box.

[0117] It is understandable that the process of step S3 is repeated using the optimized dynamic target 2D-box position information of each frame in step S4 to recalculate the pose information of the dynamic target 3D-box.

[0118] S6. Combine the dynamic target 3D-box pose and vehicle pose information generated from multiple frames of images to optimize the dynamic target 3D-box pose and generate the final 3D-box pose information of the dynamic target.

[0119] Specifically, based on the pose information of the dynamic target 3D-box obtained in step S5, a motion model is established to optimize the 3D-box pose information. The following model is established using the differential model for the same vehicle between consecutive frames:

[0120]

[0121] where, where, (t x , t y , θ) represents the displacement in the x-direction, displacement in the y-direction, and yaw angle of the vehicle head of the dynamic target in the relative coordinate system at the previous moment, and (t x ’, t y ’, θ’) represents the displacement in the x-direction, displacement in the y-direction, and yaw angle of the vehicle head of the dynamic target in the relative coordinate system at the next moment, v represents the motion speed of the dynamic target, Δt represents the time period, Φ represents the steering angle of the vehicle, and L represents the distance between the centers of the front and rear wheels of the vehicle.

[0122] The residual equation between the motion models is:

[0123] e mo = [t′ x , t′ y , θ′] - [t x , t y , θ];

[0124] Based on the established motion model and residual equation, the dynamic target 3D-box pose is optimized to generate the tracking trajectory of the dynamic target, and finally the trajectory points of the dynamic target are output.

[0125] Embodiment 2

[0126] A method for optimizing the pose information of a dynamic target. Refer to Figure 3 , and input each single-frame image captured by a monocular camera into a deep learning model to obtain the position information of the dynamic target 2D-box and generate a mask image of the dynamic target. On the other hand, line detection is performed on the single-frame image to generate all detected line segments in the image, forming the first detected line segment set L.

[0127] Generate the position information of the dynamic object 3D-box in the world coordinate system based on the position information of the dynamic object 2D-box, the mask image of the dynamic object, and the first set of detection line segments L. Based on the pose angle information of the monocular camera, reproject the position information of the dynamic object 3D-box in the world coordinate system to the image coordinate system to optimize the position information of the 2D-box.

[0128] Generate the corresponding position information of the dynamic object 3D-box based on the optimized 2D-box position information, the mask image of the dynamic object, and the set of detection line segments. According to the vehicle body's own pose information, use an optimization algorithm to optimize the position information of the dynamic object 3D-box and output the tracking trajectory of the dynamic object. For example, output the tracking trajectory of the dynamic vehicle.

[0129] Embodiment III

[0130] A dynamic object pose information optimization system, see Figure 4 , this system includes a first generation module 401, a second generation module 402, a third generation module 403, a first optimization module 404, a fourth generation module 405, and a second optimization module 406.

[0131] Among them, the first generation module 401 is used to, for a single-frame image captured by a monocular camera, output the position information of the dynamic object 2D-box in the single-frame image based on a deep learning model and generate a mask image of the single-frame image; the second generation module 402 is used to, for a single-frame image, generate all detection line segments of the single-frame image based on a line detection algorithm to form a first set of detection line segments; the third generation module 403 is used to generate the corresponding dynamic object 3D-box based on the position information of the dynamic object 2D-box, the mask image of the single-frame image, and the first set of detection line segments; the first optimization module 404 is used to, according to the position information of the corresponding dynamic object 3D-box, use the reprojection error to optimize the position information of the dynamic object 2D-box in each frame of image; the fourth generation module 405 is used to generate the corresponding position information of the dynamic object 3D-box based on the optimized position information of the dynamic object 2D-box; the second optimization module 406 is used to jointly optimize the pose of the dynamic object 3D-box by combining the pose of the dynamic object 3D-box generated from multiple frames of images and the vehicle body pose information to generate the final 3D-box pose information of the dynamic object.

[0132] It can be understood that a dynamic object pose information optimization system provided by the present invention corresponds to the dynamic object pose information optimization methods provided in the foregoing embodiments. The relevant technical features of the dynamic object pose information optimization system can refer to the relevant technical features of the dynamic object pose information optimization method, which will not be elaborated here.

[0133] Embodiment IV

[0134] Please refer to Figure 5 , Figure 5 , which is a schematic diagram of an embodiment of an electronic device provided by an embodiment of the present invention. As Figure 5 shown, an embodiment of the present invention provides an electronic device 500, including a memory 510, a processor 520, and a computer program 511 stored on the memory 510 and executable on the processor 520. When the processor 520 executes the computer program 511, the following steps are implemented: for a single-frame image captured by a monocular camera, based on a deep learning model, output the position information of the dynamic vehicle 2D-box in the single-frame image, and generate a mask image of the single-frame image; for a single-frame image, based on a line detection algorithm, generate all detection lines of the single-frame image to form a first detection line set; based on the position information of the dynamic vehicle 2D-box, the mask image of the single-frame image, and the first detection line set, generate a corresponding dynamic vehicle 3D-box; according to the position information of the corresponding dynamic vehicle 3D-box, use the reprojection error to optimize the position information of the dynamic vehicle 2D-box of each frame of image; based on the optimized position information of the dynamic vehicle 2D-box, generate the position information of the corresponding dynamic vehicle 3D-box; combine the dynamic vehicle 3D-box poses and vehicle body pose information generated by multiple frames of images, optimize the dynamic vehicle 3D-box poses, and generate the final 3D-box pose information of the dynamic vehicle.

[0135] Embodiment 5

[0136] Please refer to Figure 6 , Figure 6 , which is a schematic diagram of an embodiment of a computer-readable storage medium provided by the present invention. As Figure 6 shown, this embodiment provides a computer-readable storage medium 600, on which a computer program 611 is stored. When the computer program 611 is executed by a processor, the following steps are implemented: for a single-frame image captured by a monocular camera, based on a deep learning model, output the position information of the dynamic vehicle 2D-box in the single-frame image, and generate a mask image of the single-frame image; for a single-frame image, based on a line detection algorithm, generate all detection lines of the single-frame image to form a first detection line set; based on the position information of the dynamic vehicle 2D-box, the mask image of the single-frame image, and the first detection line set, generate a corresponding dynamic vehicle 3D-box; according to the position information of the corresponding dynamic vehicle 3D-box, use the reprojection error to optimize the position information of the dynamic vehicle 2D-box of each frame of image; based on the optimized position information of the dynamic vehicle 2D-box, generate the position information of the corresponding dynamic vehicle 3D-box; combine the dynamic vehicle 3D-box poses and vehicle body pose information generated by multiple frames of images, optimize the dynamic vehicle 3D-box poses, and generate the final 3D-box pose information of the dynamic vehicle.

[0137] A method for optimizing dynamic target pose information provided by an embodiment of the present invention obtains continuous multi-frame images through a monocular camera. By analyzing the edge line detection of each frame of image, that is, based on the detection of the dynamic target box, multiple frames of images are combined to jointly estimate the 3D-box of the dynamic target. Optimize the reprojection error of the dynamic target 2d-box in each frame of image to improve the position accuracy of the 2d-box in the image coordinate system. Based on continuous multi-frame images, combined with the pose information of the vehicle body at this time, local optimization of the dynamic target pose information is performed to improve the accuracy of dynamic target pose estimation.

[0138] It should be noted that in the above embodiments, the descriptions of each embodiment have their own emphases. For the parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0139] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0140] The present invention is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram, as well as the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for realizing the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0141] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device, and the instruction device realizes the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0142] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus, so that a series of operation steps are performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions for implementing the process Figure 1 one process or more processes and / or blocks Figure 1 steps for implementing the functions specified in one block or more blocks.

[0143] Although the preferred embodiments of the present invention have been described, additional changes and modifications can be made by those skilled in the art once they learn the basic inventive concept. Therefore, the appended claims are intended to be construed to cover the preferred embodiments as well as all changes and modifications falling within the scope of the present invention.

[0144] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalent technologies, the present invention is also intended to include these modifications and variations.

Claims

1. A method for optimizing dynamic target pose information, characterized in that Including: For a single-frame image captured by a monocular camera, based on a deep learning model, output the position information of the dynamic target 2D-box in the single-frame image, and generate a mask image of the single-frame image; For a single-frame image, based on a line detection algorithm, generate all the detected line segments of the single-frame image to form a first detected line segment set; Based on the position information of the dynamic target 2D-box, the mask image of the single-frame image, and the first detected line segment set, generate the corresponding dynamic target 3D-box; According to the position information of the corresponding dynamic target 3D-box, use the reprojection error to optimize the position information of the dynamic target 2D-box in each frame of the image; Based on the optimized position information of the dynamic target 2D-box, generate the position information of the corresponding dynamic target 3D-box; Combine the pose information of the dynamic target 3D-box and the vehicle pose information generated from multiple frames of images to optimize the pose of the dynamic target 3D-box and generate the final 3D-box pose information of the dynamic target; The generating the corresponding dynamic target 3D-box based on the position information of the dynamic target 2D-box, the mask image of the single-frame image, and the first detected line segment set includes: Based on the mask image, filter the detected line segments with a gray value of 255 in the first detected line segment set L, and perform merging and deletion operations on the detected line segments with a gray value of 0 to obtain a second detected line segment set Lv; Based on the attitude angle information of the monocular camera, screen the detected line segments in the second detected line segment set Lv to obtain a third detected line segment set Lvp; Based on the position information of the dynamic target 2D-box and the third detected line segment set Lvp, generate the position information of the dynamic target 3D-box corresponding to the dynamic target 2D-box; The screening the detected line segments in the second detected line segment set Lv based on the attitude angle information of the monocular camera to obtain a third detected line segment set Lvp includes: Sample the attitude angle information of the monocular camera; Based on the sampled attitude angle information of the camera, calculate the vanishing point; Based on the vanishing point, find the detected line segments forming the vanishing point from the second detected line segment set Lv; Calculate the angle between the vanishing point and the midpoint of each detected line segment; Based on the angle threshold, screen the detected line segments to obtain the screened third detected line segment set Lvp.

2. The dynamic target pose information optimization method according to claim 1, wherein The outputting the position information of the dynamic target 2D-box in the single-frame image based on the deep learning model for a single-frame image captured by a monocular camera includes: Input the single-frame image into the deep learning model to obtain the position information of the dynamic target 2D-box recognized by the deep learning model in the image coordinate system. The position information is represented by a position vector [x, y, width, height], where [x, y] represents the upper left corner coordinates of the dynamic target 2D-box, and [width, height] represents the width and height of the dynamic target 2D-box.

3. The dynamic target pose information optimization method according to claim 1, characterized in that The generating all the detected line segments of the single-frame image based on a line detection algorithm to form a first detected line segment set for a single-frame image includes: For a single-frame image, based on the Canny edge detection algorithm, all detected line segments of the single-frame image are generated. All the detected line segments in the single-frame image form a first set of detected line segments. The vector representation of each detected line segment is: [x1, y1, x2, y2], where (x1, y1) is the starting coordinate of the detected line segment, and (x2, y2) is the ending coordinate of the detected line segment, and x1 < x2.

4. The dynamic target pose information optimization method according to claim 1, characterized in that Performing merging and deletion operations on the detected line segments with a gray value of 0 to obtain a second set of detected line segments Lv, including: Calculating the angles of all detected line segments with a gray value of 0 in the first set of detected line segments, where the angles are characterized by their slopes; Calculating the angle difference and distance between each adjacent pair of detected line segments. The distance between each adjacent pair of detected line segments refers to the distance between the end point of one detected line segment and the starting point of the other detected line segment; When the angle difference is less than a preset angle threshold and the distance is less than a preset distance threshold, merging the two adjacent detected line segments; Deleting the detected line segments with a length greater than a preset length threshold to obtain the second set of detected line segments Lv.

5. The dynamic target pose information optimization method according to claim 1, characterized in that Generating the position information of the dynamic target 3D-box corresponding to the dynamic target 2D-box based on the position information of the dynamic target 2D-box and the third set of detected line segments Lvp, including: Sampling the upper edge of the target box corresponding to the position information of the dynamic target 2D-box to obtain a coordinate set {top_pos} of the top sampling points of the dynamic target 3D-box; For each sampling point, calculating the 8 edge corner coordinates of the corresponding 3D-box to form the position information of the dynamic target 3D-box.

6. The dynamic target pose information optimization method according to claim 1, characterized in that Optimizing the position information of the dynamic target 2D-box in each frame of image according to the position information of the corresponding dynamic target 3D-box using the reprojection error, including: According to the pose transformation matrix Rt of the current monocular camera, reprojecting the position information of the dynamic target 3D-box to generate the 2D-box position information in the image coordinate system; Based on the reprojected 2D-box position information and the original 2D-box position information extracted by the deep learning model, optimizing the original position information of the dynamic target 2D-box.

7. The dynamic target pose information optimization method according to claim 6, characterized in that Optimizing the original position information of the dynamic target 2D-box based on the reprojected 2D-box position information and the original 2D-box position information extracted by the deep learning model, including: Using the least squares method to establish an optimization residual model: ； Among them, is the 2D-box position information of the dynamic target obtained after reprojection, is the original 2D-box position information of the dynamic target; Combining the association of the same dynamic target in multiple frames of images and using a non-linear optimization algorithm to deduce the optimal 2D-box position information of the dynamic target in each frame.

8. The dynamic target pose information optimization method according to claim 1, characterized in that Optimizing the pose of the dynamic target 3D-box by combining the pose information of the dynamic target 3D-box and the vehicle pose information generated from multiple frames of images to generate the final 3D-box pose information of the dynamic target, including: Based on the obtained 3D-box pose of the dynamic target, a motion model of the dynamic target is established to optimize the pose information of the 3D-box. Among them, the following model is established based on the differential speed model of the same dynamic target between consecutive frames: ； Among them, (t x , t y , θ) represents the displacement in the x-direction, the displacement in the y-direction, and the yaw angle of the vehicle head of the dynamic target at the previous moment in the relative coordinate system, (t x ’, t y ’) represents the displacement in the x-direction, the displacement in the y-direction, and the yaw angle of the vehicle head of the dynamic target at the next moment in the relative coordinate system, v represents the moving speed of the dynamic target, represents the time period, Φ represents the steering angle of the vehicle, and L represents the distance between the centers of the front and rear wheels of the vehicle; The residual equation between motion models is: ； Based on the established motion model and residual equation, the 3D-box pose of the dynamic target is optimized to generate the tracking trajectory of the dynamic target.

Images