A method, system, device, and medium for annotating vehicle key points

Abstract

The application provides a method, system, device and medium for labeling vehicle key points, comprising: determining a vehicle model of a vehicle to be labeled, obtaining a corresponding three-dimensional labeling model according to the determined vehicle model, and performing three-dimensional key point labeling on the vehicle to be labeled; performing two-dimensional projection on the three-dimensional key point labeling result to obtain a two-dimensional projection result of all key points; and performing two-dimensional key point labeling on the vehicle to be labeled, and determining a final key point labeling result of the vehicle to be labeled based on the two-dimensional key point labeling result and the two-dimensional projection result. By correlating the two-dimensional projection result and the two-dimensional key point labeling result as the key point labeling result of the vehicle to be labeled, the application can use the labeling result of the 2D labeling model and the two-dimensional projection result after the projection of the 3D labeling model as the key point labeling result of the vehicle to be labeled, thereby solving the problem of labeling error when using the 2D labeling model or the 3D labeling model alone.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to a method, system, device, and medium for annotating key points of a vehicle. Background Technology

[0002] Vehicle detection and recognition is one of the main tasks in automatic video image recognition systems, with widespread demand in applications such as security, vehicle Re-identification (ReID), and video structuring. Vehicle annotation is a crucial part of the recognition system; the accuracy of the annotation determines the accuracy of the algorithm, and the dimensionality and density of the annotated information greatly influence the deep neural network's understanding of target features.

[0003] Common vehicle annotation methods include 2D (2-dimensional) keypoint annotation and 3D (3-dimensional) annotation based on CAD (Computer-Aided Design) models. While 2D keypoint annotation can accurately capture the location information of key corner points on a vehicle, its high difficulty, cost, and limited number of keypoints often result in insufficient information density. 3D annotation based on CAD models, on the other hand, allows for the acquisition of 3D vehicle information by dragging a quasi-3D annotation model and projecting it onto the image to align with the vehicle target. Simultaneously, 2D keypoints can be obtained from the projection of the 3D annotation model. However, this CAD model-based 3D annotation method requires preparing a CAD model beforehand based on actual data, and then manually selecting a model to fit the vehicle model during the annotation process. Since actual data often includes hundreds of vehicle models, precise model selection is impossible, leading to potential errors when using this method for vehicle annotation. Summary of the Invention

[0004] In view of the shortcomings of the prior art described above, the purpose of this invention is to provide a method, system, device and medium for annotating key points of vehicles, so as to solve the problems of high difficulty, high cost and easy error in annotation when annotating vehicles in the prior art.

[0005] To achieve the above and other related objectives, the present invention provides a method for marking key points of a vehicle, comprising the following steps:

[0006] Determine the vehicle model to be labeled;

[0007] Obtain the corresponding 3D annotation model based on the determined vehicle model, and use the obtained 3D annotation model to annotate the 3D key points of the vehicle to be annotated;

[0008] Perform a two-dimensional projection on the three-dimensional key point annotation results to obtain the two-dimensional projection results of all key points in the vehicle to be annotated;

[0009] Two-dimensional key point annotation is performed on the vehicle to be annotated, and the final key point annotation result of the vehicle to be annotated is determined based on the two-dimensional key point annotation result and the two-dimensional projection result.

[0010] Optionally, after annotating the vehicle to be annotated with two-dimensional key points, the method further includes:

[0011] Calculate the error value between the two-dimensional key point annotation result and the two-dimensional projection result;

[0012] If the error value is greater than a preset threshold, the three-dimensional annotation model is adjusted at least once so that the error value between the new two-dimensional projection result obtained by using the adjusted three-dimensional annotation model and the two-dimensional key point annotation result is less than or equal to the preset threshold.

[0013] If the error value is less than or equal to a preset threshold, the two-dimensional projection result and the two-dimensional key point annotation result at the current moment are associated as the final key point annotation result of the vehicle to be annotated, and the projection relationship of the three-dimensional key point annotation result at the current moment is obtained as the two-dimensional projection result.

[0014] Optionally, when the error value exceeds a preset threshold, the adjustment process for the 3D annotation model includes:

[0015] Obtain the projection equation between two-dimensional key points and three-dimensional key points, and obtain the projection relationship when the three-dimensional key point annotation result is projected into a two-dimensional projection result based on the projection equation;

[0016] The two-dimensional key point annotation results of each key point in the vehicle to be annotated are matched with the three-dimensional key point annotation results obtained from the initial three-dimensional annotation model or the three-dimensional annotation model after the last adjustment to obtain multiple key point pairs.

[0017] Based on the projection equation and the multiple key point pairs, the key points with annotation errors are determined, and the least squares method is used to determine the correction value for updating the position of the key points with annotation errors.

[0018] The 3D annotation model corresponding to the error value exceeding the preset threshold is adjusted according to the projection relationship and the correction value to obtain the adjusted 3D annotation model.

[0019] Optionally, the key features of the vehicle to be labeled include at least one of the following:

[0020] The top left corner of the front roof, the outer left corner of the front headlight, the outer left corner of the bottom left of the front chassis, the center left of the front wheel, the center left of the rear wheel, the left corner of the rear chassis, the outer left corner of the taillight, the top left corner of the rear roof, the front end of the bottom edge of the left window, the end of the bottom edge of the left window, the top right corner of the front roof, the outer right corner of the front headlight, the outer right corner of the bottom right of the front chassis, the center right of the front wheel, the center right of the rear wheel, the right corner of the rear chassis, the outer right corner of the taillight, the top right corner of the rear roof, the front end of the bottom edge of the right window, and the end of the bottom edge of the right window.

[0021] Optionally, when annotating the vehicle to be annotated with two-dimensional key points or three-dimensional key points, the annotation category includes at least one of the following: visible, self-occluded, other occluded, truncated, and unknown.

[0022] Optionally, the vehicle type includes at least one of the following: sedan, minivan, truck, pickup truck.

[0023] The present invention also provides a system for marking key points of a vehicle, comprising:

[0024] The vehicle model module is used to determine the vehicle model to be labeled;

[0025] The first annotation module is used to obtain the corresponding three-dimensional annotation model according to the determined vehicle model, and to use the obtained three-dimensional annotation model to annotate the three-dimensional key points of the vehicle to be annotated.

[0026] The projection module is used to perform two-dimensional projection on the three-dimensional key point annotation results to obtain the two-dimensional projection results of all key points in the vehicle to be annotated;

[0027] The second annotation module is used to annotate the vehicle to be annotated with two-dimensional key points, and to determine the final key point annotation result of the vehicle to be annotated based on the two-dimensional key point annotation result and the two-dimensional projection result.

[0028] Optionally, after the second annotation module annotates the two-dimensional key points of the vehicle to be annotated, it further includes:

[0029] Calculate the error value between the two-dimensional key point annotation result and the two-dimensional projection result;

[0030] If the error value is greater than a preset threshold, the three-dimensional annotation model is adjusted at least once so that the error value between the new two-dimensional projection result obtained by using the adjusted three-dimensional annotation model and the two-dimensional key point annotation result is less than or equal to the preset threshold.

[0031] If the error value is less than or equal to a preset threshold, the two-dimensional projection result and the two-dimensional key point annotation result at the current moment are associated as the final key point annotation result of the vehicle to be annotated, and the projection relationship of the three-dimensional key point annotation result at the current moment is obtained as the two-dimensional projection result.

[0032] Optionally, when the error value exceeds a preset threshold, the adjustment process for the 3D annotation model includes:

[0033] Obtain the projection equation between two-dimensional key points and three-dimensional key points, and obtain the projection relationship when the three-dimensional key point annotation result is projected into a two-dimensional projection result based on the projection equation;

[0034] The two-dimensional key point annotation results of each key point in the vehicle to be annotated are matched with the three-dimensional key point annotation results obtained from the initial three-dimensional annotation model or the three-dimensional annotation model after the last adjustment to obtain multiple key point pairs.

[0035] Based on the projection equation and the multiple key point pairs, the key points with annotation errors are determined, and the least squares method is used to determine the correction value for updating the position of the key points with annotation errors.

[0036] The 3D annotation model corresponding to the error value exceeding the preset threshold is adjusted according to the projection relationship and the correction value to obtain the adjusted 3D annotation model.

[0037] Optionally, the key features of the vehicle to be labeled include at least one of the following:

[0038] The top left corner of the front roof, the outer left corner of the top left of the front headlight, the outer left corner of the bottom left of the front chassis, the center left of the front wheel, the center left of the rear wheel, the left corner of the rear chassis, the outer left corner of the top left of the taillight, the top left corner of the rear roof, the front end of the bottom edge of the left window, the end of the bottom edge of the left window, the top right corner of the front roof, the outer right corner of the top right of the front headlight, the outer right corner of the bottom right of the front chassis, the center right of the front wheel, the center right of the rear wheel, the right corner of the rear chassis, the outer right corner of the top right of the taillight, the top right corner of the rear roof, the front end of the bottom edge of the right window, and the end of the bottom edge of the right window.

[0039] Optionally, when annotating the vehicle to be annotated with two-dimensional key points or three-dimensional key points, the annotation category includes at least one of the following: visible, self-occluded, other occluded, truncated, and unknown.

[0040] The present invention also provides a device for marking key points of a vehicle, comprising:

[0041] One or more processors; and

[0042] A computer-readable medium storing instructions that, when executed by the one or more processors, cause the device to perform the method as described in any of the foregoing descriptions.

[0043] The present invention also provides a computer-readable medium, characterized in that it stores instructions thereon, which, when executed by one or more processors, cause a device to perform the method as described in any of the foregoing.

[0044] As described above, the present invention provides a method, system, device, and medium for marking key points on a vehicle, which has the following beneficial effects:

[0045] This invention first determines the vehicle model to be annotated, then obtains the corresponding 3D annotation model based on the determined model, and uses the obtained 3D annotation model to annotate the key points of the vehicle to be annotated. Next, it performs 2D projection on the 3D key point annotation results to obtain the 2D projection results of all key points in the vehicle to be annotated; and then performs 2D key point annotation on the vehicle to be annotated, and associates the 2D projection results and the 2D key point annotation results as the key point annotation results of the vehicle to be annotated. This invention addresses the shortcomings of current CAD model-based annotation tools by designing a vehicle 3D annotation method based on a deformable key point model. By associating the 2D projection results and the 2D key point annotation results as the key point annotation results of the vehicle to be annotated, this invention can use both the annotation results of the 2D annotation model and the 2D projection results of the 3D annotation model as the key point annotation results of the vehicle to be annotated, thereby solving the annotation error problem that exists when using only the 2D or 3D annotation model. Furthermore, this invention can guide the deformation of the 3D annotation model by using the key point positions in the 2D key point annotation results, solving the problem of incompatibility of existing 3D annotation models. The annotation process eliminates the need to manually select vehicle models from hundreds of 3D annotation models, reducing the annotation difficulty. Simultaneously, by annotating the 3D information of the vehicle, the information dimensionality and density can be increased. Moreover, by adjusting and deforming the 3D annotation model, this invention not only improves annotation accuracy but also eliminates the need to select specific vehicle models, thus reducing the difficulty of annotating key points of the vehicle to be annotated. Furthermore, relying on the 2D projection results of the 3D annotation model and the 2D annotation results directly from the 2D annotation model, this invention can output the positions and categories of all 2D key points corresponding to the vehicle model to be annotated. Attached Figure Description

[0046] Figure 1 A schematic flowchart of a method for annotating key points of a vehicle provided in one embodiment;

[0047] Figures 2a to 2e A schematic diagram of key points of a car model provided in one embodiment;

[0048] Figure 3 A schematic diagram of a 3D key point model of a truck provided in one embodiment;

[0049] Figure 4 A flowchart illustrating a method for annotating key points of a vehicle, as provided in another embodiment;

[0050] Figure 5 A schematic diagram of the hardware structure of a system for marking key points on a vehicle, provided in one embodiment;

[0051] Figure 6 A schematic diagram of the hardware structure of a terminal device provided in one embodiment;

[0052] Figure 7 A schematic diagram of the hardware structure of a terminal device provided in another embodiment.

[0053] Component designation explanation

[0054] M10 vehicle module

[0055] M20 First Annotation Module

[0056] M30 projection module

[0057] M40 Second Annotation Module

[0058] 1100 Input Device

[0059] 1101 First Processor

[0060] 1102 Output device

[0061] 1103 First Memory

[0062] 1104 Communication Bus

[0063] 1200 processing components

[0064] 1201 Second Processor

[0065] 1202 Second Memory

[0066] 1203 Communication Component

[0067] 1204 Power Supply Unit

[0068] 1205 Multimedia Components

[0069] 1206 Audio Component

[0070] 1207 Input / Output Interface

[0071] 1208 Sensor Assembly Detailed Implementation

[0072] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0073] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0074] KDM: Keypoint Deformable Model.

[0075] Please see Figure 1 As shown, the present invention provides a method for marking key points of a vehicle, comprising the following steps:

[0076] S100, Determine the vehicle type to be labeled; for example, the labeler can select the vehicle type to be labeled, and then determine the vehicle type. In this method, the vehicle types available for labeler to select include, but are not limited to: sedans, minivans, vans, trucks, pickup trucks, etc.

[0077] S200, obtain the corresponding 3D annotation model according to the determined vehicle model, and use the obtained 3D annotation model to annotate the vehicle to be annotated with 3D key points; as an example, the 3D annotation model obtained according to the determined vehicle model includes a 3D annotation model that matches the vehicle model in the current image and a 3D annotation model that does not match the vehicle model in the current image.

[0078] S300 performs two-dimensional projection on the three-dimensional key point annotation results to obtain the two-dimensional projection results of all key points in the vehicle to be annotated.

[0079] S400, perform two-dimensional key point annotation on the vehicle to be annotated, and determine the final key point annotation result of the vehicle to be annotated based on the two-dimensional key point annotation result and the two-dimensional projection result. As an example, in this embodiment, when performing two-dimensional or three-dimensional key point annotation on the vehicle to be annotated, the annotation category includes at least one of the following: visible, self-occluded, other occluded, truncated, and unknown.

[0080] Therefore, this method addresses the shortcomings of current CAD model-based annotation tools by designing a vehicle 3D annotation method based on a deformable keypoint model. By linking the 2D projection results with the 2D keypoint annotation results, this method serves as the keypoint annotation result for the vehicle to be annotated. Essentially, this method can use both the annotation results of the 2D annotation model and the 2D projection results of the 3D annotation model as the keypoint annotation result for the vehicle to be annotated, thus solving the annotation error problem that exists when using only the 2D or 3D annotation model. Furthermore, relying on the 2D projection results of the 3D annotation model and the 2D annotation results directly from the 2D annotation model, this method can output the location and category of all 2D keypoints corresponding to the vehicle model to be annotated.

[0081] According to the above description, in this method, the key points of the vehicle to be annotated include at least one of the following: the upper left corner of the front roof, the outer left corner of the upper left edge of the front headlight, the outer left corner of the lower left edge of the front chassis, the center left of the front wheel, the center left of the rear wheel, the left corner of the rear chassis, the outer left corner of the upper left edge of the taillight, the upper left corner of the rear roof, the front end of the bottom edge of the left window, the end of the bottom edge of the left window, the upper right corner of the front roof, the outer right corner of the upper right edge of the front headlight, the outer right corner of the lower right edge of the front chassis, the center right of the front wheel, the center right of the rear wheel, the center right of the rear chassis, the outer right corner of the upper right edge of the taillight, the upper right corner of the rear roof, the front end of the bottom edge of the right window, and the end of the bottom edge of the right window. As an example, the key points of a certain sedan are as follows: Figures 2a to 2e As shown, 0 represents the top left corner of the front roof, 1 represents the outer corner of the top left edge of the front headlight, 2 represents the outer corner of the bottom left edge of the front chassis, 3 represents the center left edge of the front wheel, 4 represents the center left edge of the rear wheel, 5 represents the left corner of the rear chassis, 6 represents the outer corner of the top left edge of the taillight, 7 represents the top left corner of the rear roof, 8 represents the front edge of the bottom edge of the left window, 9 represents the end of the bottom edge of the left window, 10 represents the top right corner of the front roof, 11 represents the outer right corner of the top right edge of the front headlight, 12 represents the outer right corner of the bottom right edge of the front chassis, 13 represents the center right edge of the front wheel, 14 represents the center right edge of the rear wheel, 15 represents the right corner of the rear chassis, 16 represents the outer right corner of the top right edge of the taillight, 17 represents the top right corner of the rear roof, 18 represents the front edge of the bottom edge of the right window, and 19 represents the end of the bottom edge of the right window.

[0082] In an exemplary embodiment, after annotating the vehicle to be annotated with two-dimensional key points, the method further includes: calculating the error value between the two-dimensional key point annotation result and the two-dimensional projection result. If the error value is greater than a preset threshold, the three-dimensional annotation model is adjusted, and the three-dimensional key points of the vehicle to be annotated are re-annotated using the adjusted three-dimensional annotation model, and the error value between the new two-dimensional projection result and the two-dimensional key point annotation result is calculated; if the error value obtained at this time is still greater than the preset threshold, the three-dimensional annotation model is iteratively adjusted until the error value between the new two-dimensional projection result obtained using the adjusted three-dimensional annotation model and the two-dimensional key point annotation result is less than or equal to the preset threshold, at which point the iterative adjustment of the three-dimensional annotation model is stopped. If the error value is less than or equal to the preset threshold, the two-dimensional projection result and the two-dimensional key point annotation result at the current moment are associated as the final key point annotation result of the vehicle to be annotated, and the projection relationship between the three-dimensional key point annotation result at the current moment and the two-dimensional projection result is obtained. As an example, in this embodiment, when calculating the error value between the two-dimensional key point annotation result and the two-dimensional projection result, the method further includes establishing projection equations for vehicle key points in the two-dimensional plane and three-dimensional space. Vehicle key points in the two-dimensional plane are denoted as two-dimensional key points or 2D key points, and vehicle key points in the three-dimensional space are denoted as three-dimensional key points or 3D key points. The projection equations are:

[0083]

[0084]

[0085] In the formula, For the exterior orientation elements of the camera, where X is the angular element of the camera, representing the camera's orientation in the real world; S Y S Z S The line element represents the camera's spatial position in the real world. (X) A Y A Z A (x, y) represents the coordinates of the vehicle's key points in the real world, and (x, y) is the coordinate of the 3D key points projected onto the image.

[0086] Then, solvePnP is used to calculate the exterior orientation elements of the camera, obtaining the projection relationship from the 3D keypoint annotation results to the 2D projection results; that is, solvePnP is used to calculate... These are the six parameters. Specifically, when calculating the camera's exterior orientation elements, first, all visible 2D keypoints in the image are found. Then, these are compared with the 3D keypoints annotated by a pre-prepared 3D annotation model (i.e., an unadjusted 3D annotation model or an initial 3D annotation model) to form n 2D-3D keypoint pairs. Each keypoint pair can be used to formulate two equations: one about the x-axis and one about the y-axis. When n≥3, solvePnP is used to calculate... These 6 parameters.

[0087] Calculated using solvePnP These six parameters, after determining the projection relationship from the 3D keypoint annotation results to the 2D projection results, also include calculating the error value between the 2D keypoint annotation results and the 2D projection results:

[0088]

[0089]

[0090] In the formula, (x′, y′) is the two-dimensional annotation structure after two-dimensional annotation of vehicle key points in the image using a 2D annotation model, that is, the annotation coordinates after two-dimensional annotation of vehicle key points in the image using a 2D annotation model. Since solvePnP can be used to calculate... These 6 parameters, namely Since the values ​​are fixed, we can determine the projection relationship between the 3D keypoint annotation results and the 2D projection results. Therefore, the errors Δx and Δy can be considered as relating to X. A Y A Z A The function. In this embodiment, the preset threshold can be set according to the actual situation. As an example, the preset threshold can be a value calculated according to the least squares method.

[0091] According to the above description, when the error value is greater than a preset threshold, the adjustment process of the 3D annotation model includes: obtaining the projection equation between the 2D key points and the 3D key points, and obtaining the projection relationship when the 3D key point annotation result is projected into the 2D projection result according to the projection equation; matching the 2D key point annotation result of each key point in the vehicle to be annotated with the 3D key point annotation result obtained from the initial 3D annotation model or the 3D annotation model after the last adjustment to obtain multiple key point pairs; determining the key points with annotation errors according to the projection equation and the multiple key point pairs, and using the least squares method to determine the correction value used to update the position of the key points with annotation errors; adjusting the 3D annotation model corresponding to the error value being greater than the preset threshold according to the projection relationship and the correction value to obtain the adjusted 3D annotation model. Specifically, this embodiment uses solvePnP to calculate the projection equation in the above projection equation. These six parameters yield the projection relationship from the 3D keypoint annotation result to the 2D projection result. Then, keypoints with annotation errors are identified, and the least squares method is used to calculate and update the correction values ​​for the keypoint positions. Finally, the 3D annotation model is adjusted based on the obtained projection relationship and the calculated correction values ​​to obtain the adjusted 3D annotation model. As an example, let's take... Figures 2a to 2e Taking 20 key points of a sedan as an example, since the key points of a sedan are symmetrical from left to right, this embodiment identifies key points numbered 0 to 9 as key points on the left side of the vehicle, and the key points on the right side are numbered 10 more than the numbers of the key points on the left side. Furthermore, according to the above embodiment, the errors Δx and Δy can be considered as relating to X. A Y A Z A The function, so for X A In reality, its size is determined by the distance between keypoint A and keypoint A+10, i.e., the width W. A Therefore, for W A The error equation can be written as:

[0092] Δx A =F X (W A )-x A ′;

[0093] Δy A =F Y (W A )-y A ′;

[0094] Δx A+10 =F X (W A )-x A+10 ′;

[0095] ΔyA+10 =F Y (W A )-y A+10 ′;

[0096] For W A If one of points A and A+10 is visible, then two equations can be written; if both are visible, then four equations can be written. The least squares method is used to solve for W. A The error equation can be solved to obtain W. A This allows the projection error to be minimized. Similarly, for Y... A In reality, its size is determined by the height H shared by keypoint A and keypoint A+10. A For Z A In reality, its size is determined by the length L shared by keypoint A and keypoint A+10. A Therefore, this embodiment uses the method of solving W. A The same method is used to solve H A L A Finally, we obtain the new 3D coordinates of key point A and key point A+10.

[0097] By iterating through and solving for the W, H, and L parameters of keypoints 0 to 9 using the method described above, new 3D coordinates for these keypoints are obtained. Then, based on the new 3D coordinates of the 20 keypoints in the car and the pre-determined projection relationship, the projection positions of the new 3D coordinates on the image can be obtained. Simultaneously, based on the projection positions of the new and original 3D coordinates on the image, correction values ​​for the 3D annotation model can be calculated. These correction values ​​are then used to adjust the 3D annotation model, resulting in an adjusted 3D annotation model. Furthermore, based on the spatial relationships of the adjusted 3D annotation model, it can be determined whether one or more keypoints are occluded by the vehicle itself. If not occluded by the vehicle and not labeled, the annotation category is "Other Occlusion"; if the keypoint is outside the image range, the annotation category is "Truncation". Thus, five annotation categories are obtained: Visible, Truncation, Self-Occlusion, Other Occlusion, and Unknown. In this context, "truncation" refers to a 2D keypoint of a vehicle in an image that extends beyond or lies on the image edge; "visible" means the 2D keypoint of a vehicle in the image is directly observable to the naked eye; "self-occlusion" means the 2D keypoint of a vehicle in the image is obscured by the vehicle itself, or is not prominent due to angle concealment; "other occlusion" means the 2D keypoint of a vehicle is obscured by other objects; and "unknown" refers to points that do not exist in the image, and their type is unknown. In this embodiment, truncation can be labeled as needed; visible points must be labeled; points obscured by the vehicle itself can be labeled as needed; self-occlusion that is not prominent due to angle concealment can be labeled as much as possible; other occlusions can be labeled as needed; and unknown points do not need to be labeled. Figure 3 As shown, Figure 3 Create a 3D keypoint annotation model for the truck, including but not limited to 20 keypoints.

[0098] As another example, such as Figure 4 As shown, the present invention also provides a method for marking key points of a vehicle, comprising:

[0099] S101. Select the corresponding vehicle type based on the vehicle to be labeled. For example, the labeler can select the vehicle type corresponding to the vehicle to be labeled, where the available vehicle types include, but are not limited to, sedans, buses, vans, trucks, pickups, etc.

[0100] S102, Mark the location and category of key points in the vehicle to be marked. For different types, the location of the vehicle's key points is predefined in this embodiment. For each key point, the marked category includes, but is limited to: visible, truncated, self-occluded, other-occluded, and unknown. As an example, taking the key points of a sedan as an example, such as... Figures 2a to 2eAs shown, the car predefines 20 key points, where 0 represents the upper left corner of the front roof, 1 represents the outer left corner of the upper left edge of the headlight, 2 represents the outer left corner of the lower left edge of the front chassis, 3 represents the left center point of the front wheel, 4 represents the left center point of the rear wheel, 5 represents the left corner of the rear chassis, 6 represents the outer left corner of the upper left edge of the taillight, 7 represents the upper left corner of the rear roof, 8 represents the front edge of the bottom edge of the left window, 9 represents the end of the bottom edge of the left window, 10 represents the upper right corner of the front roof, 11 represents the outer right corner of the upper right edge of the headlight, 12 represents the outer right corner of the lower right edge of the front chassis, 13 represents the right center point of the front wheel, 14 represents the right center point of the rear wheel, 15 represents the right corner of the rear chassis, 16 represents the outer right corner of the upper right edge of the taillight, 17 represents the upper right corner of the rear roof, 18 represents the front edge of the bottom edge of the right window, and 19 represents the end of the bottom edge of the right window.

[0101] S103, after completing the annotation of key points, the selected vehicle model and the annotated 2D key points are used to guide the adjustment and deformation of the 3D annotation model, and the projection relationship from the 3D annotation result to the 2D projection result is obtained, resulting in the 2D projection result of all 3D key points. As an example, the 3D annotation model in this embodiment is implemented using a self-developed Keypoint Deformable Model (KDM). The adjustment and deformation process of the Keypoint Deformable Model is as follows:

[0102] Establish projection equations for vehicle keypoints in a two-dimensional plane and a three-dimensional space. Vehicle keypoints in the two-dimensional plane are denoted as 2D keypoints, and vehicle keypoints in the three-dimensional space are denoted as 3D keypoints. The projection equations are:

[0103]

[0104]

[0105] In the formula, For the exterior orientation elements of the camera, where X is the angular element of the camera, representing the camera's orientation in the real world; S Y S Z S The line element represents the camera's spatial position in the real world. (X) A Y A Z A (x, y) represents the coordinates of the vehicle's key points in the real world, and (x, y) is the coordinate of the 3D key points projected onto the image.

[0106] The exterior orientation elements of the camera are calculated using solvePnP, i.e., solvePnP is used to calculate... These six parameters determine the projection relationship from the 3D keypoint annotation result to the 2D projection result. Specifically, when calculating the camera's exterior orientation elements, all visible 2D keypoints in the image are first found. Then, these are compared with the 3D keypoints annotated by a pre-prepared 3D annotation model (i.e., the unadjusted 3D annotation model or the initial 3D annotation model) to form n 2D-3D keypoint pairs. Each keypoint pair can be used to formulate two equations: one about the x-axis and one about the y-axis. When n≥3, solvePnP is then used to calculate... These 6 parameters.

[0107] Because the pre-determined 3D annotation model may not be compatible with the vehicle model in the current image, the resulting 2D projection of the 3D keypoint annotations obtained from the pre-determined 3D annotation model onto the image will result in an error between the actual 2D keypoint annotations and the projection result. This error can be expressed as:

[0108]

[0109]

[0110] In the formula, (x′, y′) is the two-dimensional annotation structure after two-dimensional annotation of vehicle key points in the image using a 2D annotation model, that is, the annotation coordinates after two-dimensional annotation of vehicle key points in the image using a 2D annotation model. Since solvePnP can be used to calculate... These 6 parameters, namely Since the values ​​are fixed, we can determine the projection relationship between the 3D keypoint annotation results and the 2D projection results. Therefore, the errors Δx and Δy can be considered as relating to X. A Y A Z A The function. In this embodiment, the preset threshold can be set according to the actual situation. As an example, the preset threshold can be a value calculated according to the least squares method.

[0111] Since the key points of a car are symmetrical from left to right, this embodiment identifies key points numbered 0 to 9 as key points on the left side of the vehicle, and the key points on the right side are numbered by adding 10 to the number of the left key points. Furthermore, as can be seen from the above embodiment, the errors Δx and Δy can be considered as relating to X. A Y A Z A The function, so for X A In reality, its size is determined by the distance between keypoint A and keypoint A+10, i.e., the width W. A Therefore, for W A The error equation can be written as:

[0112] Δx A =F X (W A )-x A ′;

[0113] Δy A =F Y (W A )-y A ′;

[0114] Δx A+10 =F X (W A )-x A+10 ′;

[0115] Δy A+10 =F Y (W A )-y A+10 ′;

[0116] For W A If one of points A and A+10 is visible, then two equations can be written; if both are visible, then four equations can be written. The least squares method is used to solve for W. A The error equation can be solved to obtain W. A This allows the projection error to be minimized. Similarly, for Y... A In reality, its size is determined by the height H shared by keypoint A and keypoint A+10. A For Z A In reality, its size is determined by the length L shared by keypoint A and keypoint A+10. A Therefore, this embodiment uses the method of solving W. A The same method is used to solve H A L A Finally, we obtain the new 3D coordinates of key point A and key point A+10.

[0117] Following the above process, the W, H, and L parameters of key points 0 to 9 are solved to obtain the new 3D coordinates of key points 0 to 9. Then, based on the new 3D coordinates corresponding to the 20 key points in the car and the pre-determined projection relationship, the projection position of the new 3D coordinates on the image can be obtained. At the same time, based on the projection position of the new 3D coordinates on the image and the projection position of the original 3D coordinates on the image, the correction value of the 3D annotation model can be calculated. Then, the obtained correction value is used to adjust the 3D annotation model to obtain the adjusted 3D annotation model.

[0118] S104. Using the adjusted 3D annotation model, key points are annotated on the vehicle to be annotated, obtaining the annotation positions and categories of all key points. The annotation positions and categories obtained using the 3D annotation model are compared with the 2D annotation results for error, and the annotator determines whether the error is within a preset range. If the error is not within the preset range, the adjusted 3D annotation model is iteratively adjusted until the annotator confirms that the key point annotation positions and categories obtained from the adjusted 3D annotation model are within the preset range.

[0119] S105 outputs the adjusted 3D annotation model, along with the annotation results for the vehicle to be annotated, and the projection relationship from the 3D keypoint annotation results to the 2D projection results. The annotation results include: the annotation categories and coordinates of all keypoints in the 3D annotation model, and the 2D projection positions and categories of all 3D keypoints.

[0120] In summary, this invention provides a method for annotating key points of a vehicle. First, the vehicle model to be annotated is determined. Then, a corresponding 3D annotation model is obtained based on the determined model, and 3D key points of the vehicle are annotated using the obtained 3D annotation model. Next, the 3D key point annotation results are projected into 2D to obtain the 2D projection results of all key points in the vehicle. Finally, 2D key point annotations are performed on the vehicle, and the final key point annotation result of the vehicle is determined based on the 2D key point annotation results and the 2D projection results. This method addresses the shortcomings of current CAD model-based annotation tools by designing a 3D vehicle annotation method based on a deformable key point model. By associating the 2D projection results and the 2D key point annotation results as the key point annotation results of the vehicle, this method can use both the annotation results of the 2D annotation model and the 2D projection results of the 3D annotation model as the key point annotation results of the vehicle, thus solving the annotation error problem that exists when using only the 2D or 3D annotation model. Furthermore, this method can guide the deformation of the 3D annotation model by using the key point positions in the 2D key point annotation results, solving the problem of incompatibility of existing 3D annotation models. The annotation process eliminates the need to manually select vehicle models from hundreds of 3D annotation models, reducing the annotation difficulty. Simultaneously, by annotating the 3D information of the vehicle, the information dimensionality and density can be increased. Moreover, by adjusting and deforming the 3D annotation model, this method not only improves annotation accuracy but also eliminates the need to select specific vehicle models, thus reducing the difficulty of annotating key points for the vehicle to be annotated. Furthermore, relying on the 2D projection results of the 3D annotation model and the 2D annotation results directly from the 2D annotation model, this method can output the positions and categories of all 2D key points corresponding to the vehicle model to be annotated.

[0121] like Figure 5 As shown, the present invention also provides a system for marking key points of a vehicle, comprising:

[0122] The vehicle model module M10 is used to determine the vehicle model of the vehicle to be labeled;

[0123] The first annotation module M20 is used to obtain the corresponding three-dimensional annotation model according to the determined vehicle model, and to use the obtained three-dimensional annotation model to annotate the three-dimensional key points of the vehicle to be annotated.

[0124] The projection module M30 is used to perform two-dimensional projection on the three-dimensional key point annotation results to obtain the two-dimensional projection results of all key points in the vehicle to be annotated.

[0125] The second annotation module M40 is used to annotate the two-dimensional key points of the vehicle to be annotated, and to associate the two-dimensional projection result and the two-dimensional key point annotation result as the key point annotation result of the vehicle to be annotated.

[0126] Therefore, this system addresses the shortcomings of current CAD model-based annotation tools by designing a vehicle 3D annotation method based on a deformable keypoint model. By linking the 2D projection results with the 2D keypoint annotation results, this method serves as the keypoint annotation result for the vehicle to be annotated. Essentially, this system can use both the annotation results of the 2D annotation model and the 2D projection results of the 3D annotation model as the keypoint annotation result for the vehicle to be annotated, thus resolving the annotation error problem that exists when using only the 2D or 3D annotation model. Furthermore, relying on the 2D projection results of the 3D annotation model and the 2D annotation results directly from the 2D annotation model, this system can output the location and category of all 2D keypoints corresponding to the vehicle model to be annotated.

[0127] According to the above records, in this system, the key points of the vehicle to be labeled include at least one of the following: the upper left corner of the front roof, the outer left corner of the upper left edge of the front headlight, the outer left corner of the lower left edge of the front chassis, the center left of the front wheel, the center left of the rear wheel, the left corner of the rear chassis, the outer left corner of the upper left edge of the taillight, the upper left corner of the rear roof, the front end of the bottom edge of the left window, the end of the bottom edge of the left window, the upper right corner of the front roof, the outer right corner of the upper right edge of the front headlight, the outer right corner of the lower right edge of the front chassis, the center right of the front wheel, the center right of the rear wheel, the center right of the rear chassis, the outer right corner of the upper right edge of the taillight, the upper right corner of the rear roof, the front end of the bottom edge of the right window, and the end of the bottom edge of the right window. As an example, the key points of a certain sedan are as follows: Figures 2a to 2eAs shown, 0 represents the top left corner of the front roof, 1 represents the outer corner of the top left edge of the front headlight, 2 represents the outer corner of the bottom left edge of the front chassis, 3 represents the center left edge of the front wheel, 4 represents the center left edge of the rear wheel, 5 represents the left corner of the rear chassis, 6 represents the outer corner of the top left edge of the taillight, 7 represents the top left corner of the rear roof, 8 represents the front edge of the bottom edge of the left window, 9 represents the end of the bottom edge of the left window, 10 represents the top right corner of the front roof, 11 represents the outer right corner of the top right edge of the front headlight, 12 represents the outer right corner of the bottom right edge of the front chassis, 13 represents the center right edge of the front wheel, 14 represents the center right edge of the rear wheel, 15 represents the right corner of the rear chassis, 16 represents the outer right corner of the top right edge of the taillight, 17 represents the top right corner of the rear roof, 18 represents the front edge of the bottom edge of the right window, and 19 represents the end of the bottom edge of the right window.

[0128] In an exemplary embodiment, after the second annotation module M40 performs two-dimensional keypoint annotation on the vehicle to be annotated, it further includes: calculating the error value between the two-dimensional keypoint annotation result and the two-dimensional projection result; if the error value is greater than a preset threshold, adjusting the three-dimensional annotation model, and using the adjusted three-dimensional annotation model to re-annotate the three-dimensional keypoints of the vehicle to be annotated, and calculating the error value between the new two-dimensional projection result and the two-dimensional keypoint annotation result; if the error value obtained at this time is still greater than the preset threshold, iteratively adjusting the three-dimensional annotation model continues until the error value between the new two-dimensional projection result obtained using the adjusted three-dimensional annotation model and the two-dimensional keypoint annotation result is less than or equal to the preset threshold, at which point the iterative adjustment of the three-dimensional annotation model stops. If the error value is less than or equal to the preset threshold, the two-dimensional projection result and the two-dimensional keypoint annotation result at the current moment are associated as the final keypoint annotation result of the vehicle to be annotated, and the projection relationship of the three-dimensional keypoint annotation result projected into the two-dimensional projection result at the current moment is obtained.

[0129] As an example, in this embodiment, when calculating the error value between the two-dimensional key point annotation result and the two-dimensional projection result, the method further includes establishing projection equations for vehicle key points in the two-dimensional plane and three-dimensional space. Vehicle key points in the two-dimensional plane are denoted as two-dimensional key points or 2D key points, and vehicle key points in the three-dimensional space are denoted as three-dimensional key points or 3D key points. The projection equations are:

[0130]

[0131]

[0132] In the formula, For the exterior orientation elements of the camera, where X is the angular element of the camera, representing the camera's orientation in the real world; S YS Z S The line element represents the camera's spatial position in the real world. (X) A Y A Z A (x, y) represents the coordinates of the vehicle's key points in the real world, and (x, y) is the coordinate of the 3D key points projected onto the image.

[0133] Then, solvePnP is used to calculate the exterior orientation elements of the camera, obtaining the projection relationship from the 3D keypoint annotation results to the 2D projection results; that is, solvePnP is used to calculate... These are the six parameters. Specifically, when calculating the camera's exterior orientation elements, first, all visible 2D keypoints in the image are found. Then, these are compared with the 3D keypoints annotated by a pre-prepared 3D annotation model (i.e., an unadjusted 3D annotation model or an initial 3D annotation model) to form n 2D-3D keypoint pairs. Each keypoint pair can be used to formulate two equations: one about the x-axis and one about the y-axis. When n≥3, solvePnP is used to calculate... These 6 parameters.

[0134] Calculated using solvePnP These six parameters, after determining the projection relationship from the 3D keypoint annotation results to the 2D projection results, also include calculating the error value between the 2D keypoint annotation results and the 2D projection results:

[0135]

[0136]

[0137] In the formula, (x′, y′) is the two-dimensional annotation structure after two-dimensional annotation of vehicle key points in the image using a 2D annotation model, that is, the annotation coordinates after two-dimensional annotation of vehicle key points in the image using a 2D annotation model. Since solvePnP can be used to calculate... These 6 parameters, namely Since the values ​​are fixed, we can determine the projection relationship between the 3D keypoint annotation results and the 2D projection results. Therefore, the errors Δx and Δy can be considered as relating to X. A Y A Z A The function. In this embodiment, the preset threshold can be set according to the actual situation. As an example, the preset threshold can be a value calculated according to the least squares method.

[0138] According to the above description, when the error value is greater than a preset threshold, the adjustment process of the 3D annotation model includes: obtaining the projection equation between the 2D key points and the 3D key points, and obtaining the projection relationship when the 3D key point annotation result is projected into the 2D projection result according to the projection equation; matching the 2D key point annotation result of each key point in the vehicle to be annotated with the 3D key point annotation result obtained from the initial 3D annotation model or the 3D annotation model after the last adjustment to obtain multiple key point pairs; determining the key points with annotation errors according to the projection equation and the multiple key point pairs, and using the least squares method to determine the correction value used to update the position of the key points with annotation errors; adjusting the 3D annotation model corresponding to the error value being greater than the preset threshold according to the projection relationship and the correction value to obtain the adjusted 3D annotation model. Specifically, this embodiment uses solvePnP to calculate the projection equation in the above projection equation. These six parameters yield the projection relationship from the 3D keypoint annotation result to the 2D projection result. Then, keypoints with annotation errors are identified, and the least squares method is used to calculate and update the correction values ​​for the keypoint positions. Finally, the 3D annotation model is adjusted based on the obtained projection relationship and the calculated correction values ​​to obtain the adjusted 3D annotation model. As an example, let's take... Figures 2a to 2e Taking 20 key points of a sedan as an example, since the key points of a sedan are symmetrical from left to right, this embodiment identifies key points numbered 0 to 9 as key points on the left side of the vehicle, and the key points on the right side are numbered 10 more than the numbers of the key points on the left side. Furthermore, according to the above embodiment, the errors Δx and Δy can be considered as relating to X. A Y A Z A The function, so for X A In reality, its size is determined by the distance between keypoint A and keypoint A+10, i.e., the width W. A Therefore, for W A The error equation can be written as:

[0139] Δx A =F X (W A )-x A ′;

[0140] Δy A =F Y (W A )-y A ′;

[0141] Δx A+10 =F X (W A )-x A+10 ′;

[0142] ΔyA+10 =F Y (W A )-y A+10 ′;

[0143] For W A If one of points A and A+10 is visible, then two equations can be written; if both are visible, then four equations can be written. The least squares method is used to solve for W. A The error equation can be solved to obtain W. A This allows the projection error to be minimized. Similarly, for Y... A In reality, its size is determined by the height H shared by keypoint A and keypoint A+10. A For Z A In reality, its size is determined by the length L shared by keypoint A and keypoint A+10. A Therefore, this embodiment uses the method of solving W. A The same method is used to solve H A L A Finally, we obtain the new 3D coordinates of key point A and key point A+10.

[0144] By iterating through and solving for the W, H, and L parameters of keypoints 0 to 9 using the method described above, new 3D coordinates for these keypoints are obtained. Then, based on the new 3D coordinates of the 20 keypoints in the car and the pre-determined projection relationship, the projection positions of the new 3D coordinates on the image can be obtained. Simultaneously, based on the projection positions of the new and original 3D coordinates on the image, correction values ​​for the 3D annotation model can be calculated. These correction values ​​are then used to adjust the 3D annotation model, resulting in an adjusted 3D annotation model. Furthermore, based on the spatial relationships of the adjusted 3D annotation model, it can be determined whether one or more keypoints are occluded by the vehicle itself. If not occluded by the vehicle and not labeled, the annotation category is "Other Occlusion"; if the keypoint is outside the image range, the annotation category is "Truncation". Thus, five annotation categories are obtained: Visible, Truncation, Self-Occlusion, Other Occlusion, and Unknown. In this context, "truncation" refers to a 2D keypoint of a vehicle in an image that extends beyond or lies on the image edge; "visible" means the 2D keypoint of a vehicle in the image is directly observable to the naked eye; "self-occlusion" means the 2D keypoint of a vehicle in the image is obscured by the vehicle itself, or is not prominent due to angle concealment; "other occlusion" means the 2D keypoint of a vehicle is obscured by other objects; and "unknown" refers to points that do not exist in the image, and their type is unknown. In this embodiment, truncation can be labeled as needed; visible points must be labeled; points obscured by the vehicle itself can be labeled as needed; self-occlusion that is not prominent due to angle concealment can be labeled as much as possible; other occlusions can be labeled as needed; and unknown points do not need to be labeled. Figure 3 As shown, Figure 3 Create a 3D keypoint annotation model for the truck, including but not limited to 20 keypoints.

[0145] As another example, the present invention also provides a system for annotating key points of a vehicle, for implementing... Figure 4 The method for annotating key points of vehicles is described above. The technical functions and effects of this system are detailed in the above embodiments and will not be repeated here.

[0146] In summary, this invention provides a system for annotating key points of a vehicle. First, the vehicle model to be annotated is determined. Then, a corresponding 3D annotation model is obtained based on the determined model, and 3D key points of the vehicle are annotated using the obtained 3D annotation model. Next, the 3D key point annotation results are projected into 2D to obtain the 2D projection results of all key points in the vehicle. Finally, 2D key point annotations are performed on the vehicle, and the final key point annotation result of the vehicle is determined based on the 2D key point annotation results and the 2D projection results. This system addresses the shortcomings of current CAD model-based annotation tools by designing a 3D vehicle annotation method based on a deformable key point model. By associating the 2D projection results and the 2D key point annotation results as the key point annotation results of the vehicle, this system can use both the annotation results of the 2D annotation model and the 2D projection results of the 3D annotation model as the key point annotation results of the vehicle, thus solving the annotation error problem that exists when using only the 2D or 3D annotation model. Furthermore, this system can guide the deformation of the 3D annotation model by using the key point positions in the 2D key point annotation results, solving the problem of incompatibility of existing 3D annotation models. The annotation process eliminates the need to manually select vehicle models from hundreds of 3D annotation models, reducing the annotation difficulty. Simultaneously, by annotating the 3D information of the vehicle, the system can improve the information dimensionality and density. Moreover, by adjusting and deforming the 3D annotation model, this system not only improves annotation accuracy but also eliminates the need to select specific vehicle models, thus reducing the difficulty of annotating key points for the vehicle to be annotated. At the same time, relying on the 2D projection results of the 3D annotation model and the 2D annotation results directly from the 2D annotation model, this system can output the positions and categories of all 2D key points corresponding to the vehicle model to be annotated.

[0147] This application also provides a device for marking key points on a vehicle. The device may include: one or more processors; and one or more machine-readable media storing instructions thereon, which, when executed by the one or more processors, cause the device to perform... Figure 1The method described herein. In practical applications, the device can function as a terminal device or a server. Examples of terminal devices include: smartphones, tablets, e-book readers, MP3 (Moving Picture Experts Group Audio Layer III) players, MP4 (Moving Picture Experts Group Audio Layer IV) players, laptops, in-vehicle computers, desktop computers, set-top boxes, smart TVs, wearable devices, etc. This application does not limit the specific devices described.

[0148] This application also provides a non-volatile readable storage medium storing one or more modules (programs). When these modules are applied to a device, they enable the device to execute embodiments of this application. Figure 1 The instructions for the steps involved in marking key points on a vehicle.

[0149] Figure 6 This is a schematic diagram of the hardware structure of a terminal device provided in an embodiment of this application. As shown in the figure, the terminal device may include: an input device 1100, a first processor 1101, an output device 1102, a first memory 1103, and at least one communication bus 1104. The communication bus 1104 is used to realize communication connections between components. The first memory 1103 may include a high-speed RAM memory, and may also include non-volatile memory (NVM), such as at least one disk storage device. The first memory 1103 can store various programs for performing various processing functions and implementing the method steps of this embodiment.

[0150] Optionally, the first processor 1101 may be implemented as a central processing unit (CPU), application-specific integrated circuit (ASIC), digital signal processor (DSP), digital signal processing device (DSPD), programmable logic device (PLD), field-programmable gate array (FPGA), controller, microcontroller, microprocessor or other electronic components. The processor 1101 is coupled to the input device 1100 and output device 1102 via wired or wireless connection.

[0151] Optionally, the input device 1100 may include a variety of input devices, such as a user interface, a device interface, a programmable software interface, a camera, and a sensor. Optionally, the device interface may be a wired interface for data transmission between devices, or a hardware insertion interface (e.g., USB interface, serial port) for data transmission between devices. Optionally, the user interface may be, for example, user-facing control buttons, a voice input device for receiving voice input, or a touch sensing device for receiving user touch input (e.g., a touchscreen, touchpad, etc.). Optionally, the programmable software interface may be, for example, an entry point for users to edit or modify programs, such as a chip input pin interface or input interface. The output device 1102 may include a display, speakers, and other output devices.

[0152] In this embodiment, the processor of the terminal device includes functions for executing the functions of each module of the voice recognition device in each device. The specific functions and technical effects can be referred to in the above embodiment, and will not be repeated here.

[0153] Figure 7 This is a schematic diagram of the hardware structure of a terminal device provided for another embodiment of this application. Figure 7 Yes Figure 6 This is a specific embodiment of the implementation process. As shown in the figure, the terminal device of this embodiment may include a second processor 1201 and a second memory 1202.

[0154] The second processor 1201 executes the computer program code stored in the second memory 1202 to implement the above embodiments. Figure 1 The method is described above.

[0155] The second memory 1202 is configured to store various types of data to support operation on the terminal device. Examples of this data include instructions for any application or method operating on the terminal device, such as messages, pictures, videos, etc. The second memory 1202 may include random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device.

[0156] Optionally, the second processor 1201 is disposed in the processing component 1200. The terminal device may also include: a communication component 1203, a power supply component 1204, a multimedia component 1205, an audio component 1206, an input / output interface 1207, and / or a sensor component 1208. The specific components included in the terminal device are determined according to actual needs, and this embodiment does not limit this.

[0157] Processing component 1200 typically controls the overall operation of the terminal device. Processing component 1200 may include one or more second processors 1201 to execute instructions to perform the above-mentioned tasks. Figure 1 The method shown may include all or part of the steps. Furthermore, the processing component 1200 may include one or more modules to facilitate interaction between the processing component 1200 and other components. For example, the processing component 1200 may include a multimedia module to facilitate interaction between the multimedia component 1205 and the processing component 1200.

[0158] Power supply component 1204 provides power to various components of the terminal device. Power supply component 1204 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the terminal device.

[0159] Multimedia component 1205 includes a display screen that provides an output interface between a terminal device and a user. In some embodiments, the display screen may include a liquid crystal display (LCD) and a touch panel (TP). If the display screen includes a touch panel, the display screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation.

[0160] Audio component 1206 is configured to output and / or input voice signals. For example, audio component 1206 includes a microphone (MIC) configured to receive external voice signals when the terminal device is in an operating mode, such as a voice recognition mode. The received voice signals may be further stored in a second memory 1202 or transmitted via communication component 1203. In some embodiments, audio component 1206 also includes a speaker for outputting voice signals.

[0161] Input / output interface 1207 provides an interface between processing component 1200 and peripheral interface modules, such as click wheels, buttons, etc. These buttons may include, but are not limited to, volume buttons, start buttons, and lock buttons.

[0162] Sensor assembly 1208 includes one or more sensors for providing status assessments of various aspects of the terminal device. For example, sensor assembly 1208 can detect the on / off state of the terminal device, the relative positioning of components, and the presence or absence of user contact with the terminal device. Sensor assembly 1208 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact, including detecting the distance between the user and the terminal device. In some embodiments, sensor assembly 1208 may also include a camera, etc.

[0163] Communication component 1203 is configured to facilitate wired or wireless communication between the terminal device and other devices. The terminal device can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one embodiment, the terminal device may include a SIM card slot for inserting a SIM card, enabling the terminal device to log in to a GPRS network and establish communication with a server via the Internet.

[0164] As can be seen from the above, in Figure 7 The communication component 1203, audio component 1206, input / output interface 1207, and sensor component 1208 involved in the embodiment can all be used as... Figure 6 The implementation method of the input device in the embodiment.

[0165] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A method for annotating key points of a vehicle, characterized in that, The method comprises the following steps: determining the vehicle model of a vehicle to be labeled; acquiring a corresponding three-dimensional labeling model according to the determined vehicle model, and performing three-dimensional key point labeling on the vehicle to be labeled by using the acquired three-dimensional labeling model; projecting the three-dimensional key point labeling result in two dimensions to acquire a two-dimensional projection result of all key points in the vehicle to be labeled; performing two-dimensional key point labeling on the vehicle to be labeled, and calculating an error value of the two-dimensional key point labeling result and the two-dimensional projection result; and when the error value is less than or equal to a preset threshold, determining a final key point labeling result of the vehicle to be labeled based on the two-dimensional key point labeling result and the two-dimensional projection result at the current time; wherein the process of calculating the error value of the two-dimensional key point labeling result and the two-dimensional projection result comprises: denoting a vehicle key point in a two-dimensional plane as a two-dimensional key point, and a vehicle key point in a three-dimensional space as a three-dimensional key point, establishing a projection equation of the vehicle key point in the two-dimensional plane and the three-dimensional space, which is: wherein ω, k, X S ,Y S ,Z S are extrinsic elements of the camera, wherein ω, k are angular elements of the camera, representing the orientation of the camera in the real world; X S ,Y S ,Z S are linear elements of the camera, representing the spatial position of the camera in the real world, (X A ,Y A ,Z A ) represents the coordinates of the vehicle key points in the real world, (x, y) is the coordinates of the three-dimensional key points projected onto the image. obtaining a projection relationship of the three-dimensional key point labeling result projected into the two-dimensional projection result by using the exterior orientation elements of the camera, and calculating the error value of the two-dimensional key point labeling result and the two-dimensional projection result, which is: In the formula, (x', y') is a two-dimensional labeling structure of a vehicle key point in an image after two-dimensional labeling by a two-dimensional labeling model, and errors Δx and Δy are functions of X A , Y A , and Z A . and the distance between keypoint A and keypoint A+10 is recorded as width W A and through width W A determines the size of X A ; where if there is one visible point in points A and A+10, then W A The error equation for W can be written as: If point A is visible in point A+10, then W A The error equation can be written as: Δx A = F X (W A )- x A '; Δy A = F Y (W A )- y A ′; Δx A+10 = F X (W A )- x A+10 '; Δy A+10 = F Y (W A )- y A+10 ′; The error equation of W is solved using the least square method to make the projection error smaller. A The error equation of W is solved using the least square method to make the projection error smaller.

2. The method of annotating vehicle key points of claim 1, wherein, The method further comprises: if the error value is greater than the preset threshold, adjusting the three-dimensional labeling model at least once, so that the error value of the new two-dimensional projection result obtained by using the adjusted three-dimensional labeling model and the two-dimensional key point labeling result is less than or equal to the preset threshold; when the error value is less than or equal to the preset threshold, further comprising obtaining a projection relationship of the three-dimensional key point labeling result projected into the two-dimensional projection result at the current time.

3. The method of annotating vehicle key points of claim 2, wherein, When the error value is greater than the preset threshold, the adjustment process of the three-dimensional labeling model comprises: obtaining a projection equation between the two-dimensional key point and the three-dimensional key point, and obtaining a projection relationship of the three-dimensional key point labeling result projected into the two-dimensional projection result according to the projection equation; matching the two-dimensional key point labeling result of each key point in the vehicle to be labeled and the three-dimensional key point labeling result obtained by the initial three-dimensional labeling model or the three-dimensional labeling model after the last adjustment to obtain a plurality of key point pairs; determining a key point with labeling error according to the projection equation and the plurality of key point pairs, and determining a correction value for updating the position of the key point with labeling error by using the least square method; adjusting the three-dimensional labeling model corresponding to the error value greater than the preset threshold according to the projection relationship and the correction value to obtain an adjusted three-dimensional labeling model.

4. The method of annotating vehicle key points of claim 1, wherein, The key points of the vehicle to be labeled comprise at least one of: The front roof left side upper edge corner point, the front headlamp left side upper edge outer corner point, the front chassis left side lower edge outer corner point, the front wheel left center point, the rear wheel left center point, the rear chassis left side corner point, the rear tail lamp left side upper edge outer corner point, the rear roof left side upper edge corner point, the left side window lowermost edge front end, the left side window lowermost edge end, the front roof right side upper edge corner point, the front headlamp right side upper edge outer corner point, the front chassis right side lower edge outer corner point, the front wheel right center point, the rear wheel right center point, the rear chassis right side corner point, the rear tail lamp right side upper edge outer corner point, the rear roof right side upper edge corner point, the right side window lowermost edge front end, and the right side window lowermost edge end.

5. The method of annotating vehicle key points of claim 1, wherein, When the two-dimensional key point labeling or the three-dimensional key point labeling is performed on the vehicle to be labeled, the labeled categories include at least one of the following: visible, self-occluded, other-occluded, truncated, and unknown.

6. The method of annotating vehicle key points of claim 1, wherein, The vehicle type includes at least one of the following: a sedan, a bus, a van, a truck, and a pickup truck.

7. A system for annotating key points of a vehicle, the system comprising: The vehicle type module is configured to determine a vehicle type of the vehicle to be labeled. The first labeling module is configured to acquire a corresponding three-dimensional labeling model according to the determined vehicle type, and perform three-dimensional key point labeling on the vehicle to be labeled by using the acquired three-dimensional labeling model. The projection module is configured to project the three-dimensional key point labeling result to two dimensions to acquire a two-dimensional projection result of all key points in the vehicle to be labeled. The second labeling module is configured to perform two-dimensional key point labeling on the vehicle to be labeled, and calculate an error value of the two-dimensional key point labeling result and the two-dimensional projection result. When the error value is less than or equal to a preset threshold, the final key point labeling result of the vehicle to be labeled is determined according to the two-dimensional key point labeling result and the two-dimensional projection result at the current time. The process of calculating the error value of the two-dimensional key point labeling result and the two-dimensional projection result includes: The vehicle key points on a two-dimensional plane are recorded as two-dimensional key points, and the vehicle key points in a three-dimensional space are recorded as three-dimensional key points, a projection equation of the vehicle key points on the two-dimensional plane and the three-dimensional space is established, and there is: The exterior orientation elements of the camera are used to obtain a projection relationship of the projection of the three-dimensional key point labeling result to the two-dimensional projection result, and the error value of the two-dimensional key point labeling result and the two-dimensional projection result is calculated, and there is: wherein ω, k, X S ,Y S ,Z S are extrinsic elements of the camera, wherein ω, k are angular elements of the camera, representing the orientation of the camera in the real world; X S ,Y S ,Z S are linear elements of the camera, representing the spatial position of the camera in the real world, (X A ,Y A ,Z A ) represents the coordinates of the vehicle key points in the real world, (x, y) is the coordinates of the three-dimensional key points projected onto the image. The second labeling module further includes: In the formula, (x', y') is a two-dimensional labeling structure of a vehicle key point in an image after two-dimensional labeling by a two-dimensional labeling model, and errors Δx and Δy are functions of X A , Y A , and Z A . Also, the distance between keypoint A and keypoint A+10 is denoted as width W. A And through width W A Decision X A Size; where if there is one visible point in points A and A+10, then W A The error equation for W can be written as: If point A is visible in point A+10, then W A The error equation can be written as: Δx A = F X (W A )- x A ′; Δy A = F Y (W A )- y A ′; Δx A+10 = F X (W A )- x A+10 ′; Δy A+10 = F Y (W A )- y A+10 ′; The error equation of W is solved using the least square method to make the projection error smaller. A The error equation of W is solved using the least square method to make the projection error smaller.

8. The system for annotating vehicle key points of claim 7, wherein, If the error value is greater than the preset threshold, the three-dimensional labeling model is adjusted at least once, so that a new two-dimensional projection result obtained by using the adjusted three-dimensional labeling model has an error value less than or equal to the preset threshold with the two-dimensional key point labeling result. When the error value is less than or equal to the preset threshold, the projection relationship of the projection of the three-dimensional key point labeling result to the two-dimensional projection result at the current time is further acquired. When the error value is greater than the preset threshold, the adjustment process of the three-dimensional labeling model includes:

9. The system for annotating vehicle key points of claim 8, wherein, The projection equation between the two-dimensional key points and the three-dimensional key points is acquired, and the projection relationship of the projection of the three-dimensional key point labeling result to the two-dimensional projection result is acquired according to the projection equation; The two-dimensional key point labeling result of each key point in the vehicle to be labeled and the three-dimensional key point labeling result obtained by the initial three-dimensional labeling model or the three-dimensional labeling model after the last adjustment are matched to acquire a plurality of key point pairs. ​ According to the projection equation and the plurality of key points, a key point with a labeling error is determined, and a correction value for updating a position of the key point with the labeling error is determined by using a least square method; According to the projection relationship and the correction value, a three-dimensional labeling model corresponding to an error value greater than a preset threshold is adjusted to obtain an adjusted three-dimensional labeling model.

10. The system for annotating vehicle key points of claim 7, wherein, The key points of the vehicle to be labeled include at least one of: a front roof left side upper edge corner point, a front headlamp left side upper edge outer corner point, a front chassis left side lower edge outer corner point, a front wheel left center point, a rear wheel left center point, a rear chassis left side corner point, a rear tail lamp left side upper edge outer corner point, a rear roof left side upper edge corner point, a front end of a lowermost edge of a left side window, a tail end of the lowermost edge of the left side window, a front roof right side upper edge corner point, a front headlamp right side upper edge outer corner point, a front chassis right side lower edge outer corner point, a front wheel right center point, a rear wheel right center point, a rear chassis right side corner point, a rear tail lamp right side upper edge outer corner point, a rear roof right side upper edge corner point, a front end of a lowermost edge of a right side window, and a tail end of the lowermost edge of the right side window.

11. The system for annotating vehicle key points of claim 7, wherein, When performing two-dimensional key point labeling or three-dimensional key point labeling on the vehicle to be labeled, the labeling categories include at least one of the following: visible, self-occluded, other-occluded, truncated, and unknown.

12. An apparatus for annotating key points of a vehicle, the apparatus comprising: comprise: one or more processors; and a computer readable medium storing instructions that, when executed by the one or more processors, cause the device to perform the method of any one of claims 1-6.

13. A computer readable medium characterized by having instructions stored thereon that, when executed by one or more processors, cause a device to perform the method of any one of claims 1-6.

Images