Image data target detection inclined frame automatic labeling method and system

Abstract

The application provides an image data target detection inclined frame automatic labeling method and system, wherein the labeling method comprises collecting target object images and related information of the target object and an image collector; calculating world coordinates of each vertex of a minimum external cuboid of the target object; determining pixel coordinates of each reference vertex of a minimum rectangular surrounding frame of each vertex of the minimum external cuboid of the target object in a pixel coordinate system according to the pixel coordinates of each vertex of the minimum external cuboid of the target object; calculating a side length of a target detection inclined frame and an inclination angle of the target detection inclined frame according to the pixel coordinates of each reference vertex of the minimum rectangular surrounding frame; and generating the target detection inclined frame based on the pixel coordinates of a center point of the target detection inclined frame, the side length and the inclination angle. The method and system can realize automatic generation of the target detection inclined frame with the inclination angle without human intervention, can simplify an operation process, and ensure labeling accuracy.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a method and system for automatic annotation of tilted bounding boxes in image data target detection. Background Technology

[0002] In existing technologies, supervised learning requires a large amount of labeled data, which can be done entirely manually, automatically by machines, or through a combination of human and machine methods. These three modes are respectively called manual labeling, automatic labeling, and semi-automatic labeling. For example, in autonomous driving, the perception system has a high demand for labeling, especially for obstacle perception. Automatic labeling mainly utilizes mutual labeling between different sensors. There are three main types of sensors for obstacle perception: LiDAR, cameras, and millimeter-wave radar. From the data flow of automatic labeling, it generally relies on LiDAR and millimeter-wave radar to label cameras. This automatic labeling method requires the cooperation of additional LiDAR or millimeter-wave radar to complete.

[0003] Deep learning-based object detection requires a large amount of labeled data to train deep neural networks. While raw data can be collected using cameras or webcams or downloaded from publicly available online sources, it cannot be used directly for training. Raw data must be labeled before it can be used to train an object detection model. Data labeling is a very tedious and time-consuming task, making deep learning research exceptionally costly.

[0004] Current research on automated data annotation mainly adopts the following approach: 1. Use a preliminary model to detect a small batch of unlabeled data. This preliminary model can be trained on a small dataset or a publicly available one; 2. Manually correct the detection results; 3. Train a new model using the corrected data; 4. Use the new model to detect a medium batch of unlabeled data; 5. Repeat steps 1-4. Through iterative iteration of steps 1-5, the data can be gradually refined. Although this greatly reduces the workload, it still requires manual intervention and multiple repetitions. The above approach is used to first train a small batch of data for annotation, then the learning results are used to annotate the remaining dataset, and finally, manual correction and iterative refinement are performed. This object detection data annotation method cannot completely eliminate manual intervention and is still a semi-automated data annotation method. Therefore, to achieve accurate automatic annotation of object detection data, it is first necessary to automatically generate accurate object detection boxes, and then further implement automatic labeling of object categories based on these boxes. Summary of the Invention

[0005] To overcome the aforementioned deficiencies of the prior art, the present invention aims to provide an automatic tilted bounding box annotation method and system for target detection in image data. This method and system address the problems of cumbersome and time-consuming processes requiring manual intervention in the prior art. To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended as a general commentary, nor is it intended to identify key / important components or describe the scope of protection of these embodiments. Its sole purpose is to present some concepts in a simple form as a prelude to the detailed description that follows.

[0006] To achieve the above-mentioned objectives, this invention proposes an automatic annotation method for tilted bounding boxes in image data target detection, which is improved by including the following steps:

[0007] Acquire images of the target object and obtain its length, width, and height, as well as the world coordinates of the target object's center point, the target object's pose information, the world coordinates of the image acquisition device, and the pose information of the image acquisition device;

[0008] Based on the length, width, and height of the target object, as well as the world coordinates of the target object's center point, the target object's pose information, the world coordinates of the image acquisition device, and the image acquisition device's pose information, the pixel coordinates of each vertex of the target object's smallest circumscribed cuboid are obtained through coordinate transformation.

[0009] Based on the pixel coordinates of each vertex of the minimum bounding box of the target object, determine the pixel coordinates of each reference vertex of the minimum bounding box of the target object in the pixel coordinate system.

[0010] The side length and tilt angle of the target detection tilt box are calculated based on the pixel coordinates of each reference vertex of the minimum rectangular bounding box. The target detection tilt box is then generated based on the obtained pixel coordinates of the center point of the target detection tilt box, the side length, and the tilt angle.

[0011] The target detection tilted box is used to label target objects in the image.

[0012] Preferably, the step of obtaining the pixel coordinates of each vertex of the smallest circumscribed cuboid of the target object through coordinate transformation based on the length, width, and height of the target object, as well as the world coordinates of the target object's center point, the target object's pose information, the world coordinates of the image acquisition device, and the image acquisition device's pose information, includes:

[0013] Based on the world coordinates of the center point of the target object, the attitude information of the target object, and the length, width, and height of the target object, calculate the world coordinates of each vertex of the minimum circumscribed cuboid of the target object;

[0014] Based on the world coordinates of each vertex of the minimum circumscribed cuboid of the target object, the world coordinates of the image acquisition device, and the pose information of the image acquisition device, the pixel coordinates of each vertex of the minimum circumscribed cuboid of the target object are obtained through coordinate transformation from the world coordinate system to the pixel coordinate system.

[0015] Preferably, the step of calculating the world coordinates of each vertex of the smallest circumscribed cuboid of the target object based on the world coordinates of the target object's center point, the target object's attitude information, and the target object's length, width, and height includes:

[0016] Construct a target object coordinate system with the center of the target object as the origin, and make the x, y, and z axes of the target object coordinate system parallel to the length, width, and height directions of the target object, respectively;

[0017] Calculate the coordinates of each vertex of the smallest circumscribed cuboid of the target object in the target object coordinate system based on the length, width, and height of the target object;

[0018] Based on the coordinates of each vertex of the minimum circumscribed cuboid of the target object in the target object coordinate system, the attitude information of the target object, and the world coordinates of the center point of the target object, the world coordinates of each vertex of the minimum circumscribed cuboid of the target object are obtained through coordinate transformation from the target object coordinate system to the world coordinate system.

[0019] Preferably, the step of calculating the coordinates of each vertex of the smallest circumscribed cuboid of the target object in the target object coordinate system based on the length, width, and height of the target object includes:

[0020] The coordinates of each vertex of the smallest circumscribed cuboid are calculated using the following formula:

[0021]

[0022]

[0023] In the formula, Let P be the coordinates of the i-th vertex of the smallest circumscribed cuboid in the target object coordinate system. ti The value of l; i is the number of vertices of the smallest circumscribed cuboid, i∈[1,8]; t w t h t These are the length, width, and height of the target object, respectively.

[0024] Preferably, the step of obtaining the world coordinates of each vertex of the smallest circumscribed cuboid of the target object based on the coordinates of each vertex of the target object in the target object coordinate system, the attitude information of the target object, and the world coordinates of the center point of the target object through a coordinate transformation from the target object coordinate system to the world coordinate system includes:

[0025] The world coordinates of each vertex of the smallest bounding cuboid of the target object in the world coordinate system are calculated using the following formula:

[0026]

[0027]

[0028]

[0029] In the formula, Let P be the world coordinates of the i-th vertex of the minimum circumscribed cuboid. wi The value of T, i∈[1,8]; t It is a translation vector; The world coordinates P of the center point of the target object w The value of θ; t , ω t These are the target's attitude information, including yaw, pitch, and roll angles; R zt R yt R xt Rotation θ around the Z-axis t The first rotation matrix obtained, rotated around the Y-axis The obtained second rotation matrix and rotation ω around the X-axis t The resulting third rotation matrix; R t For R zt R yt R xt The rotation matrix obtained after superposition; R t -1 For R t The inverse matrix.

[0030] Preferably, the step of obtaining the pixel coordinates of each vertex of the minimum circumscribed cuboid of the target object through coordinate transformation from the world coordinate system to the pixel coordinate system based on the world coordinates of each vertex of the minimum circumscribed cuboid of the target object, the world coordinates of the image acquisition device, and the pose information of the image acquisition device includes:

[0031] Based on the world coordinates of each vertex of the minimum circumscribed cuboid of the target object, the world coordinates of the image acquisition device, and the attitude information of the image acquisition device, the pixel coordinates of each vertex of the minimum circumscribed cuboid of the target object in the pixel coordinate system are obtained by sequentially transforming the coordinates from the world coordinate system to the image acquisition device coordinate system, from the image acquisition device coordinate system to the image coordinate system, and from the image coordinate system to the pixel coordinate system.

[0032] Preferably, the attitude information of the image acquisition device includes the yaw angle, pitch angle, and roll angle of the image acquisition device.

[0033] Preferably, determining the pixel coordinates of each reference vertex of the minimum bounding box of the minimum bounding box of the target object in the pixel coordinate system based on the pixel coordinates of each vertex of the minimum bounding box of the target object includes:

[0034] Based on the pixel coordinates of each vertex of the minimum bounding cuboid of the target object, the convex hull algorithm is used to construct the convex hull of each vertex of the minimum bounding cuboid of the target object in the pixel coordinate system.

[0035] The pixel coordinates of each reference vertex of the minimum bounding box of the minimum circumscribed cuboid of the target object are calculated based on the convex hull.

[0036] Preferably, the step of constructing the convex hull of each vertex of the minimum bounding cuboid of the target object in the pixel coordinate system using the convex hull algorithm based on the pixel coordinates of each vertex includes:

[0037] Select the minimum or maximum pixel coordinates of each vertex of the smallest circumscribed cuboid of the target object in the direction of the horizontal or vertical axis of the pixel coordinate system as the initial vertices;

[0038] Connect the initial vertex to each of the remaining vertices to obtain corresponding line segments. Select a line segment that satisfies the constraints, and use another vertex on that line segment other than the initial point as the initial point. Repeat this step until the selected vertices are connected one by one to form a polygon that can enclose all the vertices of the minimum circumscribed cuboid. The polygon then serves as the convex hull of each vertex of the minimum circumscribed cuboid in the pixel coordinate system.

[0039] Preferably, the constraint condition includes: all line segments other than the selected line segment are located on the same side of the selected line segment.

[0040] Preferably, the step of calculating the pixel coordinates of each reference vertex of the minimum bounding box of the minimum circumscribed cuboid of the target object in the pixel coordinate system based on the convex hull includes:

[0041] Construct the corresponding rectangle for each edge of the convex hull;

[0042] Compare the areas of all rectangles, select the rectangle with the smallest area as the smallest bounding box of the target object's smallest circumscribed cuboid in pixel coordinates, and use the pixel coordinates of the vertices of the smallest rectangle as the pixel coordinates of the reference vertices of the smallest bounding box.

[0043] Preferably, the step of constructing a corresponding rectangle based on each edge of the convex hull includes:

[0044] Draw the first straight line passing through the two vertices of an edge of the convex hull;

[0045] Project all vertices except the two vertices on the first line onto the first line, and draw the second and third lines perpendicular to the first line along the two vertices with the farthest projection distance.

[0046] Find the vertex in the convex hull that is farthest from the first line, and draw a fourth line through that vertex that is parallel to the first line;

[0047] Using the four points where the first to fourth lines intersect as the vertices of the rectangle, we obtain the rectangle corresponding to the convex hull of that edge.

[0048] Preferably, the step of calculating the side length of the target detection tilt box based on the pixel coordinates of each reference vertex of the minimum rectangular bounding box includes:

[0049] The pixel coordinates of any vertex of the minimum rectangular bounding box are used as the reference point;

[0050] Select the two reference vertices with shorter straight-line distances from the reference point from the remaining three reference vertices, and use the straight-line distance between the reference point and the two reference vertices as the lengths of the two sides that intersect the reference point in the minimum rectangular bounding box.

[0051] If the two sides intersecting at the reference point in the minimum bounding box are of different lengths, then the shortest side is taken as the width of the target detection tilt box, and the longest side is taken as the length of the target detection tilt box.

[0052] If the two sides of the smallest bounding rectangle that intersect at the reference point have equal lengths, then either side length will be used as the side length of each side of the target detection tilt box.

[0053] Preferably, the method for calculating the tilt angle of the target detection tilt box includes:

[0054] If the two sides intersecting at the reference point in the minimum bounding box are of unequal length, the tilt angle between the target detection tilt box and the horizontal axis of the pixel coordinate system is calculated based on the projection length of the longest side onto the vertical and horizontal axes of the pixel coordinate system.

[0055] If the two sides intersecting at the reference point within the smallest bounding rectangle are of equal length, then the tilt angle between the target detection tilt box and the horizontal axis of the pixel coordinate system is calculated based on the projection length of either side length onto the vertical and horizontal axes of the pixel coordinate system.

[0056] Preferably, the method for obtaining the pixel coordinates of the center point of the target detection tilted bounding box includes:

[0057] The pixel coordinates of the center point of the target detection tilt box are obtained by averaging the pixel coordinates of the reference vertices of the smallest bounding box.

[0058] Preferably, the method for obtaining the pixel coordinates of the center point of the target detection tilted bounding box includes:

[0059] Based on the world coordinates of the target object's center point, the world coordinates of the image acquisition device, and the attitude information of the image acquisition device, the pixel coordinates of the target object's center point in the pixel coordinate system are obtained by sequentially transforming the coordinates from the world coordinate system to the image acquisition device coordinate system, from the image acquisition device coordinate system to the image coordinate system, and from the image coordinate system to the pixel coordinate system.

[0060] Use the pixel coordinates of the center point of the target object as the pixel coordinates of the center point of the target detection tilt box.

[0061] Secondly, embodiments of the present invention also propose an automatic annotation system for tilted bounding boxes in image data target detection, which is improved by including the following components:

[0062] Image acquisition device, used to acquire images of target objects;

[0063] The sensor is used to acquire the length, width, and height of the target object, as well as the world coordinates of the target object's center point, the target object's attitude information, the world coordinates of the image acquisition device, and the attitude information of the image acquisition device.

[0064] The first calculator is used to obtain the pixel coordinates of each vertex of the smallest bounding cube of the target object through coordinate transformation based on the length, width and height of the target object, as well as the world coordinates of the center point of the target object, the pose information of the target object, the world coordinates of the image acquisition device and the pose information of the image acquisition device.

[0065] The second calculator is used to determine the pixel coordinates of each reference vertex of the minimum bounding box of the minimum bounding box of the target object in the pixel coordinate system based on the pixel coordinates of each vertex of the minimum bounding box of the target object.

[0066] A generator is used to calculate the side length and tilt angle of the target detection tilt box based on the pixel coordinates of each reference vertex of the minimum rectangular bounding box, and to generate the target detection tilt box based on the obtained pixel coordinates of the center point of the target detection tilt box, the side length and the tilt angle.

[0067] The target detection tilted box is used to label target objects in the image.

[0068] Thirdly, this invention also proposes an automatic annotation method for tilted bounding boxes in image data target detection, which is improved by including the following steps:

[0069] Generate the target detection tilt box in the image using any of the methods described above;

[0070] Obtain the object category in the image, and add category labels to the pixel coordinates of the center point of the object detection tilt box, the side length of the object detection tilt box, and the tilt angle according to the object category to obtain the final annotation data of the object in the image.

[0071] Fourthly, embodiments of the present invention also propose an automatic annotation system for tilted bounding boxes in image data target detection, the improvement of which is that it includes the following components:

[0072] A generating apparatus for generating a target detection tilted bounding box in an image by performing any of the methods described above;

[0073] The annotation device is used to obtain the category of the target object in the image, and add category labels to the pixel coordinates of the center point of the target detection tilt box, the side length of the target detection tilt box and the tilt angle according to the target object category, so as to obtain the final annotation data of the target object in the image.

[0074] Compared with the closest existing technology, the technical solution provided by the present invention has the following beneficial effects:

[0075] This invention provides an automatic annotation method and system for target detection tilted bounding boxes in image data. By using relevant information about the target object and the image acquisition device, the pixel coordinates of each vertex of the target object's smallest bounding box are obtained through coordinate transformation. Based on the pixel coordinates of each vertex of the target object's smallest bounding box, the pixel coordinates of each reference vertex of the smallest bounding box in the pixel coordinate system are determined. The side length and tilt angle of the target detection tilted bounding box are calculated based on the pixel coordinates of the reference vertices of the smallest bounding box. The target detection tilted bounding box is then generated based on the obtained pixel coordinates of the center point of the target detection tilted bounding box, the side length, and the tilt angle. This method and system can truly achieve automated generation of target detection tilted bounding boxes with tilt angles without human intervention; it also simplifies the operation process and ensures the accuracy of annotation.

[0076] The present invention provides an automatic annotation method and system for target detection of image data with tilted bounding boxes. The method uses the convex hull algorithm to calculate the pixel coordinates of each reference vertex of the minimum bounding box of the minimum circumscribed cuboid of the target object in the pixel coordinate system. The principle is simple and easy to implement.

[0077] The present invention provides an automatic labeling method and system for target detection tilted boxes in image data. In addition to acquiring images with target objects, it only needs to obtain the length, width and height of the target object in the image, the world coordinates of the target object's center point, the target object's attitude information, the world coordinates of the image acquisition device and the attitude information of the image acquisition device. It does not require the configuration of complex lidar and millimeter-wave radar, which simplifies the initial configuration work and saves costs.

[0078] This invention provides an automatic annotation method and system for target detection tilted bounding boxes in image data. First, tilted bounding boxes for target detection in the image are automatically generated using the aforementioned generation method. Then, category labels are added based on the target object category, including the pixel coordinates of the center point of the tilted bounding box, the side length of the tilted bounding box, and the tilt angle, to obtain the final annotation data for the target object in the image. This method and system truly achieve automated annotation of target detection data without human intervention.

[0079] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0080] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0081] Figure 1 This is a flowchart illustrating the automatic annotation method for tilted bounding boxes in image data target detection according to an embodiment of the present invention;

[0082] Figure 2 This is a schematic diagram of the target detection tilted box generated in the image in an embodiment of the present invention;

[0083] Figure 3 This is a schematic diagram of coordinate transformation from the world coordinate system to the camera coordinate system in an embodiment of the present invention;

[0084] Figure 4 This is a schematic diagram of coordinate transformation from camera coordinate system to image coordinate system in an embodiment of the present invention;

[0085] Figure 5 This is a schematic diagram of coordinate transformation from image coordinate system to pixel coordinate system in an embodiment of the present invention;

[0086] Figure 6 This is a flowchart illustrating another method for automatically labeling tilted bounding boxes in image data target detection according to an embodiment of the present invention. Detailed Implementation

[0087] The method for generating tilted bounding boxes for target detection, the method for automatically labeling image data, and the system provided in the embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0088] Example 1

[0089] The image data target detection tilted bounding box automatic annotation method (i.e. target detection tilted bounding box generation method) provided in Embodiment 1 of the present invention, such as... Figure 1As shown, the specific steps include the following:

[0090] S11. Acquire the image of the target object, and obtain the length, width and height of the target object, as well as the world coordinates of the center point of the target object, the pose information of the target object, the world coordinates of the image acquisition device and the pose information of the image acquisition device;

[0091] S12. Based on the length, width, and height of the target object, as well as the world coordinates of the target object's center point, the target object's pose information, the world coordinates of the image acquisition device, and the image acquisition device's pose information, obtain the pixel coordinates of each vertex of the target object's smallest circumscribed cuboid through coordinate transformation.

[0092] S13. Based on the pixel coordinates of each vertex of the minimum bounding box of the target object, determine the pixel coordinates of each reference vertex of the minimum bounding box of the target object in the pixel coordinate system.

[0093] S14. Calculate the side length and tilt angle of the target detection tilt box based on the pixel coordinates of each reference vertex of the minimum rectangular bounding box, and generate the target detection tilt box based on the obtained pixel coordinates of the center point of the target detection tilt box, the side length and the tilt angle.

[0094] The target detection tilted box is used to label target objects in the image.

[0095] The image acquisition device in this embodiment can be any image acquisition device known to those skilled in the art. In this embodiment, a camera is used as an example for detailed description.

[0096] The specific solutions in S11 above may include:

[0097] Point the camera towards the target object so that the target object appears in the camera's frame, and collect image data containing the target object;

[0098] While the camera is acquiring image data, the length l of the target object is acquired by the sensor. t Width w t and height h t And the world coordinates P of the target's center point in the pre-constructed world coordinate system. w The target's attitude information and the camera's world coordinates O c And camera pose information;

[0099] In this embodiment, the world coordinate system can be a three-dimensional rectangular coordinate system, preferably a northeast-northeast coordinate system;

[0100] The target's attitude information may include the target's yaw angle θ. t Pitch angle and roll angle ωt The camera's attitude information can include the camera's yaw angle θ and pitch angle. And the roll angle ω, the world coordinates P of the target object's center point w The value is [x w y w z w ] T The world coordinates of the camera O c The value is [x o y o z o ] T .

[0101] The specific solutions in S12 above may include:

[0102] S121. Based on the world coordinates of the center point of the target object, the attitude information of the target object, and the length, width, and height of the target object, calculate the world coordinates of each vertex of the smallest circumscribed cuboid of the target object.

[0103] S122. Based on the world coordinates of each vertex of the minimum circumscribed cuboid of the target object, the world coordinates of the image acquisition device, and the pose information of the image acquisition device, the pixel coordinates of each vertex of the minimum circumscribed cuboid of the target object are obtained through coordinate transformation from the world coordinate system to the pixel coordinate system.

[0104] An optional solution in Embodiment 1 of the present invention is that, in S121 above, calculating the world coordinates of each vertex of the smallest circumscribed cuboid of the target object may include the following steps:

[0105] S1211. Construct a target object coordinate system with the center of the target object as the origin of the target object coordinate system, and make the x, y and z axes of the target object coordinate system parallel to the length, width and height directions of the target object, respectively.

[0106] S1212. Calculate the coordinates of each vertex of the smallest circumscribed cuboid of the target object in the target object coordinate system based on the length, width and height of the target object;

[0107] S1213. Based on the coordinates of each vertex of the smallest circumscribed cuboid of the target object in the target object coordinate system, the attitude information of the target object, and the world coordinates of the center point of the target object, the world coordinates of each vertex of the smallest circumscribed cuboid of the target object in the world coordinate system are obtained through coordinate transformation from the target object coordinate system to the world coordinate system.

[0108] An optional solution in Embodiment 1 of the present invention is that, in S1212 above, the coordinates of each vertex of the smallest circumscribed cuboid of the target object in the target object coordinate system can be calculated by the following formula:

[0109]

[0110]

[0111] In the formula, Let P be the coordinates of the i-th vertex of the smallest circumscribed cuboid in the target object coordinate system. ti The value of ; i is the number of vertices of the smallest circumscribed cuboid; i∈[1,8].

[0112] An optional solution in Embodiment 1 of the present invention is that, in S1213 above, the world coordinates of each vertex of the smallest circumscribed cuboid of the target object in the world coordinate system can be calculated by the following formula:

[0113]

[0114]

[0115]

[0116] In the formula, Let P be the world coordinates of the i-th vertex of the minimum circumscribed cuboid. wi The value of T; t It is a translation vector; The world coordinates P of the center point of the target object w The value of θ; t , ω t These are the target's yaw angle, pitch angle, and roll angle, respectively; R zt R yt R xt Rotation θ around the Z-axis t The first rotation matrix obtained, rotated around the Y-axis The obtained second rotation matrix and rotation ω around the X-axis t The resulting third rotation matrix; R t For R zt R yt R xt The rotation matrix obtained after superposition; R t -1 For R t The inverse matrix.

[0117] For example, in S121 above, based on the world coordinates P of the target object's center point... w =[x w y w z w ] T The length of the target object l t Width w t and height h t The target's attitude angle (yaw angle θ) t Pitch angle and roll angle ω t), calculate the world coordinates P of the eight vertices of the smallest circumscribed cuboid of the target object. w1 =[x w1 y w1 z w1 ] T P w2 =[x w2 y w2 z w2 ] T P w3 =[x w3 y w3 z w3 ] T P w4 =[x w4 y w4 z w4 ] T P w5 =[x w5 y w5 z w5 ] T P w6 =[x w6 y w6 z w6 ] T P w7 =[x w7 y w7 z w7 ] T P w8 =[x w8 y w8 z w8 ] T The specific methods are as follows:

[0118] In the target object coordinate system, the center of the target object is the origin, and the coordinates of the eight vertices are as follows:

[0119]

[0120] The coordinate transformation from the world coordinate system to the target object coordinate system is as follows:

[0121]

[0122] Where R t Let T be the rotation matrix. t It is a translation vector.

[0123] Rotate θ around the Z-axis t Rotation matrix

[0124]

[0125] Rotate around the Y-axis Rotation matrix

[0126]

[0127] Rotate ω around the X-axis t Rotation matrix

[0128]

[0129] R t =R zt R yt R xt ,.................................................(5)

[0130]

[0131] From equations (1)-(6), the coordinate transformation from the target coordinate system to the world coordinate system can be obtained as follows:

[0132]

[0133] Therefore, the world coordinates of the eight vertices of the smallest circumscribed cuboid of the target object are:

[0134]

[0135]

[0136]

[0137]

[0138]

[0139]

[0140]

[0141]

[0142] An optional solution in Embodiment 1 of the present invention is that obtaining the pixel coordinates of each vertex of the smallest circumscribed cuboid of the target object in S122 above may include the following steps:

[0143] Based on the world coordinates of each vertex of the minimum circumscribed cuboid of the target object, the world coordinates of the image acquisition device, and the attitude information of the image acquisition device, the pixel coordinates of each vertex of the minimum circumscribed cuboid of the target object in the pixel coordinate system are obtained by sequentially transforming the coordinates from the world coordinate system to the image acquisition device coordinate system (i.e., the camera coordinate system in this embodiment), from the image acquisition device coordinate system to the image coordinate system, and from the image coordinate system to the pixel coordinate system.

[0144] An optional embodiment of the present invention is that, in step S122 above, the following steps may also be included:

[0145] S1221. Based on the world coordinates of each vertex of the minimum circumscribed cuboid of the target object, the world coordinates of the camera, and the attitude information of the camera, the coordinates of each vertex of the minimum circumscribed cuboid of the target object in the camera coordinate system are obtained through coordinate transformation from the world coordinate system to the camera coordinate system.

[0146] S1222. Based on the camera's internal parameters, obtain the coordinates of each vertex of the smallest circumscribed cuboid of the target object in the image coordinate system through coordinate transformation from the camera coordinate system to the image coordinate system.

[0147] Among these, the camera's internal parameters may include the camera's focal length;

[0148] S1223. Based on the coordinates of each vertex of the smallest circumscribed cuboid of the target object in the image coordinate system, obtain the pixel coordinates of each vertex of the smallest circumscribed cuboid of the target object in the pixel coordinate system through coordinate transformation from the image coordinate system to the pixel coordinate system.

[0149] An optional solution in Embodiment 1 of the present invention may include the following steps in S13:

[0150] S131. Based on the pixel coordinates of each vertex of the minimum bounding cuboid of the target object, construct the convex hull of each vertex of the minimum bounding cuboid of the target object in the pixel coordinate system using the convex hull algorithm.

[0151] S132. Calculate the pixel coordinates of each reference vertex of the minimum bounding box of the minimum circumscribed cuboid of the target object in the pixel coordinate system based on the convex hull.

[0152] An optional solution in Embodiment 1 of the present invention may include the following steps in S131:

[0153] S1311. Select the minimum or maximum pixel coordinates of each vertex of the smallest circumscribed cuboid of the target object in the direction of the horizontal or vertical axis of the pixel coordinate system as the initial vertex.

[0154] S1312. Connect the initial vertex to each of the remaining vertices to obtain corresponding line segments. Select a line segment that satisfies the constraint conditions. Take the other vertex on the line segment other than the initial point as the initial point and repeat this step until the selected vertices are connected one by one to form a polygon that can enclose all the vertices of the minimum circumscribed cuboid. The polygon is then used as the convex hull of each vertex of the minimum circumscribed cuboid of the target object in the pixel coordinate system.

[0155] The constraints include: all line segments except the selected line segment must be located on the same side of the selected line segment.

[0156] For example, the construction of the convex hull of each vertex of the minimum bounding cuboid of the target object in the pixel coordinate system in S131 above may also include the following steps:

[0157] S1311' Select the minimum or maximum value of the horizontal or vertical axis of the 8 vertices in the pixel coordinate system as the initial point;

[0158] S1312': Connect the initial point with the remaining 7 points to obtain 7 line segments. Select one line segment such that all other 6 points are located to the left or right of the line segment. The other point on the selected line segment besides the initial point is the second point.

[0159] S1313', Take the second point as the initial point and connect it with all the other 7 points to obtain 7 line segments. Select a line segment such that all other 6 points are located to the left or right of the line segment (the left and right sides need to be consistent with the first point). The other point on the selected line segment other than the second point is the third point.

[0160] S1314' Repeat the above steps with the third point as the initial point until the last selected point becomes the first point;

[0161] S1315', The selected line segment forms the convex hull of the target object's minimum bounding cuboid with 8 vertices in pixel coordinates.

[0162] An optional solution in Embodiment 1 of the present invention may include the following steps in S132:

[0163] S1321. Construct the corresponding rectangle based on each edge of the convex hull;

[0164] S1322. Compare the areas of all rectangles, select the rectangle with the smallest area as the smallest bounding box of the target object in pixel coordinates, and use the pixel coordinates of the vertices of the smallest rectangle as the pixel coordinates of the reference vertices of the smallest bounding box.

[0165] An optional solution in Embodiment 1 of the present invention may include the following steps in S1321:

[0166] S13211. Draw the first straight line passing through the two vertices of an edge of the convex hull;

[0167] S13212. Project all vertices except the two vertices on the first line onto the first line, and draw a second line and a third line perpendicular to the first line along the two vertices with the farthest projection distance.

[0168] S13213. Find the vertex in the convex hull that is farthest from the first line, and draw a fourth line through that vertex that is parallel to the first line.

[0169] S13214. Take the four points where the first to fourth straight lines intersect as the vertices of the rectangle, and obtain the rectangle corresponding to the convex hull of that side.

[0170] For example, in S132 above, the following operation can also be performed on each edge of the convex hull first:

[0171] S13211', Draw the first straight line passing through the two endpoints of this edge;

[0172] S13212' Project all points onto the straight line, and draw a second and a third straight line perpendicular to the straight line through the two points with the farthest projection distance from the straight line;

[0173] S13213' Find the point farthest from the edge, and draw the fourth straight line parallel to the current edge through that point;

[0174] S13214': Using the four points where the above four lines intersect as vertices, we obtain the rectangle corresponding to the side of the convex hull.

[0175] Then compare the areas of the rectangles obtained by S13211'-S13214' for each side of the convex hull, and select the one with the smallest area as the smallest bounding rectangle of the target object's 8 vertices in the pixel coordinate system.

[0176] An optional solution in Embodiment 1 of the present invention is that the calculation of the side length of the target detection tilt box based on the pixel coordinates of each reference vertex of the minimum rectangular bounding box in S14 may include the following steps:

[0177] The pixel coordinates of any vertex of the minimum rectangular bounding box are used as the reference point;

[0178] Select the two reference vertices with shorter straight-line distances from the reference point from the remaining three reference vertices, and use the straight-line distance between the reference point and the two reference vertices as the lengths of the two sides that intersect the reference point in the minimum rectangular bounding box.

[0179] If the two sides of the minimum bounding box that intersect at the reference point are of different lengths, then the shortest side is taken as the width of the target detection tilt box and the longest side is taken as the length of the target detection tilt box, thereby determining the four side lengths of the minimum bounding box.

[0180] If the two sides of the smallest bounding rectangle that intersect at the reference point have equal lengths, then either side length will be used as the side length of each side of the target detection tilt box.

[0181] An optional solution in Embodiment 1 of the present invention is that, in step S14 above, calculating the tilt angle of the target detection tilt box based on the pixel coordinates of each reference vertex of the minimum rectangular bounding box can include the following steps:

[0182] If the two sides intersecting at the reference point in the minimum bounding box are of unequal length, the tilt angle between the target detection tilt box and the horizontal axis of the pixel coordinate system is calculated based on the projection length of the longest side onto the vertical and horizontal axes of the pixel coordinate system.

[0183] If the two sides intersecting at the reference point within the smallest bounding rectangle are of equal length, then the tilt angle between the target detection tilt box and the horizontal axis of the pixel coordinate system is calculated based on the projection length of either side length onto the vertical and horizontal axes of the pixel coordinate system.

[0184] An optional solution in Embodiment 1 of the present invention is that, in S14 above, the target detection tilted bounding box is generated based on the pixel coordinates of the center point of the acquired target detection tilted bounding box, the side length, and the tilt angle, and any of the following solutions can be adopted:

[0185] The first approach includes the following steps:

[0186] S1411. Obtain the pixel coordinates of the center point of the target detection tilt box based on the average pixel coordinates of each reference vertex of the minimum rectangular bounding box.

[0187] S1412. Generate a target detection tilted box based on the pixel coordinates of the center point of the target detection tilted box, the side length of the target detection tilted box, and the tilt angle between the target detection tilted box and the horizontal axis of the pixel coordinate system.

[0188] The second approach includes the following steps:

[0189] S1421. Based on the world coordinates of the target object's center point, the world coordinates of the image acquisition device, and the pose information of the image acquisition device, the pixel coordinates of the target object's center point are used as the pixel coordinates of the target detection tilt box center point through coordinate transformation from the world coordinate system to the pixel coordinate system.

[0190] S1422. Generate a target detection tilted box based on the pixel coordinates of the center point of the target detection tilted box, the side length of the target detection tilted box, and the tilt angle between the target detection tilted box and the horizontal axis of the pixel coordinate system.

[0191] An optional solution in Embodiment 1 of the present invention may include the following steps in S1421:

[0192] Based on the world coordinates of the target object's center point, the world coordinates of the image acquisition device, and the attitude information of the image acquisition device, the pixel coordinates of the target object's center point in the pixel coordinate system are obtained by sequentially transforming the coordinates from the world coordinate system to the image acquisition device coordinate system, from the image acquisition device coordinate system to the image coordinate system, and from the image coordinate system to the pixel coordinate system.

[0193] Preferably, the attitude information of the image acquisition device includes the yaw angle, pitch angle, and roll angle of the image acquisition device.

[0194] An optional embodiment of the present invention is that, in step S1421 above, the following steps may also be included:

[0195] S14211. Based on the world coordinates of the target object's center point, the camera's world coordinates, and the camera's attitude information, obtain the coordinates of the target object's center point in the camera coordinate system through coordinate transformation from the world coordinate system to the camera coordinate system.

[0196] S14212. Based on the camera's internal parameters, obtain the coordinates of the target's center point in the image coordinate system through coordinate transformation from the camera coordinate system to the image coordinate system;

[0197] Among these, the camera's internal parameters may include the camera's focal length;

[0198] S14213. Based on the coordinates of the center point of the target object in the image coordinate system, obtain the pixel coordinates of the center point of the target object in the pixel coordinate system through coordinate transformation from the image coordinate system to the pixel coordinate system.

[0199] For example, in S13-S14 above, the minimum bounding rectangle of the 8 vertices of the minimum bounding cuboid obtained in S12 is calculated, such as... Figure 2 As shown, A i Let i = 1, 2...8 be the positions of the eight vertices of the bounding rectangle of the target object obtained by S13 on the image. Starting from A1, connect A1 with A2, A3, A4, A5, A6, A7, and A8 to obtain A1A2, A1A3, A1A4, A1A5, A1A6, A1A7, and A1A8. Since A3, A4, A5, A6, A7, and A8 are all located to the left of edge A1A2, A1A2 is the selected first edge. Starting from A2, similarly, we can obtain A2A4, and similarly, A4A5, A5A6, A6A7, and A7A1. A1A2A4A5A6A7 is A... i The convex hull of i = 1, 2...8.

[0200] Obtain line l1 through A1 and A2; using A1A2 as the reference side, find the point farthest from A6, and draw line l2 parallel to A1A2 through A6. i All vertices i = 1, 2...8 are projected onto A1A2. The two vertices with the farthest projection distance are A4 and A7. Lines l3 and l4, perpendicular to A1A2, are drawn through A4 and A7 respectively. The lines l1, l2, l3, and l4 intersect at points R1, R2, R3, and R4. R1R2R3R4 forms the circumscribed rectangle corresponding to the edge of A1A2 on the convex hull. Similarly, the circumscribed rectangles corresponding to A2A4, A4A5, A5A6, A6A7, and A7A1 can be obtained. The rectangle with the smallest area is taken as A... i Find the smallest bounding rectangle for i = 1, 2, ..., 8.

[0201] Let R1R2R3R4 be A i The smallest bounding rectangle for i = 1, 2, ..., 8 is given by l = max{R1R2 R2R3} and w = min{R1R2R2R3}. Let l be the angle between side l and axis U, where l v Let l be the projection length of l onto the V-axis. u Let l be the projected length along the U-axis;

[0202] The pixel coordinates R = [uv] of the center point of the smallest rectangular bounding box of the target object in the image. T Where u = (u1 + u2 + u3 + u4) / 4, v = (v1 + v2 + v3 + v4) / 4, and combining the minimum bounding box length l, width w, and tilt angle of the minimum bounding box obtained above, the final target detection tilt box can be obtained.

[0203] Based on the same inventive concept, the image data target detection tilted bounding box automatic annotation system (i.e., target detection tilted bounding box generation system) provided in Embodiment 1 of this invention solves problems in a principle similar to the aforementioned method. Therefore, the functions, structures, and parameters of each component in the system can be found in the implementation of the aforementioned method, and repeated details will not be repeated. The functions of each component are described below:

[0204] Image acquisition unit 21 is used to acquire images of the target object;

[0205] Sensor 22 is used to acquire the length, width and height of the target object, as well as the world coordinates of the center point of the target object, the attitude information of the target object, the world coordinates of the image acquisition device and the attitude information of the image acquisition device;

[0206] The first calculator 23 is used to obtain the pixel coordinates of each vertex of the smallest bounding cuboid of the target object through coordinate transformation based on the length, width and height of the target object, as well as the world coordinates of the center point of the target object, the pose information of the target object, the world coordinates of the image acquisition device and the pose information of the image acquisition device.

[0207] The second calculator 24 is used to determine the pixel coordinates of each reference vertex of the minimum bounding box of the minimum bounding box of the target object in the pixel coordinate system based on the pixel coordinates of each vertex of the minimum bounding box of the target object.

[0208] Generator 25 is used to calculate the side length of the target detection tilt box and the tilt angle of the target detection tilt box based on the pixel coordinates of each reference vertex of the minimum rectangular bounding box, and to generate the target detection tilt box based on the obtained pixel coordinates of the center point of the target detection tilt box, the side length and the tilt angle.

[0209] The target detection tilted box is used to label target objects in the image.

[0210] Preferably, the first calculator 23 may include the following components:

[0211] The first world coordinate calculation module 231 is used to calculate the world coordinates of each vertex of the smallest circumscribed cuboid of the target object based on the world coordinates of the center point of the target object, the attitude information of the target object, and the length, width and height of the target object.

[0212] The first pixel coordinate calculation module 232 is used to obtain the pixel coordinates of each vertex of the minimum circumscribed cuboid of the target object by transforming the world coordinate system to the pixel coordinate system based on the world coordinates of each vertex of the minimum circumscribed cuboid of the target object, the world coordinates of the image acquisition device, and the posture information of the image acquisition device.

[0213] Preferably, the first world coordinate calculation module 231 may include the following components:

[0214] Construct submodule 2311 to construct the target object coordinate system with the center of the target object as the origin of the target object coordinate system, and make the x, y and z axes of the target object coordinate system parallel to the length, width and height directions of the target object, respectively;

[0215] The first calculation submodule 2312 is used to calculate the coordinates of each vertex of the smallest circumscribed cuboid of the target object in the target object coordinate system based on the length, width and height of the target object;

[0216] The second calculation submodule 2313 is used to obtain the world coordinates of each vertex of the minimum circumscribed cuboid of the target object by means of coordinate transformation from the target object coordinate system to the world coordinate system, based on the coordinates of each vertex of the minimum circumscribed cuboid of the target object in the target object coordinate system, the attitude information of the target object, and the world coordinates of the center point of the target object.

[0217] Preferably, the first calculation submodule 2312 is specifically used for:

[0218] The coordinates of each vertex of the smallest circumscribed cuboid of the target object in the target object coordinate system are calculated using the following formula:

[0219]

[0220]

[0221] In the formula, Let P be the coordinates of the i-th vertex of the smallest circumscribed cuboid in the target object coordinate system. ti The value of ; i is the number of vertices of the smallest circumscribed cuboid; i∈[1,8].

[0222] Preferably, the second calculation submodule 2313 is specifically used for:

[0223] The world coordinates of each vertex of the smallest bounding cuboid of the target object in the world coordinate system are calculated using the following formula:

[0224]

[0225]

[0226]

[0227] In the formula, Let P be the world coordinates of the i-th vertex of the minimum circumscribed cuboid. wi The value of T; t It is a translation vector; The world coordinates P of the center point of the target object w The value of θ; t , ω t These are the target's attitude information, including yaw, pitch, and roll angles; R zt R yt R xt Rotation θ around the Z-axis t The first rotation matrix obtained, rotated around the Y-axis The obtained second rotation matrix and rotation ω around the X-axis t The resulting third rotation matrix; R t For R zt R yt R xt The rotation matrix obtained after superposition; R t -1 For R t The inverse matrix.

[0228] Prior to this, the first pixel coordinate calculation module 232 is specifically used for:

[0229] Based on the world coordinates of each vertex of the minimum circumscribed cuboid of the target object, the world coordinates of the image acquisition device, and the attitude information of the image acquisition device, the pixel coordinates of each vertex of the minimum circumscribed cuboid of the target object in the pixel coordinate system are obtained by sequentially transforming the coordinates from the world coordinate system to the image acquisition device coordinate system, from the image acquisition device coordinate system to the image coordinate system, and from the image coordinate system to the pixel coordinate system.

[0230] Preferably, the second calculator 24 may include the following components:

[0231] Module 241 is used to construct the convex hull of each vertex of the minimum bounding cuboid of the target object in the pixel coordinate system based on the pixel coordinates of each vertex of the minimum bounding cuboid of the target object using the convex hull algorithm.

[0232] The second pixel coordinate calculation module 242 is used to calculate the pixel coordinates of each reference vertex of the minimum bounding box of the target object's minimum circumscribed cuboid in the pixel coordinate system based on the convex hull.

[0233] Preferably, the construction module 241 is specifically used for:

[0234] Select the minimum or maximum pixel coordinates of each vertex of the smallest circumscribed cuboid of the target object in the direction of the horizontal or vertical axis of the pixel coordinate system as the initial vertices;

[0235] Connect the initial vertex to each of the remaining vertices to obtain corresponding line segments. Select a line segment that satisfies the constraints, and use another vertex on that line segment other than the initial point as the initial point. Repeat this step until the selected vertices are connected one by one to form a polygon that can enclose all the vertices of the minimum bounding cuboid. The polygon then serves as the convex hull of each vertex of the minimum bounding cuboid of the target object in the pixel coordinate system.

[0236] The constraints may include: all line segments other than the selected line segment are located on the same side of the selected line segment.

[0237] Preferably, the second pixel coordinate calculation module 242 is specifically used for:

[0238] Construct the corresponding rectangle for each edge of the convex hull;

[0239] Compare the areas of all rectangles, select the rectangle with the smallest area as the smallest bounding box of the target object's smallest circumscribed cuboid in pixel coordinates, and use the pixel coordinates of the vertices of the smallest rectangle as the pixel coordinates of the reference vertices of the smallest bounding box.

[0240] Preferably, a corresponding rectangle is constructed based on each edge of the convex hull, including:

[0241] Draw the first straight line passing through the two vertices of an edge of the convex hull;

[0242] Project all vertices except the two vertices on the first line onto the first line, and draw the second and third lines perpendicular to the first line along the two vertices with the farthest projection distance.

[0243] Find the vertex in the convex hull that is farthest from the first line, and draw a fourth line through that vertex that is parallel to the first line;

[0244] Using the four points where the first to fourth lines intersect as the vertices of the rectangle, we obtain the rectangle corresponding to the convex hull of that edge.

[0245] Preferably, generator 25 may include the following components:

[0246] The side length calculation module 251 is used to calculate the side length of the target detection tilt box based on the pixel coordinates of each reference vertex of the minimum rectangular bounding box.

[0247] The tilt angle calculation module 252 is used to calculate the tilt angle of the target detection tilt box based on the pixel coordinates of each reference vertex of the minimum rectangular bounding box.

[0248] The acquisition module 253 is used to obtain the pixel coordinates of the center point of the target detection tilted box based on the average pixel coordinates of each reference vertex of the minimum rectangular bounding box; and / or to obtain the pixel coordinates of the center point of the target detection tilted box through coordinate transformation from the world coordinate system to the pixel coordinate system based on the world coordinates of the target center point, the world coordinates of the image acquisition device, and the pose information of the image acquisition device.

[0249] The tilted box generation module 254 is used to generate a target detection tilted box based on the pixel coordinates of the center point of the target detection tilted box, the side length of the target detection tilted box, and the tilt angle.

[0250] Preferably, the side length calculation module 251 is specifically used for:

[0251] The pixel coordinates of any vertex of the minimum rectangular bounding box are used as the reference point;

[0252] Select the two reference vertices with shorter straight-line distances from the reference point from the remaining three reference vertices, and use the straight-line distance between the reference point and the two reference vertices as the lengths of the two sides that intersect the reference point in the minimum rectangular bounding box.

[0253] If the two sides intersecting at the reference point in the minimum bounding box are of different lengths, then the shortest side is taken as the width of the target detection tilt box, and the longest side is taken as the length of the target detection tilt box.

[0254] If the two sides of the smallest bounding rectangle that intersect at the reference point have equal lengths, then either side length will be used as the side length of each side of the target detection tilt box.

[0255] Preferably, the tilt angle calculation module 252 is specifically used for:

[0256] If the two sides intersecting at the reference point in the minimum bounding box are of unequal length, the tilt angle between the target detection tilt box and the horizontal axis of the pixel coordinate system is calculated based on the projection length of the longest side onto the vertical and horizontal axes of the pixel coordinate system.

[0257] If the two sides intersecting at the reference point within the smallest bounding rectangle are of equal length, then the tilt angle between the target detection tilt box and the horizontal axis of the pixel coordinate system is calculated based on the projection length of either side length onto the vertical and horizontal axes of the pixel coordinate system.

[0258] Preferably, the acquisition module 253, based on the world coordinates of the target object's center point, the world coordinates of the image acquisition device, and the pose information of the image acquisition device, obtains the pixel coordinates of the target detection tilted bounding box center point through a coordinate transformation from the world coordinate system to the pixel coordinate system. Specifically, this may include:

[0259] Based on the world coordinates of the target object's center point, the world coordinates of the image acquisition device, and the attitude information of the image acquisition device, the pixel coordinates of the target object's center point in the pixel coordinate system are obtained by sequentially transforming the coordinates from the world coordinate system to the image acquisition device coordinate system, from the image acquisition device coordinate system to the image coordinate system, and from the image coordinate system to the pixel coordinate system.

[0260] Preferably, the attitude information of the image acquisition device includes the yaw angle, pitch angle, and roll angle of the image acquisition device.

[0261] An optional solution in Embodiment 1 of the present invention is to obtain the pixel coordinates of the target detection tilted bounding box center point by means of the world coordinates of the target object center point, the world coordinates of the image acquisition device, and the pose information of the image acquisition device in the acquisition module 253, through coordinate transformation from the world coordinate system to the pixel coordinate system. Specifically, it may further include:

[0262] Based on the world coordinates of the target's center point, the camera's world coordinates, and the camera's attitude information, the coordinates of the target's center point in the camera coordinate system are obtained through coordinate transformation from the world coordinate system to the camera coordinate system.

[0263] Based on the camera's internal parameters, the coordinates of the target's center point in the image coordinate system are obtained through coordinate transformation from the camera coordinate system to the image coordinate system.

[0264] Among these, the camera's internal parameters may include the camera's focal length;

[0265] Based on the coordinates of the center point of the target object in the image coordinate system, the pixel coordinates of the center point of the target object in the pixel coordinate system are obtained through coordinate transformation from the image coordinate system to the pixel coordinate system.

[0266] The present invention provides a method and system for generating tilted bounding boxes for target detection, which can truly realize the automated generation of tilted bounding boxes for target detection with tilt angles without human intervention; it can also simplify the operation process and ensure the accuracy of annotation; in addition, it eliminates the need for complex lidar and millimeter-wave radar configurations, simplifying the initial configuration work and saving costs.

[0267] The target detection tilted bounding box generation method and system provided by this invention, compared with existing rectangular bounding boxes, achieves better alignment and more accurate positioning of the tilted bounding box with the target in air-to-ground images with arbitrary rotation angles and large horizontal-to-vertical ratios. In air-to-ground images with multiple small and densely packed targets, the overlap between the labeled tilted bounding boxes is significantly reduced, which is beneficial for training target detection algorithms using the tilted bounding boxes generated by this invention.

[0268] Example 2

[0269] This embodiment of the invention, based on the world coordinates of the target object's center point, the world coordinates of the image acquisition device, and the image acquisition device's pose information in S1421 of Embodiment 1, sequentially performs coordinate transformations from the world coordinate system to the image acquisition device coordinate system, from the image acquisition device coordinate system to the image coordinate system, and from the image coordinate system to the pixel coordinate system to obtain the pixel coordinates of the target object's center point in the pixel coordinate system. The following is a detailed description:

[0270] S14211', Based on the world coordinates P of the target object's center point w =[x w y w z w ] T The world coordinates of the camera O c =[x o y o z o ] T and attitude information (including yaw angle θ, pitch angle) And the roll angle ω), calculate the coordinates P of the target object in the camera coordinate system. c =[x c y c z c ] T The transformation from the world coordinate system to the camera coordinate system is a rigid body transformation, requiring rotation and translation, such as... Figure 3 As shown.

[0271]

[0272] Where R is the rotation matrix and T is the translation vector;

[0273] Rotation θ around the Z-axis, rotation matrix

[0274]

[0275] Rotate around the Y-axis Rotation matrix

[0276]

[0277] Rotation ω around the X-axis, rotation matrix

[0278]

[0279] R = R z R y R x ,.................................................(20)

[0280] Translation matrix

[0281]

[0282] Combining (16) and (21), we can obtain:

[0283]

[0284] S14212', From the camera coordinate system to the image coordinate system, this is a perspective projection relationship. The transformation from 3D to 2D involves point P, which has been transformed from the world coordinate system to the camera coordinate system, represented as P(x). c y c , z c ),like Figure 4 As shown.

[0285] Based on the principle of similar triangles, and using formula (23), the coordinates of the center point of the target object in the image coordinate system are obtained:

[0286]

[0287] In the formula, f is the focal length of the camera.

[0288] In this embodiment of the invention, the coordinate transformation from the camera coordinate system to the image coordinate system based on the principle of similar triangles can have the above proportional relationship:

[0289]

[0290] S14213', the pixel coordinate system, and the image coordinate system all lie on the imaging plane, but their origins and units of measurement differ.Figure 5 As shown, the origin of the image coordinate system is the intersection of the camera's optical axis and the imaging plane, usually the midpoint of the imaging plane, called the principal point. The unit of the image coordinate system is mm, a physical unit, while the unit of the pixel coordinate system is pixel. We usually describe a pixel as having several rows and columns. Therefore, the conversion between the two is as follows:

[0291]

[0292] In the formula, u0 and v0 represent the difference in horizontal and vertical pixels between the pixel coordinates of the center point (origin of the image coordinate system) and the pixel coordinates of the image origin, respectively, which are 1 / 2 of the camera resolution; dx and dy represent how many mm each column and each row represent, that is, 1 pixel = dx mm.

[0293] Finally, combining S14211' to S14213', we can obtain:

[0294]

[0295] Therefore, the pixel coordinates of the target object's center can be obtained as A = [uv]. T .

[0296] Similarly, based on the world coordinates of each vertex of the target object's smallest circumscribed cuboid, the camera's world coordinates, and the camera's pose information, referring to the coordinate transformation process described above, executing S1221-S1223 yields the pixel coordinates A1 = [u1 v1] of the eight vertices of the target object's smallest circumscribed cuboid. T A2 = [u2 v2] T A3 = [u3 v3] T A4 = [u4 v4] T A5 = [u5 v5] T A6 = [u6v6] T A7 = [u7 v7] T A8 = [u8 v8] T .

[0297] Example 3

[0298] Embodiment 3 of the present invention also provides a method for automatic annotation of tilted bounding boxes for target detection in image data, such as... Figure 6 As shown, the specific steps include the following:

[0299] S31. Generate a tilted bounding box for target detection in the image;

[0300] S32. Obtain the category of the target object in the image, and add category labels to the target detection bounding boxes in the image according to the target object category to obtain the final annotation data of the target object in the image;

[0301] In step S31, the target detection tilt box in the image can be generated using either the method in Example 1 or Example 2.

[0302] Based on the same inventive concept, Embodiment 3 of the present invention also proposes an automatic tilted bounding box annotation system for target detection in image data, specifically including the following components:

[0303] Generating device 41 is used to generate a target detection tilted bounding box in an image;

[0304] The annotation device 42 is used to obtain the category of the target object in the image and add category labels to the target detection bounding box according to the target object category, so as to obtain the final annotation data of the target object in the image;

[0305] The generating device 41 can generate a target detection tilt box in the image by executing any of the methods in Embodiment 1 or 2.

[0306] In the above-mentioned S32 and labeling device 42, category labels can be added according to the target object category, specifically the pixel coordinates of the center point of the target detection tilted box in the image, the side length of the target detection tilted box, and the tilt angle between the target detection tilted box and the horizontal axis of the pixel coordinate system.

[0307] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0308] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0309] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0310] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0311] The above are merely embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of the claims of the present invention pending approval.

Claims

1. A method for automatically annotating tilted bounding boxes in image data target detection, characterized in that, include: Acquire images of the target object and obtain its length, width, and height, as well as the world coordinates of the target object's center point, the target object's pose information, the world coordinates of the image acquisition device, and the pose information of the image acquisition device; Based on the length, width, and height of the target object, as well as the world coordinates of the target object's center point, the target object's pose information, the world coordinates of the image acquisition device, and the image acquisition device's pose information, the pixel coordinates of each vertex of the target object's smallest circumscribed cuboid are obtained through coordinate transformation. Based on the pixel coordinates of each vertex of the minimum bounding box of the target object, determine the pixel coordinates of each reference vertex of the minimum bounding box of the target object in the pixel coordinate system. The side length and tilt angle of the target detection tilt box are calculated based on the pixel coordinates of each reference vertex of the minimum rectangular bounding box. The target detection tilt box is then generated based on the obtained pixel coordinates of the center point of the target detection tilt box, the side length, and the tilt angle. The target detection tilted box is used to label target objects in the image.

2. The method as described in claim 1, characterized in that, The step of obtaining the pixel coordinates of each vertex of the smallest circumscribed cuboid of the target object through coordinate transformation based on the length, width, and height of the target object, as well as the world coordinates of the target object's center point, the target object's pose information, the world coordinates of the image acquisition device, and the image acquisition device's pose information, includes: Based on the world coordinates of the center point of the target object, the attitude information of the target object, and the length, width, and height of the target object, calculate the world coordinates of each vertex of the minimum circumscribed cuboid of the target object; Based on the world coordinates of each vertex of the minimum circumscribed cuboid of the target object, the world coordinates of the image acquisition device, and the pose information of the image acquisition device, the pixel coordinates of each vertex of the minimum circumscribed cuboid of the target object are obtained through coordinate transformation from the world coordinate system to the pixel coordinate system.

3. The method as described in claim 2, characterized in that, The step of calculating the world coordinates of each vertex of the minimum circumscribed cuboid of the target object based on the world coordinates of the target object's center point, the target object's attitude information, and the target object's length, width, and height includes: Construct a target object coordinate system with the center of the target object as the origin, and make the x, y, and z axes of the target object coordinate system parallel to the length, width, and height directions of the target object, respectively; calculate the coordinates of each vertex of the smallest circumscribed cuboid of the target object in the target object coordinate system based on the length, width, and height of the target object. Based on the coordinates of each vertex of the minimum circumscribed cuboid of the target object in the target object coordinate system, the attitude information of the target object, and the world coordinates of the center point of the target object, the world coordinates of each vertex of the minimum circumscribed cuboid of the target object are obtained through coordinate transformation from the target object coordinate system to the world coordinate system.

4. The method as described in claim 3, characterized in that, The calculation of the coordinates of each vertex of the smallest circumscribed cuboid of the target object in the target object coordinate system based on the length, width, and height of the target object includes: The coordinates of each vertex of the smallest circumscribed cuboid are calculated using the following formula: In the formula, Let P be the coordinates of the i-th vertex of the smallest circumscribed cuboid in the target object coordinate system. ti The value of ; i is the number of vertices of the smallest circumscribed cuboid; i∈[1,8]; These are the length, width, and height of the target object, respectively.

5. The method as described in claim 3, characterized in that, The process of obtaining the world coordinates of each vertex of the minimum circumscribed cuboid of the target object based on the coordinates of each vertex in the target object coordinate system, the attitude information of the target object, and the world coordinates of the center point of the target object, through coordinate transformation from the target object coordinate system to the world coordinate system, includes: The world coordinates of each vertex of the smallest bounding cuboid of the target object in the world coordinate system are calculated using the following formula: In the formula, Let P be the world coordinates of the i-th vertex of the minimum circumscribed cuboid. wi The value of i, i∈[1,8]; It is a translation vector; The world coordinates of the center point of the target object The value; These are the attitude information of the target object, including yaw angle, pitch angle, and roll angle; Rotation around the Z-axis The first rotation matrix obtained, rotated around the Y-axis The obtained second rotation matrix and rotation around the X-axis The resulting third rotation matrix; To The rotation matrix obtained after superposition; for The inverse matrix.

6. The method as described in claim 2, characterized in that, The process of obtaining the pixel coordinates of each vertex of the minimum circumscribed cuboid of the target object through coordinate transformation from the world coordinate system to the pixel coordinate system, based on the world coordinates of each vertex of the minimum circumscribed cuboid of the target object, the world coordinates of the image acquisition device, and the pose information of the image acquisition device, includes: Based on the world coordinates of each vertex of the minimum circumscribed cuboid of the target object, the world coordinates of the image acquisition device, and the attitude information of the image acquisition device, the pixel coordinates of each vertex of the minimum circumscribed cuboid of the target object in the pixel coordinate system are obtained by sequentially transforming the coordinates from the world coordinate system to the image acquisition device coordinate system, from the image acquisition device coordinate system to the image coordinate system, and from the image coordinate system to the pixel coordinate system.

7. The method as described in claim 6, characterized in that, The attitude information of the image acquisition device includes the yaw angle, pitch angle, and roll angle of the image acquisition device.

8. The method as described in claim 1, characterized in that, The step of determining the pixel coordinates of each reference vertex of the minimum bounding box of the target object in the pixel coordinate system based on the pixel coordinates of each vertex of the minimum bounding box of the target object includes: Based on the pixel coordinates of each vertex of the minimum bounding cuboid of the target object, the convex hull algorithm is used to construct the convex hull of each vertex of the minimum bounding cuboid of the target object in the pixel coordinate system. The pixel coordinates of each reference vertex of the minimum bounding box of the minimum circumscribed cuboid of the target object are calculated based on the convex hull.

9. The method according to claim 8, characterized in that, The step of constructing the convex hull of each vertex of the minimum bounding cuboid of the target object in the pixel coordinate system using the convex hull algorithm includes: Select the minimum or maximum pixel coordinates of each vertex of the smallest circumscribed cuboid of the target object in the direction of the horizontal or vertical axis of the pixel coordinate system as the initial vertices; Connect the initial vertex to each of the remaining vertices to obtain corresponding line segments. Select a line segment that satisfies the constraints, and use another vertex on that line segment other than the initial point as the initial point. Repeat this step until the selected vertices are connected one by one to form a polygon that can enclose all the vertices of the minimum circumscribed cuboid. The polygon then serves as the convex hull of each vertex of the minimum circumscribed cuboid in the pixel coordinate system.

10. The method according to claim 9, characterized in that, The constraints include: all line segments other than the selected line segment are located on the same side of the selected line segment.

11. The method according to claim 8 or 9, characterized in that, The step of calculating the pixel coordinates of each reference vertex of the minimum bounding box of the minimum circumscribed cuboid of the target object in the pixel coordinate system based on the convex hull includes: Construct the corresponding rectangle for each edge of the convex hull; Compare the areas of all rectangles, select the rectangle with the smallest area as the smallest bounding box of the target object's smallest circumscribed cuboid in pixel coordinates, and use the pixel coordinates of the vertices of the smallest rectangle as the pixel coordinates of the reference vertices of the smallest bounding box.

12. The method according to claim 11, characterized in that, The step of constructing a corresponding rectangle based on each edge of the convex hull includes: Draw the first straight line passing through the two vertices of an edge of the convex hull; Project all vertices except the two vertices on the first line onto the first line, and draw the second and third lines perpendicular to the first line along the two vertices with the farthest projection distance. Find the vertex in the convex hull that is farthest from the first line, and draw a fourth line through that vertex that is parallel to the first line; Using the four points where the first to fourth lines intersect as the vertices of the rectangle, we obtain the rectangle corresponding to the convex hull of that edge.

13. The method according to claim 1, characterized in that, The step of calculating the side length of the target detection tilted bounding box based on the pixel coordinates of each reference vertex of the minimum rectangular bounding box includes: The pixel coordinates of any vertex of the minimum rectangular bounding box are used as the reference point; Select the two reference vertices with shorter straight-line distances from the reference point from the remaining three reference vertices, and use the straight-line distance between the reference point and the two reference vertices as the lengths of the two sides that intersect the reference point in the minimum rectangular bounding box. If the two sides intersecting at the reference point in the minimum bounding rectangle are of different lengths, then the shortest side is taken as the width of the target detection tilted box, and the longest side is taken as the length of the target detection tilted box. If the two sides of the smallest bounding rectangle that intersect at the reference point have equal lengths, then either side length will be used as the side length of each side of the target detection tilt box.

14. The method according to claim 13, characterized in that, The method for calculating the tilt angle of the target detection bounding box includes: If the two sides intersecting at the reference point in the minimum bounding box are of unequal length, the tilt angle between the target detection tilt box and the horizontal axis of the pixel coordinate system is calculated based on the projection length of the longest side onto the vertical and horizontal axes of the pixel coordinate system. If the two sides of the smallest bounding box that intersect at the reference point have equal lengths, then the tilt angle between the target detection tilt box and the horizontal axis of the pixel coordinate system is calculated based on the projection length of either side length onto the vertical and horizontal axes of the pixel coordinate system.

15. The method according to claim 14, characterized in that, The method for obtaining the pixel coordinates of the center point of the target detection tilted bounding box includes: The pixel coordinates of the center point of the target detection tilt box are obtained by averaging the pixel coordinates of the reference vertices of the smallest rectangular bounding box. Preferably, the method for obtaining the pixel coordinates of the center point of the target detection tilted bounding box includes: Based on the world coordinates of the target object's center point, the world coordinates of the image acquisition device, and the attitude information of the image acquisition device, the pixel coordinates of the target object's center point in the pixel coordinate system are obtained through coordinate transformation from the world coordinate system to the pixel coordinate system. Use the pixel coordinates of the center point of the target object as the pixel coordinates of the center point of the target detection tilt box.

16. An automatic tilted bounding box annotation system for target detection in image data, characterized in that, include: Image acquisition device, used to acquire images of target objects; The sensor is used to acquire the length, width, and height of the target object, as well as the world coordinates of the target object's center point, the target object's attitude information, the world coordinates of the image acquisition device, and the attitude information of the image acquisition device. The first calculator is used to obtain the pixel coordinates of each vertex of the smallest bounding cube of the target object through coordinate transformation based on the length, width and height of the target object, as well as the world coordinates of the center point of the target object, the pose information of the target object, the world coordinates of the image acquisition device and the pose information of the image acquisition device. The second calculator is used to determine the pixel coordinates of each reference vertex of the minimum bounding box of the minimum bounding box of the target object in the pixel coordinate system based on the pixel coordinates of each vertex of the minimum bounding box of the target object. A generator is used to calculate the side length of the target detection tilt box based on the pixel coordinates of each reference vertex of the minimum bounding box. And the tilt angle of the target detection tilt box, and generate the target detection tilt box based on the pixel coordinates of the center point of the target detection tilt box, the side length and the tilt angle; The target detection tilted box is used to label target objects in the image.

17. A method for automatically annotating tilted bounding boxes in image data target detection, characterized in that, include: Using the method described in any one of claims 1-15, a target detection tilt box is generated in the image; Obtain the object category in the image, and add category labels to the pixel coordinates of the center point of the object detection tilt box, the side length of the object detection tilt box, and the tilt angle according to the object category to obtain the final annotation data of the object in the image.

18. An automatic tilted bounding box annotation system for target detection in image data, characterized in that, include: A generating apparatus for generating a target detection tilted bounding box in an image by performing the method as described in any one of claims 1-15; The annotation device is used to obtain the category of the target object in the image, and add category labels to the pixel coordinates of the center point of the target detection tilt box, the side length of the target detection tilt box and the tilt angle according to the target object category, so as to obtain the final annotation data of the target object in the image.

Images