Labeling method, system and equipment

Abstract

The invention relates to a labeling method, system and equipment, and the method comprises the steps: obtaining a plurality of image frames and a plurality of point cloud frames collected in a target region, and enabling the collection time of each image frame to be the same as the collection time of one point cloud frame; a target image frame set and a target point cloud frame set are obtained, the target image frame is the image frame containing a to-be-labeled object, and the target point cloud frame is the point cloud frame collected at the same moment as the target image frame; converting each point cloud frame in the target point cloud frame set to the same coordinate system to construct a first point cloud; projecting the first point cloud to each image frame in the target image frame set, and acquiring a second point cloud corresponding to each image frame, the second point cloud being a point cloud projected into a 2D bounding box of an object to be labeled in the image frame; and based on each of the second point clouds, obtaining a 3D bounding box corresponding to the to-be-labeled object in the image frame corresponding to each of the second point clouds.

Description

Technical Field

[0001] The invention relates to a labeling method, system and equipment, and belongs to the technical field of computer-aided driving. Background Art

[0002] In the field of autonomous driving, the labeling of traffic signs and traffic lights on the road, including regional labeling, classification labeling and semantic labeling, is an important input for vehicle decision-making planning. 4D labeling adds time series information to 3D labeling. It can not only label static targets in 3D space, but also dynamic targets with time series information. At present, deep neural network models are mainly used to identify the geometric and semantic information of signs and traffic lights. The training of the model often requires a large amount of manually labeled data, and manual labeling has problems such as high labeling cost and long labeling cycle. Summary of the invention

[0003] In view of the shortcomings of the prior art, the present invention provides a labeling method, system and device, which fuse image data and point cloud data acquired by sensors of different dimensions, and fully utilize the advantages of image data and point cloud data. Image data can accurately obtain the 2D bounding box and semantics of the object to be labeled, and point cloud data can accurately obtain the position information, geometric dimensions and angular orientation of the 3D bounding box of the object to be labeled, thereby realizing more accurate and reliable 4D automatic labeling, greatly reducing the links of manual intervention, and greatly reducing the cost of manual labeling.

[0004] According to a first aspect of the present invention, there is provided a marking method, comprising:

[0005] Acquire multiple image frames and multiple point cloud frames collected in the target area, where each of the image frames is collected at the same time as one of the point cloud frames;

[0006] Acquire a target image frame set and a target point cloud frame set, wherein the target image frame is the image frame containing the object to be annotated, and the target point cloud frame is the point cloud frame collected at the same time as the target image frame;

[0007] Convert each point cloud frame in the target point cloud frame set to the same coordinate system to construct a first point cloud;

[0008] Projecting the first point cloud to each image frame in the target image frame set, obtaining a second point cloud corresponding to each image frame, where the second point cloud is a point cloud projected into a 2D bounding box of the object to be annotated in the image frame;

[0009] Based on each of the second point clouds, a 3D bounding box corresponding to the object to be annotated in the image frame corresponding to each of the second point clouds is obtained.

[0010] According to one embodiment, the marking method further includes:

[0011] Calculate the similarity of the 3D bounding boxes corresponding to the objects to be annotated in every two adjacent image frames in the target image frame set, obtain an image frame sequence that meets a preset similarity threshold and has a continuous frame number greater than a preset frame number threshold, and determine the target 3D bounding box;

[0012] The target 3D bounding box is semantically bound to a 2D bounding box of the object to be annotated in each image frame in the corresponding image frame sequence.

[0013] According to one embodiment, converting each point cloud frame in the target point cloud frame set to the same coordinate system to construct a first point cloud includes:

[0014] According to the vehicle posture at the corresponding moment of each point cloud frame, each point cloud frame is converted to the same coordinate system;

[0015] Merge each converted point cloud frame to obtain the first point cloud.

[0016] According to one embodiment, projecting the first point cloud onto each image frame in the target image frame set to obtain a second point cloud corresponding to each image frame includes:

[0017] Projecting the first point cloud to each image frame in the target image frame set in an image coordinate system;

[0018] According to the 2D bounding box of the object to be annotated in each image frame, a two-dimensional pixel point set projected into the 2D bounding box area is obtained, and a three-dimensional point cloud corresponding to the two-dimensional pixel point set is obtained as a second point cloud corresponding to the image frame.

[0019] According to one embodiment, obtaining, based on each of the second point clouds, a 3D bounding box corresponding to the object to be annotated in the image frame corresponding to each of the second point clouds includes:

[0020] Clustering the second point cloud to obtain a point cloud cluster including a point cloud number greater than a preset number threshold and a point cloud cluster having a distance from the acquisition device less than a preset distance threshold;

[0021] Calculating the center of the point cloud cluster;

[0022] Based on the center of the point cloud cluster, plane fitting is performed on the point cloud cluster to obtain a plane normal vector as an X-axis;

[0023] Project each point of the point cloud cluster onto the X-axis, sort the projected points, and obtain the Euclidean distance between the first point and the last point in the sorting as the length of the 3D bounding box corresponding to the object to be annotated in the image frame;

[0024] Project each point of the point cloud cluster onto the Z axis, sort the projected points, and obtain the Euclidean distance between the first point and the last point in the sorting as the height of the 3D bounding box corresponding to the object to be marked in the image frame; the direction vector of the Z axis is a direction vector perpendicular to the ground;

[0025] Project each point of the point cloud cluster onto the Y axis, sort the projected points, and obtain the Euclidean distance between the first point and the last point in the sorting as the width of the 3D bounding box corresponding to the object to be annotated in the image frame; the direction vectors of the Y axis are perpendicular to the direction vectors of the X axis and the Z axis;

[0026] According to the plane normal vector, the heading angle of the 3D bounding box corresponding to the object to be marked in the image frame is calculated.

[0027] According to an embodiment, the similarity is the overlap of 3D bounding boxes.

[0028] According to a second aspect of the present invention, there is provided a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the marking method according to any one of the first aspects of the present invention is implemented.

[0029] According to a third aspect of the present invention, there is provided a labeling device, comprising a memory and a processor, wherein the memory is used to store computer instructions, and the processor is used to call the computer instructions from the memory to execute the labeling method as described in any one of the first aspects of the present invention.

[0030] According to a fourth aspect of the present invention, there is provided a marking system, comprising:

[0031] A data acquisition module, used to acquire multiple image frames and multiple point cloud frames collected in the target area, wherein each of the image frames is collected at the same time as one of the point cloud frames;

[0032] A target frame acquisition module is used to acquire a target image frame set and a target point cloud frame set, wherein the target image frame is the image frame containing the object to be annotated, and the target point cloud frame is the point cloud frame collected at the same time as the target image frame;

[0033] A first point cloud acquisition module, used for converting each point cloud frame in the target point cloud frame set into the same coordinate system to construct a first point cloud;

[0034] A second point cloud acquisition module is used to project the first point cloud to each image frame in the target image frame set to acquire a second point cloud corresponding to each image frame, where the second point cloud is a point cloud projected into a 2D bounding box of the object to be annotated in the image frame;

[0035] The 3D bounding box acquisition module is used to acquire, based on each of the second point clouds, a 3D bounding box corresponding to the object to be annotated in the image frame corresponding to each of the second point clouds.

[0036] According to one embodiment, the annotation system further includes:

[0037] A false detection filtering module is used to calculate the similarity of the 3D bounding boxes corresponding to the objects to be annotated in every two adjacent image frames in the target image frame set, obtain an image frame sequence that meets a preset similarity threshold and has a continuous frame number greater than a preset frame number threshold, and determine the target 3D bounding box;

[0038] The semantic binding module is used to semantically bind the target 3D bounding box to the 2D bounding box of the object to be annotated in each image frame in the corresponding image frame sequence.

[0039] The present invention uses image data and point cloud data collected simultaneously in a target area as input, identifies the 2D bounding box and semantics of the object to be labeled on the image data, obtains a target image frame set containing the object to be labeled and a target point cloud frame set consisting of point cloud frames collected at the same time as each image frame in the target image frame set, and then converts all point cloud frames in the target point cloud frame set to the same coordinate system according to the vehicle body posture corresponding to each point cloud frame to construct a first point cloud, and then projects the first point cloud to each image frame in the target image frame set respectively, obtains the point cloud projected into the 2D bounding box of the object to be labeled in each image frame as the second point cloud corresponding to the image frame, and then clusters the second point cloud corresponding to each image frame to obtain a point cloud cluster, and calculates the geometric information of the point cloud cluster to obtain the plane normal vector, length, width, height and heading angle of the corresponding 3D bounding box of the object to be labeled. In addition, the present invention also matches the 3D bounding boxes corresponding to the target image frames at every two adjacent moments by similarity, filters out false detections, obtains an image frame sequence with continuous matching 3D bounding boxes, determines the target 3D bounding box according to the characteristic parameters of all 3D bounding boxes corresponding to the same image frame sequence, and semantically binds the target 3D bounding box to the 2D bounding box of the object to be labeled in each image frame in the corresponding image frame sequence to generate a 4D label, thereby achieving more accurate and reliable 4D automatic labeling, greatly reducing the links of manual intervention, and greatly reducing the cost of manual labeling. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A flowchart of the main steps of a marking method provided by an embodiment of the present invention.

[0041] Figure 2 A flowchart of the main steps of another marking method provided by an embodiment of the present invention.

[0042] Figure 3A schematic diagram of main modules of a labeling system provided in an embodiment of the present invention.

[0043] Figure 4 A schematic diagram of main modules of another annotation system provided by an embodiment of the present invention.

[0044] Figure 5 A schematic diagram of the structure of a marking device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0045] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below in conjunction with specific implementation methods.

[0046] Technologies, methods and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the above-mentioned technologies, methods and equipment should be considered as part of the specification.

[0047] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0048] Figure 1 A flowchart of a marking method provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown, the marking method of this embodiment may include the following steps S101 to S105.

[0049] Step S101, acquiring multiple image frames and multiple point cloud frames collected in the target area, wherein each image frame is collected at the same time as a point cloud frame.

[0050] In some examples, a data collection vehicle is used to collect continuous image data and point cloud data in a time dimension in a target area, and the data is expressed in the form of multiple image frames and multiple point cloud frames. For example, an image frame is {I 0 ,I 1 ,...,I i ,I i+1 ,...,I n}, the point cloud frame is {P 0 , P 1 , ..., P i , P i+1 , ..., P n}, where i represents the acquisition time. The timestamps of the image data and the point cloud data are synchronized, that is, the acquisition time of each image frame is the same as the acquisition time of a point cloud frame.

[0051] Step S102, obtaining a target image frame set and a target point cloud frame set, wherein the target image frame is an image frame containing an object to be annotated, and the target point cloud frame is a point cloud frame collected at the same time as the target image frame.

[0052] The objects to be labeled may be fixed objects such as traffic lights and traffic signs. In some examples, the existing 2D image detection model is used to perform detection on continuous image frames, identify the objects to be labeled in each image frame, take the image frame containing the objects to be labeled as the target image frame, obtain the boundary of the objects to be labeled in each target image frame, and express it in the form of a 2D bounding box. The size of the 2D bounding box is the size of the objects to be labeled in the target image frame, and determine the target image frame set: {I i ,I i+1 ,...,I j According to the timestamp of each target image frame, the point cloud frame corresponding to the acquisition time is selected as the target point cloud frame to form a target point cloud frame set: {P i , P i+1 , ..., P j}.

[0053] Step S103: convert each point cloud frame in the target point cloud frame set into the same coordinate system to construct a first point cloud.

[0054] In some examples, based on each point cloud frame in the target point cloud frame set, the traditional laser mileage calculation method is used to solve the pose matrix of the vehicle at the corresponding time of each point cloud frame, which can be expressed as: in Represents the pose matrix of the vehicle at time i in the world coordinate system. According to the pose of the vehicle at the corresponding time, each point cloud frame is converted to the world coordinate system. i For example, the position matrix of the vehicle in the world coordinate system at time i is Convert the point cloud frame at time i to the world coordinate system, that is in That is, the point cloud frame at time i in the world coordinate system. Then, merge the transformed point cloud frames at each time Construct the first point cloud, i.e.

[0055]

[0056] Where P w This is the first point cloud in the world coordinate system.

[0057] Step S104, projecting the first point cloud to each image frame in the target image frame set, and obtaining a second point cloud corresponding to each image frame, where the second point cloud is a point cloud projected into a 2D bounding box of the object to be annotated in the image frame.

[0058] In some examples, the first point cloud is projected to the image coordinate system through coordinate system conversion, and then the two-dimensional pixel point set projected from the first point cloud to the 2D bounding box area is obtained according to the coordinate range of the 2D bounding box of the object to be annotated in each target image frame, and the three-dimensional point cloud point set corresponding to the two-dimensional pixel point set is obtained by reverse search as the second point cloud corresponding to the target image frame. Taking the projection of the first point cloud constructed in the above steps to the target image frame at time i as an example, the specific steps can be as follows:

[0059] Project the first point cloud in the world coordinate system to the vehicle coordinate system, taking time i as an example, that is,

[0060]

[0061] in, for The inverse matrix of is the first point cloud in the vehicle coordinate system at time i;

[0062] The point cloud in the vehicle coordinate system is converted to the image coordinate system through the calibration parameter matrix of the vehicle body and the camera device, that is,

[0063]

[0064] in, is the first point cloud in the image coordinate system at time i;

[0065] Set the coordinates of the 2D bounding box of the object to be labeled detected in the image frame at time i to {x l 、x r ,y t ,y b}, set the horizontal and vertical coordinates of each pixel point projected to the image coordinate system as According to the coordinate range of the 2D bounding box in the image frame at time i, the set of projected 2D pixel points in the box is filtered out as follows:

[0066]

[0067] Based on the projected 2D pixel point set, reversely search for the corresponding 3D point cloud point set R in the world coordinate system i , R i Represents the second point cloud corresponding to the target image frame at time i, where the second point cloud is a subset of the first point cloud.

[0068] It is worth mentioning that since the single-frame point cloud is too sparse, if the target point cloud frame at the same acquisition time is directly projected to the target image frame to filter the second point cloud, the second point cloud contains too few point cloud points, which will cause a large fitting deviation of the subsequent 3D bounding box. Through step S104, the point cloud points collected by the same object to be labeled in different target point cloud frames can be obtained, which greatly increases the accuracy of subsequent fitting.

[0069] Step S105 : based on each second point cloud, obtaining a 3D bounding box corresponding to the object to be annotated in the image frame corresponding to each second point cloud.

[0070] In some examples, taking obtaining a 3D bounding box corresponding to an object to be annotated in a single image frame as an example, step S105 may specifically include the following steps:

[0071] Clustering the second point cloud to obtain a point cloud cluster including a point cloud number greater than a preset number threshold and a point cloud cluster having a distance from the acquisition device less than a preset distance threshold;

[0072] Calculate the center of the point cloud cluster;

[0073] Based on the center of the point cloud cluster, plane fitting is performed on the point cloud cluster to obtain the plane normal vector as the X-axis;

[0074] Project each point of the point cloud cluster onto the X-axis, sort the projected points, and obtain the Euclidean distance between the first point and the last point in the sorting as the length of the 3D bounding box corresponding to the object to be annotated in the image frame;

[0075] Project each point of the point cloud cluster onto the Z axis, sort the projected points, and obtain the Euclidean distance between the first point and the last point in the sorting order as the height of the 3D bounding box corresponding to the object to be annotated in the image frame; the direction vector of the Z axis is the direction vector perpendicular to the ground;

[0076] Project each point of the point cloud cluster onto the Y axis, sort the projected points, and obtain the Euclidean distance between the first and last points in the sorting as the width of the 3D bounding box corresponding to the object to be annotated in the image frame; the direction vectors of the Y axis are perpendicular to the direction vectors of the X axis and the Z axis;

[0077] According to the plane normal vector, the heading angle of the 3D bounding box corresponding to the object to be marked in the image frame is calculated.

[0078] Clustering is a data mining technology used to divide objects in a data set into multiple groups with similar characteristics, which are called "clusters". Point cloud clustering is to divide the point cloud points in point cloud data (composed of a large number of discrete points, representing the three-dimensional information of objects or scenes) into different clusters according to certain specific standards, so that the data in the same cluster are as similar as possible, and the data in different clusters are as different as possible.

[0079] In one implementation, the DBSCAN (Density-Based Spatial Clustering of Applications with Noise) algorithm is used for point cloud clustering. DBSCAN is a density-based spatial clustering algorithm that divides areas with sufficient density into clusters. Through the DBSCAN algorithm, point cloud clusters containing a number of point clouds greater than a preset number threshold (for example, the number of point clouds contained in each point cloud cluster is greater than 20) and a distance from the acquisition device less than a preset distance threshold (for example, a distance from the vehicle is less than 100 meters) are selected from the second point cloud, which can be represented as S i , Among them, S i is a point cloud cluster obtained after clustering the second point cloud corresponding to the image frame at time i, and the point cloud cluster is a subset of the second point cloud.

[0080] Compute the centers of point cloud clusters:

[0081]

[0082] in Indicates the center coordinates of the point cloud cluster.

[0083] Based on the point cloud cluster center, the cluster point cloud is fitted into the plane, and the plane normal vector is obtained by SVD (Singular Value Decomposition) decomposition:

[0084]

[0085] in Represents the plane normal vector.

[0086] Then project each point of the point cloud cluster onto the straight line formed by the normal vector and the center of the point cloud cluster. for:

[0087]

[0088] Project the point Sort on the X axis and get the first point in the sort and the last point The Euclidean distance between these two points is taken as the length of the 3D bounding box, that is, the length of the 3D bounding box l i for:

[0089]

[0090] Then assume that the roll angle and pitch angle of the 3D bounding box are 0, take the plane normal vector as the X-axis, calculate and obtain the direction vector perpendicular to the X-axis direction vector and the direction vector perpendicular to the ground [0,0,1] as the direction vector of the Y-axis, project each point of the point cloud cluster onto the Y-axis, sort the projected points, and obtain the Euclidean distance between the first point and the last point in the sort as the width w of the 3D bounding box i , the projection and calculation methods will not be described in detail.

[0091] The direction vector of the Z axis is the direction vector perpendicular to the ground. Similarly, each point of the point cloud cluster is projected onto the Z axis, the projected points are sorted, and the Euclidean distance between the first point and the last point in the sorting is obtained as the height h of the 3D bounding box corresponding to the object to be labeled in the image frame. i .

[0092] Calculate the heading angle θ of the 3D bounding box i :

[0093]

[0094] In this way, the 3D bounding box B corresponding to the object to be labeled in the image frame at time i is obtained i :

[0095]

[0096] like Figure 2 As shown, based on the above embodiments, an embodiment of the present invention further provides a marking method, which specifically includes steps S101-S107, wherein steps S101-S105 are consistent with the steps in the above embodiments and will not be repeated here. The difference lies in steps S106 and S107, which are specifically as follows.

[0097] Step S106, calculating the similarity of the 3D bounding boxes corresponding to the objects to be annotated in every two adjacent image frames in the target image frame set, obtaining an image frame sequence that meets a preset similarity threshold and has a continuous frame number greater than a preset frame number threshold, and determining the target 3D bounding box.

[0098] The similarity is the degree of proximity or matching between two 3D bounding boxes in terms of specific features, and the specific features may be overlap, center point distance, etc.

[0099] Since the 2D detection model may have false detections when detecting objects to be labeled, in addition, other non-objects to be labeled that appear along the optical center of the camera in the 2D bounding box area of ​​the object to be labeled will also be projected into the 2D bounding box area of ​​the object to be labeled. These factors will lead to the screening of the wrong second point cloud. Therefore, it is necessary to filter the false detections and remove the 3D bounding boxes that are not actually corresponding to the objects to be labeled.

[0100] The 3D bounding box generated by the erroneous second point cloud introduced by the above two factors cannot be stably matched in time sequence. Therefore, in some examples, the matching relationship is determined by calculating the IOU (Intersection of Union) of the 3D bounding box corresponding to the object to be annotated in each two adjacent target image frames. IOU is determined by calculating the ratio of the intersection area of ​​the two boxes to the union area to determine the overlap of the two rectangular boxes. According to the preset similarity threshold (for example, IOU is greater than 0.3) and the preset frame number threshold (for example, 100 consecutive frames), the image frame sequence with continuous matching of the 3D bounding box is selected. The mean of the characteristic parameters of all 3D bounding boxes corresponding to the same image frame sequence is obtained by calculation as the characteristic parameters of the target 3D bounding box. The characteristic parameters of the 3D bounding box can be length, width, height, heading angle, center point coordinates, etc.

[0101] Step S107 : semantically binding the target 3D bounding box and the 2D bounding box of the object to be annotated in each image frame in the corresponding image frame sequence.

[0102] The geometric information of the target 3D bounding box is bounded with the semantics of the 2D bounding box of the object to be labeled in all image frames corresponding to the 3D bounding box to form a 4D label for the object to be labeled. Semantics are used to describe the specific meaning and attributes of the object to be labeled. For example, the semantic content of the 2D bounding box of a traffic light includes red light, green light, go straight, turn left, turn right, etc.

[0103] The present invention fully integrates the advantages of image data and point cloud data by constructing a dense first point cloud and projecting the first point cloud into each target image frame at different times, so that the fitting of the 3D bounding box corresponding to the object to be labeled in each target image frame is more accurate. In addition, the time dimension is introduced for verification. By calculating the similarity of the 3D bounding box corresponding to the object to be labeled in the target image frame at every two adjacent moments, false detections can be effectively filtered out to obtain true and accurate 3D geometric information of the object to be labeled. Therefore, the present invention can achieve more accurate and reliable 4D automatic labeling, greatly reduce the links of manual intervention, and greatly reduce the cost of manual labeling.

[0104] like Figure 3 As shown, an embodiment of the present invention further provides a marking system, the system comprising:

[0105] The data acquisition module 301 is used to acquire multiple image frames and multiple point cloud frames collected in the target area, and each image frame is collected at the same time as a point cloud frame;

[0106] The target frame acquisition module 302 is used to acquire a target image frame set and a target point cloud frame set, wherein the target image frame is an image frame containing an object to be annotated, and the target point cloud frame is a point cloud frame collected at the same time as the target image frame;

[0107] A first point cloud acquisition module 303 is used to convert each point cloud frame in the target point cloud frame set into the same coordinate system to construct a first point cloud;

[0108] A second point cloud acquisition module 304 is used to project the first point cloud to each image frame in the target image frame set, and acquire a second point cloud corresponding to each image frame, where the second point cloud is a point cloud projected into a 2D bounding box of the object to be annotated in the image frame;

[0109] The 3D bounding box acquisition module 305 is used to acquire, based on each second point cloud, a 3D bounding box corresponding to the object to be annotated in the image frame corresponding to each second point cloud.

[0110] In some examples, such as Figure 4 As shown, the annotation system also includes:

[0111] The false detection filtering module 306 is used to calculate the similarity of the 3D bounding boxes corresponding to the objects to be annotated in every two adjacent image frames in the target image frame set, obtain an image frame sequence that meets a preset similarity threshold and has a continuous frame number greater than a preset frame number threshold, and determine the target 3D bounding box;

[0112] The semantic binding module 307 is used to semantically bind the target 3D bounding box to the 2D bounding box of the object to be annotated in each image frame in the corresponding image frame sequence.

[0113] The embodiment of the present invention also provides a computer-readable storage medium on which a computer program is stored, and the computer program implements any of the methods in the aforementioned embodiments of the present invention when executed by a processor. Optionally, the computer-readable storage medium may be a non-transitory storage medium, but is not limited to this, and it may also be a temporary storage medium. A computer-readable storage medium is a tangible device that can hold and store instructions used by an instruction execution device. Computer-readable storage media may include, but are not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the above.

[0114] The embodiment of the present invention also provides a marking device, such as Figure 5As shown, the device 500 may include a memory 501 and a processor 502. The memory 501 may be used to store computer instructions, and the processor 502 may be used to call computer instructions from the memory 501 to execute all or part of the steps of any method in the aforementioned embodiments of the present invention. The processor may be one or more, and the one or more processors may execute instructions individually or together, and the memory may be one or more, and the one or more memories may store the above-mentioned computer instructions individually or together.

[0115] The flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the methods, systems, devices and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of an instruction, and the module, a program segment or a part of an instruction contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions. It should be noted that implementation by hardware, implementation by software, and implementation by a combination of software and hardware are all equivalent.

[0116] It should be understood that the above is only an exemplary embodiment of the present invention, and the protection scope of the present invention is not limited thereto. Those skilled in the art can easily conceive of modifications, replacements or adaptive changes without departing from the principle and essence of the present invention, which should all be covered within the protection scope of the present invention.

Claims

1. A marking method, characterized in that: include: Acquire multiple image frames and multiple point cloud frames collected in the target area, where each of the image frames is collected at the same time as one of the point cloud frames; Acquire a target image frame set and a target point cloud frame set, wherein the target image frame is the image frame containing the object to be annotated, and the target point cloud frame is the point cloud frame collected at the same time as the target image frame; Convert each point cloud frame in the target point cloud frame set to the same coordinate system to construct a first point cloud; Projecting the first point cloud to each image frame in the target image frame set, obtaining a second point cloud corresponding to each image frame, where the second point cloud is a point cloud projected into a 2D bounding box of the object to be annotated in the image frame; Based on each of the second point clouds, a 3D bounding box corresponding to the object to be annotated in the image frame corresponding to each of the second point clouds is obtained.

2. The marking method according to claim 1, characterized in that: Also includes: Calculate the similarity of the 3D bounding boxes corresponding to the objects to be annotated in every two adjacent image frames in the target image frame set, obtain an image frame sequence that meets a preset similarity threshold and has a continuous frame number greater than a preset frame number threshold, and determine the target 3D bounding box; The target 3D bounding box is semantically bound to a 2D bounding box of the object to be annotated in each image frame in the corresponding image frame sequence.

3. The marking method according to claim 1, characterized in that: The step of converting each point cloud frame in the target point cloud frame set into the same coordinate system to construct a first point cloud includes: According to the vehicle posture at the corresponding moment of each point cloud frame, each point cloud frame is converted to the same coordinate system; Merge each converted point cloud frame to obtain the first point cloud.

4. The marking method according to claim 1, characterized in that: The projecting the first point cloud to each image frame in the target image frame set to obtain a second point cloud corresponding to each image frame includes: Projecting the first point cloud to each image frame in the target image frame set in an image coordinate system; According to the 2D bounding box of the object to be annotated in each image frame, a two-dimensional pixel point set projected into the 2D bounding box area is obtained, and a three-dimensional point cloud corresponding to the two-dimensional pixel point set is obtained as a second point cloud corresponding to the image frame.

5. The marking method according to claim 1, characterized in that: The acquiring, based on each of the second point clouds, a 3D bounding box corresponding to the object to be annotated in the image frame corresponding to each of the second point clouds comprises: Clustering the second point cloud to obtain a point cloud cluster including a point cloud number greater than a preset number threshold and a point cloud cluster having a distance from the acquisition device less than a preset distance threshold; Calculating the center of the point cloud cluster; Based on the center of the point cloud cluster, plane fitting is performed on the point cloud cluster to obtain a plane normal vector as an X-axis; Project each point of the point cloud cluster onto the X-axis, sort the projected points, and obtain the Euclidean distance between the first point and the last point in the sorting as the length of the 3D bounding box corresponding to the object to be annotated in the image frame; Project each point of the point cloud cluster onto the Z axis, sort the projected points, and obtain the Euclidean distance between the first point and the last point in the sorting as the height of the 3D bounding box corresponding to the object to be marked in the image frame; the direction vector of the Z axis is a direction vector perpendicular to the ground; Project each point of the point cloud cluster onto the Y axis, sort the projected points, and obtain the Euclidean distance between the first point and the last point in the sorting as the width of the 3D bounding box corresponding to the object to be annotated in the image frame; the direction vectors of the Y axis are perpendicular to the direction vectors of the X axis and the Z axis; According to the plane normal vector, the heading angle of the 3D bounding box corresponding to the object to be marked in the image frame is calculated.

6. The marking method according to claim 2, characterized in that: The similarity is the overlap of the 3D bounding boxes.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is performed.

8. A marking device, characterized in that: The system comprises a memory and a processor, wherein the memory is used to store computer instructions, and the processor is used to call the computer instructions from the memory to execute the method according to any one of claims 1 to 6.

9. A labeling system, characterized in that: include: A data acquisition module, used to acquire multiple image frames and multiple point cloud frames collected in the target area, wherein each of the image frames is collected at the same time as one of the point cloud frames; A target frame acquisition module is used to acquire a target image frame set and a target point cloud frame set, wherein the target image frame is the image frame containing the object to be annotated, and the target point cloud frame is the point cloud frame collected at the same time as the target image frame; A first point cloud acquisition module, used for converting each point cloud frame in the target point cloud frame set into the same coordinate system to construct a first point cloud; A second point cloud acquisition module is used to project the first point cloud to each image frame in the target image frame set to acquire a second point cloud corresponding to each image frame, where the second point cloud is a point cloud projected into a 2D bounding box of the object to be annotated in the image frame; The 3D bounding box acquisition module is used to acquire, based on each of the second point clouds, a 3D bounding box corresponding to the object to be annotated in the image frame corresponding to each of the second point clouds.

10. The marking system according to claim 9, characterized in that: Also includes: A false detection filtering module is used to calculate the similarity of the 3D bounding boxes corresponding to the objects to be annotated in every two adjacent image frames in the target image frame set, obtain an image frame sequence that meets a preset similarity threshold and has a continuous frame number greater than a preset frame number threshold, and determine the target 3D bounding box; The semantic binding module is used to semantically bind the target 3D bounding box to the 2D bounding box of the object to be annotated in each image frame in the corresponding image frame sequence.

Images