Point cloud annotation method and device, vehicle

Abstract

The present application discloses a point cloud annotation method, device, and vehicle. The method includes: obtaining a target point cloud, multiple initial annotation frames in the target point cloud, and a target image, wherein the target point cloud is a collection of point data obtained by a target measurement device, the multiple initial annotation frames in the target point cloud are used to annotate the target content in the target point cloud, and the target image is an image corresponding to the target point cloud; projecting the multiple initial annotation frames in the target point cloud to the target image; processing the multiple initial annotation frames in the target point cloud based on the multiple initial annotation frames in the target image to obtain multiple target annotation frames in the target point cloud. Through the present application, the problem of low annotation accuracy when annotating on a point cloud in the related art is solved.

Description

Technical Field

[0001] The present application relates to the technical field of autonomous driving, and more specifically, to a point cloud annotation method and device, and a vehicle. Background Art

[0002] A deep learning model is one that takes images or videos as data input and predicts or returns pre-learned knowledge and labels. Deep learning models can be used for object recognition, dynamic analysis, trajectory analysis, object tracking, behavior prediction, and are widely used in computer vision and autonomous driving. In addition, deep learning models use complex multi-layer neural networks to simulate human brain neural networks and make complex decisions with extremely high accuracy requirements in a very short time. Therefore, the quality of the model's training set is particularly important.

[0003] Moreover, the content of the model's training set here specifically refers to the targets that appear in the autonomous driving scenario, which usually include roads (center lines, lane lines, road surface materials, etc.), traffic elements (traffic lights and traffic signs), traffic participants (motor vehicles, non-motor vehicles and pedestrians), road surrounding elements (including street lights, stations, trash cans, green belts, buildings), etc.

[0004] In related technologies, when constructing a model training set, the most commonly used method is to use the collected point cloud data and a graphic drawing tool to draw a label box outside the observed object. Specifically, first collect the point cloud and geographic coordinate information, and draw the point cloud on a drawing board based on the geographic coordinate information. Finally, export the category, length, width, height, x, y, z, and coordinate points of the label box in a pre-defined format.

[0005] At present, most of the annotation suppliers at home and abroad use the annotation tools in related technologies to load point clouds separately. However, due to the characteristics of the fuzzy scene of the point cloud, this implementation method will result in inaccurate observation of the target contour and omission of the target, that is, it will lead to low annotation accuracy.

[0006] With regard to the problem of low annotation accuracy when annotating point clouds in related technologies, no effective solution has been proposed so far. Summary of the invention

[0007] The main purpose of the present application is to provide a point cloud annotation method, device, and vehicle to solve the problem of low annotation accuracy when annotating on point clouds in related technologies.

[0008] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a method for annotating a point cloud is provided. The method comprises: obtaining a target point cloud, a plurality of initial annotation boxes in the target point cloud, and a target image, wherein the target point cloud is a collection of point data obtained by a target measurement device, the plurality of initial annotation boxes in the target point cloud are used to annotate the target content in the target point cloud, and the target image is an image corresponding to the target point cloud; projecting the plurality of initial annotation boxes in the target point cloud to the target image; processing the plurality of initial annotation boxes in the target point cloud according to the plurality of initial annotation boxes in the target image to obtain a plurality of target annotation boxes in the target point cloud.

[0009] Furthermore, before projecting multiple initial annotation boxes in the target point cloud to the target image, the method also includes: determining a target transformation matrix; calculating the coordinates of each initial annotation box in the target image based on the target transformation matrix and the coordinates of each initial annotation box in the target point cloud; and generating multiple initial annotation boxes in the target image based on the coordinates of each initial annotation box in the target image.

[0010] Furthermore, before determining the target transformation matrix, the method also includes: acquiring first distance information, second distance information, internal parameter information and external parameter information of a camera device, wherein the camera device is used to capture the target image, the first distance information is used to represent the distance between the target measurement device and the target object, and the second distance information is used to represent the distance between the target measurement device and a camera in the camera device; determining the target transformation matrix based on the first distance information, the second distance information, the internal parameter information and the external parameter information.

[0011] Further, based on the multiple initial annotation boxes in the target image, processing the multiple initial annotation boxes in the target point cloud to obtain multiple target annotation boxes in the target point cloud includes: adjusting the sizes of the multiple initial annotation boxes and / or adjusting the positions of the multiple initial annotation boxes based on the multiple initial annotation boxes in the target image to obtain multiple adjusted initial annotation boxes; and using the multiple adjusted initial annotation boxes as the multiple target annotation boxes.

[0012] Furthermore, after acquiring the target point cloud and the target image, the method further includes: initializing the coordinate system in the target point cloud, the coordinate system in the target image, the drawing module of the multiple initial annotation frames, and the control module of the multiple initial annotation frames.

[0013] Furthermore, before acquiring the target image, the method also includes: determining the generation time of the target point cloud and the generation time of the original image, wherein the original image is the image before the target image is generated; if the difference between the generation time of the target point cloud and the generation time of the original image is less than a preset time value, then the original image is used as the target image.

[0014] Furthermore, after obtaining multiple target annotation boxes in the target point cloud, the method also includes: obtaining attribute information of the multiple target annotation boxes, wherein the attribute information at least includes: name information and size information of each target annotation box; packaging the attribute information of the multiple target annotation boxes to obtain a target file; compressing the target file to obtain a compressed target file, and saving the compressed target file.

[0015] Furthermore, after obtaining multiple target annotation boxes in the target point cloud, the method also includes: generating a training set based on the data in the multiple target annotation boxes; and using the training set to perform learning and training on a deep learning model corresponding to the target object.

[0016] In order to achieve the above-mentioned purpose, according to another aspect of the present application, a point cloud annotation device is provided. The device comprises: a first acquisition unit, used to acquire a target point cloud, a plurality of initial annotation boxes in the target point cloud, and a target image, wherein the target point cloud is a collection of point data obtained by a target measurement device, the plurality of initial annotation boxes in the target point cloud are used to annotate the target content in the target point cloud, and the target image is an image corresponding to the target point cloud; a first projection unit, used to project the plurality of initial annotation boxes in the target point cloud to the target image; a first processing unit, used to process the plurality of initial annotation boxes in the target point cloud according to the plurality of initial annotation boxes in the target image, so as to obtain the plurality of target annotation boxes in the target point cloud.

[0017] Furthermore, the device also includes: a first determination unit, used to determine a target transformation matrix before projecting multiple initial annotation boxes in the target point cloud to the target image; a first calculation unit, used to calculate the coordinates of each initial annotation box in the target image based on the target transformation matrix and the coordinates of each initial annotation box in the target point cloud; and a first generation unit, used to generate multiple initial annotation boxes in the target image based on the coordinates of each initial annotation box in the target image.

[0018] Furthermore, the device also includes: a second acquisition unit, used to acquire first distance information, second distance information, internal parameter information and external parameter information of a camera device before determining a target transformation matrix, wherein the camera device is used to capture the target image, the first distance information is used to represent the distance between the target measurement device and the target object, and the second distance information is used to represent the distance between the target measurement device and a camera in the camera device; a second determination unit, used to determine the target transformation matrix based on the first distance information, the second distance information, the internal parameter information and the external parameter information.

[0019] Furthermore, the first processing unit includes: a first adjustment module, used to adjust the size of the multiple initial annotation boxes and / or adjust the positions of the multiple initial annotation boxes according to the multiple initial annotation boxes in the target image to obtain multiple adjusted initial annotation boxes; a first processing module, used to use the multiple adjusted initial annotation boxes as the multiple target annotation boxes.

[0020] Furthermore, the device also includes: a first initialization unit, which is used to initialize the coordinate system in the target point cloud, the coordinate system in the target image, the drawing module of the multiple initial annotation boxes, and the control module of the multiple initial annotation boxes after acquiring the target point cloud and the target image.

[0021] Furthermore, the device also includes: a third determination unit, used to determine the generation time of the target point cloud and the generation time of the original image before acquiring the target image, wherein the original image is the image before the target image is generated; and a second processing unit, used to use the original image as the target image if the difference between the generation time of the target point cloud and the generation time of the original image is less than a preset time value.

[0022] Furthermore, the device also includes: a third acquisition unit, used to obtain attribute information of the multiple target annotation boxes in the target point cloud after obtaining the multiple target annotation boxes, wherein the attribute information at least includes: name information and size information of each target annotation box; a first packaging unit, used to package the attribute information of the multiple target annotation boxes to obtain a target file; a third processing unit, used to compress the target file to obtain a compressed target file, and save the compressed target file.

[0023] Furthermore, the device also includes: a second generation unit, which is used to generate a training set based on the data in the multiple target annotation boxes after obtaining the multiple target annotation boxes in the target point cloud; and a first training unit, which is used to use the training set to perform learning and training on the deep learning model corresponding to the target object.

[0024] In order to achieve the above-mentioned purpose, according to another aspect of the present application, a vehicle is provided, which applies any one of the above-mentioned point cloud annotation methods during automatic driving.

[0025] Through this application, the following steps are adopted: obtaining a target point cloud, multiple initial annotation boxes in the target point cloud, and a target image, wherein the target point cloud is a collection of point data obtained by a target measurement device, the multiple initial annotation boxes in the target point cloud are used to annotate the target content in the target point cloud, and the target image is an image corresponding to the target point cloud; projecting the multiple initial annotation boxes in the target point cloud to the target image; processing the multiple initial annotation boxes in the target point cloud according to the multiple initial annotation boxes in the target image to obtain multiple target annotation boxes in the target point cloud, solving the problem of low annotation accuracy when annotating on the point cloud in the related art. By projecting the initial annotation box in the point cloud to the image corresponding to the point cloud, and processing the initial annotation box in the point cloud according to the initial annotation box in the image, the final annotation box in the point cloud can be obtained, so that when annotating on the point cloud, the annotation accuracy can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0027] Figure 1 is a flow chart of a point cloud annotation method provided according to an embodiment of the present application;

[0028] Figure 2 is a flowchart of an optional point cloud annotation method provided according to an embodiment of the present application;

[0029] Figure 3 It is a schematic diagram of a point cloud annotation device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0030] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0031] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, other existing embodiments obtained by ordinary technicians in the field without creative work should all fall within the scope of protection of the present application.

[0032] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0033] The present invention is described below in conjunction with preferred implementation steps. Figure 1 is a flow chart of a point cloud annotation method provided in an embodiment of the present application, such as Figure 1 As shown, the method comprises the following steps:

[0034] Step S101, obtaining a target point cloud, a plurality of initial annotation boxes in the target point cloud, and a target image, wherein the target point cloud is a collection of point data obtained by a target measurement device, the plurality of initial annotation boxes in the target point cloud are used to annotate target content in the target point cloud, and the target image is an image corresponding to the target point cloud.

[0035] For example, the target measurement device may be a radar device. Therefore, the target point cloud may be a collection of point data obtained by the radar device. The multiple initial annotation boxes in the target point cloud may be boxes that annotate the content of interest or focus in the point cloud. Moreover, the target image may be an image corresponding to the point cloud where the initial annotation boxes are located.

[0036] Step S102: projecting multiple initial annotation boxes in the target point cloud to the target image.

[0037] For example, the initial annotation box in the point cloud is projected onto the image corresponding to the point cloud.

[0038] Step S103 : processing the multiple initial annotation boxes in the target point cloud according to the multiple initial annotation boxes in the target image to obtain multiple target annotation boxes in the target point cloud.

[0039] For example, check whether the initial annotation box projected onto the image is aligned with the content to be annotated on the image. If not, adjust the position of the initial annotation box until all annotation boxes are placed in the correct position. Finally, the adjusted, i.e., aligned, annotation box is used as the final annotation box in the point cloud.

[0040] Through the above steps S101 to S103, by projecting the initial annotation box in the point cloud into the image corresponding to the point cloud, and processing the initial annotation box in the point cloud according to the initial annotation box in the image, the final annotation box in the point cloud can be obtained, thereby improving the accuracy of annotation when annotating on the point cloud.

[0041] In order to improve the accuracy of point cloud annotation, in the point cloud annotation method provided in the embodiment of the present application, the accuracy of point cloud annotation can also be improved by the following steps: the coordinate system in the target point cloud, the coordinate system in the target image, the drawing module of multiple initial annotation boxes and the control module of multiple initial annotation boxes are initialized.

[0042] For example, before annotating a point cloud, the initialization phase is first entered, and the initialization phase includes: (1) initializing the coordinate system, and using the right-hand coordinate system, that is, downloading the point cloud and the image, and loading the downloaded point cloud and the image to the corresponding positions, and then initializing the coordinate systems of the point cloud and the image respectively; (2) initializing the module for drawing the annotation box, and the specific method is to bind the drawing event with the mouse time; (3) initializing the module for controlling the annotation box, and the specific method is to bind the keyboard shortcut key with the control event.

[0043] In summary, by initializing the coordinate system, the drawing module of the annotation box, and the control module of the annotation box, the accuracy of the point cloud annotation can be improved when the point cloud is annotated subsequently.

[0044] In order to quickly and accurately obtain the target image, in the point cloud annotation method provided in the embodiment of the present application, the target image can also be obtained by the following steps: determining the generation time of the target point cloud and the generation time of the original image, wherein the original image is the image before the target image is generated; if the difference between the generation time of the target point cloud and the generation time of the original image is less than the preset time value, the original image is used as the target image.

[0045] For example, the above-mentioned preset time value may be 50 ms. Therefore, the time difference between the generation time of the finally acquired image and the generation time of the point cloud needs to be less than 50 ms.

[0046] Through the above solution, the image data corresponding to the point cloud can be obtained quickly and accurately.

[0047] In order to quickly and accurately obtain the transformation matrix, in the point cloud annotation method provided in the embodiment of the present application, the transformation matrix can also be obtained by the following steps: obtaining first distance information, second distance information, internal parameter information and external parameter information of a camera device, wherein the camera device is used to capture a target image, the first distance information is used to represent the distance between a target measuring device and a target object, and the second distance information is used to represent the distance between the target measuring device and a camera in the camera device; determining the target transformation matrix based on the first distance information, the second distance information, the internal parameter information and the external parameter information.

[0048] For example, the above-mentioned target measurement device can be a radar device, the above-mentioned camera device can be a camera, and the above-mentioned target object represents an object to be processed by a deep learning model (the processing method can be analysis processing or identification processing of the target object, etc.), and can be a vehicle or a robot, etc.

[0049] When the target object is a vehicle, first obtain the calibration information of the radar device relative to the main vehicle (which can be the distance between the radar device and the main vehicle), the calibration information of the camera for the radar (which can be the distance between the camera and the radar device), and the intrinsic and extrinsic parameters of the camera. Then, based on the above calibration information, the intrinsic and extrinsic parameters of the camera, calculate the conversion matrix (P=RK) between the radar device and the camera, where "P" represents the conversion matrix, "R" represents the intrinsic parameters of the camera, and "K" represents the extrinsic parameters of the camera.

[0050] Through the above scheme, the conversion matrix between the radar device and the camera can be calculated quickly and accurately.

[0051] In order to quickly and accurately project multiple initial annotation boxes in the target point cloud to the target image, in the point cloud annotation method provided in the embodiment of the present application, multiple initial annotation boxes in the target point cloud can also be projected to the target image through the following steps: determining the target transformation matrix; calculating the coordinates of each initial annotation box in the target image based on the target transformation matrix and the coordinates of each initial annotation box in the target point cloud; generating multiple initial annotation boxes in the target image based on the coordinates of each initial annotation box in the target image.

[0052] For example, the coordinates drawn by the annotator on the point cloud drawing board are calculated through the transformation matrix (P) at the positions of eight points in the image ("x, y, z" P), and then a labeling box (labeling frame) is drawn in the image according to the positions of the eight points. Among them, "x, y, z" represents the coordinate values ​​of the labeling box.

[0053] In summary, by using the calculated transformation matrix, the annotation box in the point cloud can be quickly and accurately projected into the image corresponding to the point cloud.

[0054] In order to obtain multiple target annotation boxes in a point cloud, in the point cloud annotation method provided in an embodiment of the present application, multiple target annotation boxes in the point cloud can also be obtained by the following steps: according to multiple initial annotation boxes in the target image, the sizes of the multiple initial annotation boxes and / or the positions of the multiple initial annotation boxes are adjusted to obtain multiple adjusted initial annotation boxes; and the multiple adjusted initial annotation boxes are used as multiple target annotation boxes.

[0055] For example, the projection function can adjust the position of the annotation box on the image or adjust the size of the annotation box on the image according to the repeated debugging of the annotation box by the annotator, and use the adjusted annotation box as the final annotation box in the point cloud.

[0056] In summary, by adjusting the initial annotation box, a more accurate annotation box can be obtained in the point cloud, which improves the accuracy of point cloud annotation.

[0057] In order to facilitate the use of multiple target annotation boxes for subsequent queries, in the point cloud annotation method provided in the embodiment of the present application, the attribute information of the target annotation boxes can also be compressed and packaged through the following steps: obtaining the attribute information of the multiple target annotation boxes, wherein the attribute information at least includes: the name information and size information of each target annotation box; packaging the attribute information of the multiple target annotation boxes to obtain a target file; compressing the target file to obtain a compressed target file, and saving the compressed target file.

[0058] For example, when all the annotation boxes are placed in the correct position, the attribute information of the annotation boxes is packaged, such as the name and size of the annotation boxes. The packaged attribute information file of the annotation boxes is compressed, and then the annotator starts the saving module to save the compressed package. If the attribute information of these annotation boxes is needed later, the compressed package can be directly downloaded to the local computer.

[0059] Through the above solution, the attribute information of the annotation box can be traced back, so that when these annotation boxes are queried and used in subsequent queries, they can be quickly found and used.

[0060] In order to learn and train the deep learning model corresponding to the target object, in the point cloud annotation method provided in the embodiment of the present application, the deep learning model corresponding to the target object can also be learned and trained through the following steps: generating a training set based on the data in multiple target annotation boxes; using the training set to learn and train the deep learning model corresponding to the target object.

[0061] For example, a training set is generated based on the data in the annotated box in the final adjusted point cloud, and then the generated training set is used to learn and train the deep model corresponding to the vehicle or robot to obtain a trained deep learning model. For example, in the vehicle's autonomous driving scene, some targets annotated in the point cloud can be combined to generate a training set. The above targets can be roads (center lines, lane lines, road surface materials, etc.), traffic elements (traffic lights and traffic signs), traffic participants (motor vehicles, non-motor vehicles and pedestrians), road surrounding elements (including street lights, stations, trash cans, green belts, buildings), etc. Then, the generated training set is used to train the deep learning model that recognizes these targets, so that a deep learning model with more accurate recognition can be obtained to identify obstacles, lane lines, etc. in autonomous driving scenes.

[0062] Through the above scheme, the training set can be accurately obtained, so that the deep learning model can be accurately trained using the training set, thereby improving the accuracy of the processing results obtained by the deep learning model.

[0063] For example, a flowchart of an optional point cloud annotation method, such as Figure 2 As shown, the optional point cloud annotation method includes the following steps: downloading the point cloud, the image and the previously annotated annotation box, initializing the coordinate system of the point cloud and the image, and loading (rendering) the point cloud, the image and the annotation box; downloading the intrinsic and extrinsic parameters of the camera, and calculating the projection matrix; initializing the control module and the drawing module of the annotation box; after the above initialization is completed, waiting for the user to start annotation; then creating the annotation box object; and projecting the created annotation box object into the image; then checking in the image whether the annotation box is aligned with the content of interest in the image; if aligned, completing the annotation; if not aligned, modifying and / or adjusting the annotation box object; after completing the annotation, compressing the annotation box file; and then saving or downloading the compressed annotation box file.

[0064] Specifically, it can be divided into the following five steps:

[0065] Step 1, install the Create-React project building tool, and introduce third-party dependencies such as threejs (a program for drawing), zip.js (a program for packaging and compressing files), and dat.gui (a program for adjusting the position of the annotation box). Download the React (a code for managing the position of the annotation box) library file from Github (a website name) and copy it to the project directory. For example, you can create a new node_modules (file name) folder in the project directory to store the relevant content of third-party dependencies.

[0066] Step 2: Use npm (a compilation tool) to install the tool and compile the entire application locally.

[0067] Step 3, use dat.gui to define some global variables as parameter configurations for the annotation box. For example, create scroll bars so that the annotator can adjust the length, width, and height of the annotation box, create a tab for the difference mode, connect the function of the difference module to the front-end user interface, and create a tab for saving compressed files so that the annotator can save the annotation file locally.

[0068] Step 4: Upload the camera's intrinsic and extrinsic parameters according to the user interface template. This application will save the calculated projection matrix.

[0069] Step 5, save locally or upload annotations, that is, after editing the annotation box, the user can choose to save it locally or upload it to the cloud.

[0070] Therefore, by comparing point clouds and images, the accuracy and return rate of annotations can be improved. In addition, this product has programmed more than ten keyboard shortcuts for advanced annotators, including switching of images and providing differential annotation. That is, when the data is continuous, the annotator uses the differential mode to automatically calculate the position of the annotation box in all frames between the first frame and the last frame, and slightly move the coordinates of the annotation box. It can also switch the perspective, etc., thereby improving the efficiency of point cloud annotation.

[0071] In summary, the point cloud annotation method provided in the embodiment of the present application obtains a target point cloud, multiple initial annotation boxes in the target point cloud, and a target image, wherein the target point cloud is a collection of point data obtained by a target measurement device, the multiple initial annotation boxes in the target point cloud are used to annotate the target content in the target point cloud, and the target image is an image corresponding to the target point cloud; the multiple initial annotation boxes in the target point cloud are projected to the target image; based on the multiple initial annotation boxes in the target image, the multiple initial annotation boxes in the target point cloud are processed to obtain multiple target annotation boxes in the target point cloud, which solves the problem of low annotation accuracy when annotating on the point cloud in the related art. By projecting the initial annotation box in the point cloud to the image corresponding to the point cloud, and processing the initial annotation box in the point cloud based on the initial annotation box in the image, the final annotation box in the point cloud can be obtained, so that when annotating on the point cloud, the annotation accuracy can be improved.

[0072] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0073] The present application also provides a point cloud annotation device. It should be noted that the point cloud annotation device of the present application can be used to execute the point cloud annotation method provided in the present application. The point cloud annotation device provided in the present application is introduced below.

[0074] Figure 3 Schematic diagram of a point cloud annotation device according to an embodiment of the present application. Figure 3 As shown, the device includes: a first acquisition unit 301, a first projection unit 302 and a first processing unit 303.

[0075] Specifically, the first acquisition unit 301 is used to acquire a target point cloud, a plurality of initial annotation boxes in the target point cloud, and a target image, wherein the target point cloud is a collection of point data obtained by a target measurement device, the plurality of initial annotation boxes in the target point cloud are used to annotate target content in the target point cloud, and the target image is an image corresponding to the target point cloud;

[0076] A first projection unit 302 is used to project a plurality of initial annotation boxes in the target point cloud to a target image;

[0077] The first processing unit 303 is used to process the multiple initial annotation boxes in the target point cloud according to the multiple initial annotation boxes in the target image to obtain multiple target annotation boxes in the target point cloud.

[0078] In summary, the point cloud annotation device provided in the embodiment of the present application acquires a target point cloud, multiple initial annotation boxes in the target point cloud, and a target image through a first acquisition unit 301, wherein the target point cloud is a collection of point data obtained by a target measurement device, and the multiple initial annotation boxes in the target point cloud are used to annotate the target content in the target point cloud, and the target image is an image corresponding to the target point cloud; the first projection unit 302 projects the multiple initial annotation boxes in the target point cloud to the target image; the first processing unit 303 processes the multiple initial annotation boxes in the target point cloud according to the multiple initial annotation boxes in the target image to obtain multiple target annotation boxes in the target point cloud, which solves the problem of low annotation accuracy when annotating on the point cloud in the related art. By projecting the initial annotation box in the point cloud to the image corresponding to the point cloud, and processing the initial annotation box in the point cloud according to the initial annotation box in the image, the final annotation box in the point cloud can be obtained, so that when annotating on the point cloud, the annotation accuracy can be improved.

[0079] Optionally, in the point cloud annotation device provided in the embodiment of the present application, the device also includes: a first determination unit, used to determine the target transformation matrix before projecting multiple initial annotation boxes in the target point cloud to the target image; a first calculation unit, used to calculate the coordinates of each initial annotation box in the target image based on the target transformation matrix and the coordinates of each initial annotation box in the target point cloud; a first generation unit, used to generate multiple initial annotation boxes in the target image based on the coordinates of each initial annotation box in the target image.

[0080] Optionally, in the point cloud annotation device provided in the embodiment of the present application, the device also includes: a second acquisition unit, used to acquire first distance information, second distance information, internal parameter information and external parameter information of a camera device before determining a target transformation matrix, wherein the camera device is used to capture a target image, the first distance information is used to represent the distance between a target measuring device and a target object, and the second distance information is used to represent the distance between the target measuring device and a camera in the camera device; a second determination unit, used to determine the target transformation matrix based on the first distance information, the second distance information, the internal parameter information and the external parameter information.

[0081] Optionally, in the point cloud annotation device provided in an embodiment of the present application, the first processing unit includes: a first adjustment module, used to adjust the size of multiple initial annotation boxes and / or adjust the positions of multiple initial annotation boxes according to multiple initial annotation boxes in the target image to obtain multiple adjusted initial annotation boxes; a first processing module, used to use the multiple adjusted initial annotation boxes as multiple target annotation boxes.

[0082] Optionally, in the point cloud annotation device provided in an embodiment of the present application, the device also includes: a first initialization unit, which is used to initialize the coordinate system in the target point cloud, the coordinate system in the target image, the drawing module of multiple initial annotation boxes, and the control module of multiple initial annotation boxes after acquiring the target point cloud and the target image.

[0083] Optionally, in the point cloud annotation device provided in the embodiment of the present application, the device also includes: a third determination unit, used to determine the generation time of the target point cloud and the generation time of the original image before acquiring the target image, wherein the original image is the image before the target image is generated; and a second processing unit, used to use the original image as the target image if the difference between the generation time of the target point cloud and the generation time of the original image is less than a preset time value.

[0084] Optionally, in the point cloud annotation device provided in the embodiment of the present application, the device also includes: a third acquisition unit, used to obtain attribute information of multiple target annotation boxes after obtaining multiple target annotation boxes in the target point cloud, wherein the attribute information at least includes: name information and size information of each target annotation box; a first packaging unit, used to package the attribute information of multiple target annotation boxes to obtain a target file; a third processing unit, used to compress the target file to obtain a compressed target file, and save the compressed target file.

[0085] Optionally, in the point cloud annotation device provided in the embodiment of the present application, the device also includes: a second generation unit, used to generate a training set based on data in the multiple target annotation boxes after obtaining multiple target annotation boxes in the target point cloud; and a first training unit, used to use the training set to perform learning and training on a deep learning model corresponding to the target object.

[0086] For example, the core of the point cloud annotation tool includes three parts, namely, the image annotation module, the point cloud annotation module, and the application coordination module. The image annotation module is used to edit and display images, calculate and draw annotation boxes on images, project annotation boxes to corresponding camera images, create image elements with projection annotation boxes and images, scale image elements, and text-bind annotations (set the name of each annotation box) to achieve the purpose of reference for point cloud annotation boxes. The functions of the point cloud annotation module include creating cameras (creating functions), visualizing point cloud files, building orbit cameras, and binding with mouse and keyboard actions to achieve full visibility of point cloud scenes; positioning, building laser objects (rays), building laser objects through mouse click data, and locating the mouse in the point cloud; drawing annotation boxes, drawing annotation boxes through the actual position of the mouse in the point cloud; building editors, and fine-tuning the annotation boxes by reading the annotation box size in the point cloud context and user input. The functions of the application coordination module include operating on the data content, providing a human-computer interaction interface, and the axios (a code used to send network requests) network framework to change, save, and trace back the data content; bridging the point cloud annotation module and the image annotation module, exchanging data, and notifying the image annotation module when updating the point cloud annotation; reading, saving, and replacing the projection matrix and data set information through the axios network framework.

[0087] In addition, the point cloud annotation tool is developed in Javascript, supports static web pages, server-side rendering, event-driven NodeJs (a program for sending network requests, extracted from the axios network framework) backend, and uses fast resolution. In addition, the protobuf data (compressed data) protocol with high efficiency and support for multiple languages ​​can be used for lightweight annotation locally. ThreeJs (a program for drawing) is responsible for the drawing and projection of all modules, such as the interpolate module (a code for automatically calculating the position of the annotation box), the projector module (a module for projection), the raphael module (a module for drawing), and the Raycaster module (a module for drawing rays). The function of the Raycaster module is to generate an intersection based on the starting position of the mouse click, a ray emitted by the angle, and the point through which the point cloud panel passes, get the first point, record the mouse stroke, record the position where the mouse is released, and get the first point of the intersection through which the ray and the point cloud panel pass. These two points and the mouse stroke will be handed over to the raphael module, and the point cloud annotation box will be drawn through the interface of webgl (a program for rendering clouds). Subsequently, the application coordination module will notify the image annotation module of the successfully drawn annotation box. The annotation module reads the projection matrix, calculates the point cloud annotation and the projection matrix, and obtains the image annotation information. The image annotation information is then handed over to raphael to draw and integrate the image and the annotation box. Therefore, on this basis, the point cloud and image are drawn and fused and annotated to achieve the purpose of improving accuracy.

[0088] The point cloud annotation device includes a processor and a memory. The first acquisition unit 301, the first projection unit 302 and the first processing unit 303 are all stored in the memory as program units, and the processor executes the program units stored in the memory to implement corresponding functions.

[0089] The processor contains a kernel, which calls the corresponding program unit from the memory. One or more kernels can be set, and the accuracy of point cloud annotation can be improved by adjusting the kernel parameters.

[0090] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0091] An embodiment of the present invention provides a vehicle, which applies any one of the above-mentioned point cloud annotation methods during automatic driving.

[0092] An embodiment of the present invention provides an electronic device, the device includes a processor, a memory, and a program stored in the memory and executable on the processor, and the processor implements the following steps when executing the program: acquiring a target point cloud, a plurality of initial annotation boxes in the target point cloud, and a target image, wherein the target point cloud is a collection of point data obtained by a target measurement device, the plurality of initial annotation boxes in the target point cloud are used to annotate target content in the target point cloud, and the target image is an image corresponding to the target point cloud; projecting the plurality of initial annotation boxes in the target point cloud to the target image; and processing the plurality of initial annotation boxes in the target point cloud according to the plurality of initial annotation boxes in the target image to obtain the plurality of target annotation boxes in the target point cloud.

[0093] When the processor executes the program, the following steps are also implemented: before projecting multiple initial annotation boxes in the target point cloud to the target image, the method also includes: determining a target transformation matrix; calculating the coordinates of each initial annotation box in the target image based on the target transformation matrix and the coordinates of each initial annotation box in the target point cloud; and generating multiple initial annotation boxes in the target image based on the coordinates of each initial annotation box in the target image.

[0094] The processor also implements the following steps when executing the program: before determining the target transformation matrix, the method also includes: obtaining first distance information, second distance information, internal parameter information and external parameter information of a camera device, wherein the camera device is used to capture the target image, the first distance information is used to represent the distance between the target measurement device and the target object, and the second distance information is used to represent the distance between the target measurement device and a camera in the camera device; determining the target transformation matrix based on the first distance information, the second distance information, the internal parameter information and the external parameter information.

[0095] When the processor executes the program, the following steps are also implemented: based on the multiple initial annotation boxes in the target image, the multiple initial annotation boxes in the target point cloud are processed to obtain multiple target annotation boxes in the target point cloud, including: based on the multiple initial annotation boxes in the target image, the sizes of the multiple initial annotation boxes and / or the positions of the multiple initial annotation boxes are adjusted to obtain multiple adjusted initial annotation boxes; and the multiple adjusted initial annotation boxes are used as the multiple target annotation boxes.

[0096] When the processor executes the program, the following steps are also implemented: after acquiring the target point cloud and the target image, the method also includes: initializing the coordinate system in the target point cloud, the coordinate system in the target image, the drawing module of the multiple initial annotation boxes, and the control module of the multiple initial annotation boxes.

[0097] When the processor executes the program, the following steps are also implemented: before acquiring the target image, the method also includes: determining the generation time of the target point cloud and the generation time of the original image, wherein the original image is the image before the target image is generated; if the difference between the generation time of the target point cloud and the generation time of the original image is less than a preset time value, the original image is used as the target image.

[0098] When the processor executes the program, the following steps are also implemented: after obtaining multiple target annotation boxes in the target point cloud, the method also includes: obtaining attribute information of the multiple target annotation boxes, wherein the attribute information at least includes: name information and size information of each target annotation box; packaging the attribute information of the multiple target annotation boxes to obtain a target file; compressing the target file to obtain a compressed target file, and saving the compressed target file.

[0099] When the processor executes the program, the following steps are also implemented: after obtaining multiple target annotation boxes in the target point cloud, the method also includes: generating a training set based on the data in the multiple target annotation boxes; using the training set to learn and train the deep learning model corresponding to the target object.

[0100] The devices in this article can be servers, PCs, PADs, mobile phones, etc.

[0101] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program that initializes the following method steps: obtaining a target point cloud, multiple initial annotation boxes in the target point cloud, and a target image, wherein the target point cloud is a collection of point data obtained by a target measurement device, the multiple initial annotation boxes in the target point cloud are used to annotate target content in the target point cloud, and the target image is an image corresponding to the target point cloud; projecting the multiple initial annotation boxes in the target point cloud to the target image; and processing the multiple initial annotation boxes in the target point cloud based on the multiple initial annotation boxes in the target image to obtain multiple target annotation boxes in the target point cloud.

[0102] When executed on a data processing device, it is also suitable for executing a program initialized with the following method steps: before projecting multiple initial annotation boxes in the target point cloud to the target image, the method also includes: determining a target transformation matrix; calculating the coordinates of each initial annotation box in the target image based on the target transformation matrix and the coordinates of each initial annotation box in the target point cloud; generating multiple initial annotation boxes in the target image based on the coordinates of each initial annotation box in the target image.

[0103] When executed on a data processing device, it is also suitable for executing a program initialized with the following method steps: before determining the target transformation matrix, the method also includes: obtaining first distance information, second distance information, internal parameter information and external parameter information of a camera device, wherein the camera device is used to capture the target image, the first distance information is used to represent the distance between the target measurement device and the target object, and the second distance information is used to represent the distance between the target measurement device and a camera in the camera device; determining the target transformation matrix based on the first distance information, the second distance information, the internal parameter information and the external parameter information.

[0104] When executed on a data processing device, it is also suitable for executing a program initialized with the following method steps: based on the multiple initial annotation boxes in the target image, processing the multiple initial annotation boxes in the target point cloud to obtain multiple target annotation boxes in the target point cloud includes: based on the multiple initial annotation boxes in the target image, adjusting the sizes of the multiple initial annotation boxes and / or adjusting the positions of the multiple initial annotation boxes to obtain multiple adjusted initial annotation boxes; using the multiple adjusted initial annotation boxes as the multiple target annotation boxes.

[0105] When executed on a data processing device, it is also suitable for executing an initialization program having the following method steps: after acquiring the target point cloud and the target image, the method also includes: initializing the coordinate system in the target point cloud, the coordinate system in the target image, the drawing module of the multiple initial annotation boxes, and the control module of the multiple initial annotation boxes.

[0106] When executed on a data processing device, it is also suitable for executing a program initialized with the following method steps: before acquiring a target image, the method also includes: determining the generation time of the target point cloud and the generation time of the original image, wherein the original image is an image before the target image is generated; if the difference between the generation time of the target point cloud and the generation time of the original image is less than a preset time value, the original image is used as the target image.

[0107] When executed on a data processing device, it is also suitable for executing a program initialized with the following method steps: after obtaining multiple target annotation boxes in the target point cloud, the method also includes: obtaining attribute information of the multiple target annotation boxes, wherein the attribute information at least includes: name information and size information of each target annotation box; packaging the attribute information of the multiple target annotation boxes to obtain a target file; compressing the target file to obtain a compressed target file, and saving the compressed target file.

[0108] When executed on a data processing device, it is also suitable for executing a program that initializes the following method steps: after obtaining multiple target annotation boxes in the target point cloud, the method also includes: generating a training set based on the data in the multiple target annotation boxes; using the training set to learn and train the deep learning model corresponding to the target object.

[0109] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the application can adopt the form of complete hardware embodiments, complete software embodiments, or embodiments in conjunction with software and hardware. Moreover, the application can adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program codes.

[0110] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0111] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0112] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0113] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0114] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0115] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0116] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0117] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) in which a computer-usable program code is present.

[0118] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, various changes and variations may be present in the present application. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A point cloud annotation method, characterized in that: The method comprises: Acquire a target point cloud, a plurality of initial annotation boxes in the target point cloud, and a target image, wherein the target point cloud is a collection of point data obtained by a target measurement device, the plurality of initial annotation boxes in the target point cloud are used to annotate target content in the target point cloud, and the target image is an image corresponding to the target point cloud; Projecting a plurality of initial annotation boxes in the target point cloud to the target image; Processing the multiple initial annotation boxes in the target point cloud according to the multiple initial annotation boxes in the target image to obtain multiple target annotation boxes in the target point cloud; Wherein, before projecting the multiple initial annotation boxes in the target point cloud to the target image, the method further includes: Determine the target transformation matrix; Calculate the coordinates of each initial annotation box in the target image according to the target transformation matrix and the coordinates of each initial annotation box in the target point cloud; A plurality of initial annotation boxes are generated in the target image according to the coordinates of each initial annotation box in the target image.

2. The method according to claim 1, characterized in that Before determining the target transformation matrix, the method further includes: Acquire first distance information, second distance information, internal parameter information and external parameter information of a camera device, wherein the camera device is used to capture the target image, the first distance information is used to indicate the distance between the target measurement device and the target object, and the second distance information is used to indicate the distance between the target measurement device and a camera in the camera device; The target transformation matrix is ​​determined according to the first distance information, the second distance information, the internal parameter information and the external parameter information.

3. The method according to claim 1, characterized in that Processing the multiple initial annotation boxes in the target point cloud according to the multiple initial annotation boxes in the target image to obtain multiple target annotation boxes in the target point cloud includes: According to the multiple initial annotation frames in the target image, adjusting the sizes of the multiple initial annotation frames and / or adjusting the positions of the multiple initial annotation frames to obtain multiple adjusted initial annotation frames; The multiple adjusted initial annotation boxes are used as the multiple target annotation boxes.

4. The method according to claim 1, characterized in that: After acquiring the target point cloud and the target image, the method further includes: The coordinate system in the target point cloud, the coordinate system in the target image, the drawing module of the multiple initial annotation frames, and the control module of the multiple initial annotation frames are all initialized.

5. The method according to claim 1, characterized in that Before acquiring the target image, the method further includes: Determine the generation time of the target point cloud and the generation time of the original image, wherein the original image is an image before the target image is generated; If the difference between the generation time of the target point cloud and the generation time of the original image is less than a preset time value, the original image is used as the target image.

6. The method according to claim 1, characterized in that After obtaining a plurality of target annotation boxes in the target point cloud, the method further includes: Acquire attribute information of the multiple target annotation boxes, wherein the attribute information at least includes: name information and size information of each target annotation box; Packing the attribute information of the multiple target annotation boxes to obtain a target file; The target file is compressed to obtain a compressed target file, and the compressed target file is saved.

7. The method according to claim 2, characterized in that: After obtaining a plurality of target annotation boxes in the target point cloud, the method further includes: Generate a training set based on the data in the multiple target annotation frames; The training set is used to perform learning and training on the deep learning model corresponding to the target object.

8. A point cloud annotation device, characterized in that: The device comprises: A first acquisition unit is used to acquire a target point cloud, a plurality of initial annotation boxes in the target point cloud, and a target image, wherein the target point cloud is a collection of point data obtained by a target measurement device, the plurality of initial annotation boxes in the target point cloud are used to annotate target content in the target point cloud, and the target image is an image corresponding to the target point cloud; A first projection unit, configured to project a plurality of initial annotation boxes in the target point cloud to the target image; A first processing unit is used to process the multiple initial annotation boxes in the target point cloud according to the multiple initial annotation boxes in the target image to obtain multiple target annotation boxes in the target point cloud; A first determining unit is used to determine a target transformation matrix before projecting multiple initial annotation boxes in the target point cloud to the target image; a first calculating unit is used to calculate the coordinates of each initial annotation box in the target image based on the target transformation matrix and the coordinates of each initial annotation box in the target point cloud; and a first generating unit is used to generate multiple initial annotation boxes in the target image based on the coordinates of each initial annotation box in the target image.

9. A vehicle, characterized in that: The vehicle applies the point cloud annotation method described in any one of claims 1 to 7 during autonomous driving.

Images